This half-day workshop seeks to bring together practitioners and academics interested in the challenges of structuring interactions for subject matter experts (SMEs) who are providing knowledge and/or feedback to an AI system, but are not well-versed in the underlying algorithms. Since the information provided by SMEs directly effects the efficacy of the final system, collecting the correct data is a problem that navigates issues ranging from curating data that may be tainted to structuring data collection tasks in such a way as to mitigate user boredom. The goal of this workshop is to discuss methods and new paradigms for productively interacting with users while collecting knowledge.