<div dir="ltr"><p class="MsoNormal"><span style="font-size:14pt;font-family:Arial,sans-serif;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">************************</span><span style="font-size:14pt;font-family:Arial,sans-serif"><br></span><b><span style="font-size:16pt;font-family:Arial,sans-serif;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">Call for papers</span></b><b><span style="font-size:16pt;font-family:Arial,sans-serif"><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">Workshop Title: </span></span></b><b><span style="font-size:16pt;font-family:Arial,sans-serif;color:black">Human in the loop dialogue systems Workshop @ NeurIPS 2020</span></b><span style="font-size:12pt;color:black"><u></u><u></u></span></p><p class="MsoNormal"><b><span style="font-size:16pt;font-family:Arial,sans-serif"><br></span></b><b><span style="font-size:14pt;font-family:Arial,sans-serif;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">Website:</span></b><span style="font-size:14pt;font-family:Arial,sans-serif;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">        </span><span style="font-size:14pt;font-family:Arial,sans-serif;color:black"><a href="https://sites.google.com/view/hlds-2020/home" target="_blank"><span style="color:rgb(5,99,193)">https://sites.google.com/view/hlds-2020/home</span></a></span><span style="font-size:14pt;font-family:Arial,sans-serif"> <br><b><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">Workshop Date:</span></b><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">  </span></span><span style="font-size:14pt;font-family:Arial,sans-serif;color:black;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">Will fall on between 12/6/2020 – 12/12/2020</span><span style="font-size:14pt;font-family:Arial,sans-serif"><br><b><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">Submission:</span></b><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">           </span></span><span style="font-size:14pt;font-family:Arial,sans-serif;color:black"><a href="https://www.google.com/url?q=https%3A%2F%2Fcmt3.research.microsoft.com%2FHLDS2020%2FSubmission%2FIndex&sa=D&sntz=1&usg=AFQjCNG945q1l4ICz1kwV7HrbSUQ5Ydy1A" target="_blank"><span style="color:rgb(5,99,193)">https://cmt3.research.microsoft.com/HLDS2020/Submission/Index</span></a></span><span style="font-size:14pt;font-family:Arial,sans-serif"><br><b><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">Submission Deadline:</span></b><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">               October 2^nd 2020 (Anywhere on Earth)</span></span><span style="font-size:12pt;color:black"><u></u><u></u></span></p><p class="MsoNormal"><b><span style="font-size:14pt;font-family:Arial,sans-serif">Contact:</span></b><span style="font-size:14pt;font-family:Arial,sans-serif"> </span><span style="font-size:12pt;font-family:Arial,sans-serif;color:rgb(33,33,33)"> </span><u><span style="font-size:14pt;font-family:Arial,sans-serif;color:rgb(0,101,128)"><a href="mailto:hld-2020-organizers@googlegroups.com" target="_blank"><span style="color:rgb(5,99,193)">hld-2020-organizers</span></a><a href="mailto:hld-2020-organizers@googlegroups.com" target="_blank"><span style="color:rgb(5,99,193)">@googlegroups.com</span></a></span></u><span style="font-size:14pt;font-family:Arial,sans-serif"><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">*************************</span><br><b><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">Description:</span></b><br></span><span style="font-size:14pt;font-family:Arial,sans-serif;color:rgb(33,33,33)">Conversational interaction systems such as Amazon Alexa, Google Assistant, Apple Siri, and Microsoft Cortana have become very popular over the recent years. Such systems have allowed users to interact with a wide variety of content on the web through a conversational interface. Research challenges such as the Dialogue System Technology Challenges, Dialogue Dodecathlon, Amazon Alexa Prize and the Vision and Language Navigation task have continued to inspire research in conversational AI. These challenges have brought together researchers from different communities such as speech recognition, spoken language understanding, reinforcement learning, language generation, and multi-modal question answering.</span><span style="font-size:12pt;color:black"><u></u><u></u></span></p><p class="MsoNormal" style="text-align:justify"><span style="font-size:14pt;font-family:Arial,sans-serif;color:rgb(33,33,33)">Unlike other popular NLP tasks, dialogue frequently has humans in the loop, whether it is for evaluation, active learning or online reward estimation. Through this workshop we aim to bring together researchers from academia and industry to discuss the challenges and opportunities in such human in the loop setups. We hope that this sparks interesting discussions about conversational agents, interactive systems, and how we can use humans most effectively when building such systems. We will highlight areas such as human evaluation setups, reliability in human evaluation, human in the loop training, interactive learning and user modeling. We also highly encourage non-English based dialogue systems in these areas.</span><span style="color:black"><u></u><u></u></span></p><p class="MsoNormal" style="text-align:justify"><span style="font-size:14pt;font-family:Arial,sans-serif;color:rgb(33,33,33)">The one-day workshop will include talks from senior technical leaders and researchers to share insights associated with training and evaluating dialogue systems. We also plan on having oral presentations and poster sessions on the works related to the workshop. Finally, we will end the workshop with an interactive panel of speakers. As an outcome we expect the participants from the NeurIPS community to walk away with better understanding of human in the loop dialogue modeling as well as key areas of research in this field. Additionally we would like to see discussions around the unification of human evaluation setups in a standardized way.</span><span style="color:black"><u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:14pt;font-family:Arial,sans-serif"> </span><span style="font-size:12pt;color:black"><u></u><u></u></span></p><p class="MsoNormal"><b><span style="font-size:14pt;font-family:Arial,sans-serif;color:black">Scope and topics included but not limited to:</span></b><span style="font-size:12pt;color:black"><u></u><u></u></span></p><ul type="disc"><li class="MsoNormal" style="margin-left:15px;color:black"><span style="font-size:14pt;font-family:Arial,sans-serif">Human Evaluation</span><u></u><u></u></li></ul><ul type="disc"><ul type="circle"><li class="MsoNormal" style="margin-left:15px;color:black"><span style="font-size:14pt;font-family:Arial,sans-serif;color:windowtext">Human evaluation setups and best practices</span><u></u><u></u></li><li class="MsoNormal" style="margin-left:15px;color:black"><span style="font-size:14pt;font-family:Arial,sans-serif;color:windowtext">Cost effective setups</span><u></u><u></u></li><li class="MsoNormal" style="margin-left:15px;color:black"><span style="font-size:14pt;font-family:Arial,sans-serif;color:windowtext">Reliability and dealing with bias of evaluation results</span><u></u><u></u></li><li class="MsoNormal" style="margin-left:15px;color:black"><span style="font-size:14pt;font-family:Arial,sans-serif;color:windowtext">Interpretability of dialogue systems</span><u></u><u></u></li></ul></ul><ul type="disc"><li class="MsoNormal" style="margin-left:15px;color:black"><span style="font-size:14pt;font-family:Arial,sans-serif">User Modeling</span><u></u><u></u></li></ul><ul type="disc"><ul type="circle"><li class="MsoNormal" style="margin-left:15px;color:black"><span style="font-size:14pt;font-family:Arial,sans-serif">Model based evaluation</span><u></u><u></u></li><li class="MsoNormal" style="margin-left:15px;color:black"><span style="font-size:14pt;font-family:Arial,sans-serif">User simulation</span><u></u><u></u></li></ul></ul><ul type="disc"><li class="MsoNormal" style="margin-left:15px;color:black"><span style="font-size:14pt;font-family:Arial,sans-serif">Human-in-the-loop</span><u></u><u></u></li></ul><ul type="disc"><ul type="circle"><li class="MsoNormal" style="margin-left:15px;color:black"><span style="font-size:14pt;font-family:Arial,sans-serif">Online and offline reinforcement learning</span><u></u><u></u></li><li class="MsoNormal" style="margin-left:15px;color:black"><span style="font-size:14pt;font-family:Arial,sans-serif">Active learning</span><u></u><u></u></li><li class="MsoNormal" style="margin-left:15px;color:black"><span style="font-size:14pt;font-family:Arial,sans-serif">Visual and language learning</span><u></u><u></u></li></ul></ul><p class="MsoNormal"><b><span style="font-size:14pt;font-family:Arial,sans-serif"><br><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">Invited Speakers:</span></span></b><span style="font-size:12pt;color:black"><u></u><u></u></span></p><ul type="disc"><li class="MsoNormal" style="margin-left:15px;color:black"><span style="font-size:14pt;font-family:Arial,sans-serif">Zhou Yu (Columbia University)</span><u></u><u></u></li><li class="MsoNormal" style="margin-left:15px;color:black"><span style="font-size:14pt;font-family:Arial,sans-serif">Alexander Rudnicky (Carnegie Mellon University)</span><u></u><u></u></li><li class="MsoNormal" style="margin-left:15px;color:black"><span style="font-size:14pt;font-family:Arial,sans-serif">Maxine Eskenazi (Carnegie Mellon University)</span><u></u><u></u></li><li class="MsoNormal" style="margin-left:15px;color:black"><span style="font-size:14pt;font-family:Arial,sans-serif">Gokhan Tur (Amazon)</span><u></u><u></u></li><li class="MsoNormal" style="margin-left:15px;color:black"><span style="font-size:14pt;font-family:Arial,sans-serif">Jason Weston (Facebook)</span><u></u><u></u></li><li class="MsoNormal" style="margin-left:15px;color:black"><span style="font-size:14pt;font-family:Arial,sans-serif">Milica Gasic (Heinrich Heine University Düsseldorf)</span><u></u><u></u></li><li class="MsoNormal" style="margin-left:15px;color:black"><span style="font-size:14pt;font-family:Arial,sans-serif">Ankur Parikh (Google)</span><u></u><u></u></li><li class="MsoNormal" style="margin-left:15px;color:black"><span style="font-size:14pt;font-family:Arial,sans-serif">Percy Liang (Stanford)</span><u></u><u></u></li><li class="MsoNormal" style="margin-left:15px;color:black"><span style="font-size:14pt;font-family:Arial,sans-serif">Larry Heck (Samsung)</span><u></u><u></u></li></ul><p class="MsoNormal"><b><span style="font-size:14pt;font-family:Arial,sans-serif;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">Organizers:</span></b><span style="font-size:12pt;color:black"><u></u><u></u></span></p><ul type="disc"><li class="MsoNormal" style="margin-left:15px;color:black"><span style="font-size:14pt;font-family:Arial,sans-serif;color:rgb(34,34,34)">Behnam Hedayatnia (Amazon)</span><u></u><u></u></li><li class="MsoNormal" style="margin-left:15px;color:black"><span style="font-size:14pt;font-family:Arial,sans-serif;color:rgb(34,34,34)">Rahul Goel (Google)</span><u></u><u></u></li><li class="MsoNormal" style="margin-left:15px;color:black"><span style="font-size:14pt;font-family:Arial,sans-serif;color:rgb(34,34,34)">Shereen Oraby (Amazon)</span><u></u><u></u></li><li class="MsoNormal" style="margin-left:15px;color:black"><span style="font-size:14pt;font-family:Arial,sans-serif;color:rgb(34,34,34)">Abigail See (Stanford)</span><u></u><u></u></li><li class="MsoNormal" style="margin-left:15px;color:black"><span style="font-size:14pt;font-family:Arial,sans-serif;color:rgb(34,34,34)">Chandra Khatri (Uber)</span><u></u><u></u></li><li class="MsoNormal" style="margin-left:15px;color:black"><span style="font-size:14pt;font-family:Arial,sans-serif;color:rgb(34,34,34);background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">Y-Lan Boureau (Facebook)</span><u></u><u></u></li><li class="MsoNormal" style="margin-left:15px;color:black"><span style="font-size:14pt;font-family:Arial,sans-serif;color:rgb(34,34,34);background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">Alborz Geramifard (Facebook)</span><u></u><u></u></li><li class="MsoNormal" style="margin-left:15px;color:black"><span style="font-size:14pt;font-family:Arial,sans-serif;color:rgb(34,34,34);background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">Marylin Walker (UC Santa Cruz)</span><u></u><u></u></li><li class="MsoNormal" style="margin-left:15px;color:black"><span style="font-size:14pt;font-family:Arial,sans-serif;color:rgb(34,34,34);background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial">Dilek Hakkani-Tur (Amazon)</span></li></ul></div>