Connectionists: Call for Participation: TREC 2017 Tasks Track

[Rishabh Mehrotra]

-----------------------------------
TREC 2017 Tasks Track
-----------------------------------

Website: http://www.cs.ucl.ac.uk/tasks-track-2017/

Deadline for TREC Application: ***16th May 2017** *(register here
<http://ir.nist.gov/trecsubmit.open/application.html>)

*Track Background:*
The primary goals of the track are to evaluate system's understanding of
tasks users aim to achieve and system's performance in helping users
complete their tasks.

Research in Information Retrieval has traditionally focused on serving the
best results for a single query, ignoring the reasons (or the task) that
might have motivated the user to submit that query. Often times search
engines are used to complete complex tasks (information needs); achieving
these tasks with current search engines requires users to issue multiple
queries. Standard evaluation mechanisms focus on evaluating the quality of
a retrieval system in terms of the relevance of the results retrieved,
completely ignoring the fact that user satisfaction mainly depends on the
usefullness of the system in helping the user complete the actual task that
led the user issue the query. The TREC Tasks Track is an attempt in
devising mechanisms for evaluating quality of retrieval systems in terms of
(1) how well they can understand the underlying task that led the user
submit a query, and (2) how useful they are for helping users complete
their tasks.


*Potential Application Areas:*
Researchers from varied different fields could use the Task track data to
evaluate their models. Few application areas include:

   - Task based retrieval
   - Digital assistants tasks
   - Query suggestions
   - Session based retrieval
   - Exploratory search
   - Entity-based results

To make evaluation and system development easier, we provide participants
with API access to the Clueweb B indexed on Microsoft Azure.


*Timeline:*
We anticipate the following timeline:

   - 16th May 2017: TREC Participation closes
   - mid June 2017: Tasks available to participants
   - 24th August 2017: Submission deadline


*Track Coordinators*
Emine Yilmaz (University College London)
Evangelos Kanoulas (University of Amsterdam)
Nick Craswell (Microsoft)
Peter Bailey (Microsoft)
Ben Carterette (University of Delaware)
Rishabh Mehrotra (University College London)


*Application Details*
Organizations wishing to participate in TREC 2017 Tasks Track should
respond to this call for participation by submitting an application.
Participants in previous TRECs who wish to participate in TREC 2017 must
submit a new application. To apply, submit the online application at

                      http://ir.nist.gov/trecsubmit.open/application.html


*Contact:*
Mailing list: tasks-track at googlegroups.com
Subscribe: https://groups.google.com/forum/#!forum/tasks-track


-- 
Rishabh Mehrotra
PhD candidate
University College London
Web: www.rishabhmehrotra.com
Twitter: https://twitter.com/erishabh
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.srv.cs.cmu.edu/pipermail/connectionists/attachments/20170515/49158471/attachment.html>