<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:1765104460;
mso-list-template-ids:-1653674652;}
@list l0:level1
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:1.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:1.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:2.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:2.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:3.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:3.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:4.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:4.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal">Workshop at ACM RecSys 2018, Vancouver, CA<o:p></o:p></p>
<p class="MsoNormal">October 7, 2018<o:p></o:p></p>
<p class="MsoNormal"><a href="https://sites.google.com/view/reveal2018/home">https://sites.google.com/view/reveal2018/home</a><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">One of the main goals of offline metrics for recommender systems is to indicate the future online performance of the same recommender system in an online setting, as measured by various user-based utility metrics, such as the time spent
by a user on the website, the number of media items consumed, the number of attributed sales or the reported user satisfaction. However, practitioners often observe significant differences between offline and online results of a new algorithm, and therefore
tend to mostly rely on online methods such as A/B testing to evaluate their algorithms.
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">To this end, we welcome contributions that advance the current state of knowledge on offline recommendation metrics that correlate well with the final performance of the online recommender system; or new recommendation algorithms that directly
optimize for online metrics.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">We invite submissions of 2-8 pages to be presented as talks or posters. The reviews will be single-blind.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Potential contributions include (but are not limited to): <o:p>
</o:p></p>
<ul style="margin-top:0in" type="square">
<li class="MsoNormal" style="mso-list:l0 level1 lfo1"><b>Framing the problem: what are we trying to solve exactly?</b> This includes work on the theoretical foundations of the recommendation task and on the corresponding offline metrics:<o:p></o:p>
<ul style="margin-top:0in" type="square">
<li class="MsoNormal" style="mso-list:l0 level2 lfo1"><b>Recommendation as a counterfactual inference problem.</b> This encompasses all work on new offline metrics and new optimization criteria for recommendation using ideas from causal inference, such as:
Learning from data <b>Missing Not at Random</b> (MNAR), <b>Counterfactual Risk Minimization</b> (CRM) and
<b>Batch Learning from Bandit Feedback</b> (BLBF), <b>Deconvolving</b> recommendation-lead from organic feedback in logged data, Causal Inference using
<b>domain adaptation </b><o:p></o:p></li><li class="MsoNormal" style="mso-list:l0 level2 lfo1"><b>Recommendation as a reinforcement learning problem.
</b>This is aimed to all work that frames recommendation as a reinforcement learning task and that borrows ideas from RL on evaluating recommendation policies, such as the use of simulation in recommender systems evaluation.<o:p></o:p></li></ul>
</li><li class="MsoNormal" style="mso-list:l0 level1 lfo1"><b>Studies on offline-online metrics correlation for Recommendation.
</b><o:p></o:p></li><li class="MsoNormal" style="mso-list:l0 level1 lfo1"><b>More realistic offline metrics.</b><o:p></o:p>
<ul style="margin-top:0in" type="square">
<li class="MsoNormal" style="mso-list:l0 level2 lfo1">Offline metrics for slate recommendation evaluation.<o:p></o:p></li><li class="MsoNormal" style="mso-list:l0 level2 lfo1">Offline metrics for logged data with sequential recommendation exposure and delayed feedback.<o:p></o:p></li></ul>
</li><li class="MsoNormal" style="mso-list:l0 level1 lfo1"><b>Datasets and toolkits.</b><o:p></o:p>
<ul style="margin-top:0in" type="square">
<li class="MsoNormal" style="mso-list:l0 level2 lfo1"><b>New exploration schemes</b> to improve the collection of more informative offline datasets.<o:p></o:p></li><li class="MsoNormal" style="mso-list:l0 level2 lfo1"><b>Open datasets.</b> All recommendation dataset releases that aim to somehow bridge the gap between offline and online metrics.<o:p></o:p></li><li class="MsoNormal" style="mso-list:l0 level2 lfo1"><b>Toolkits.</b> Including new software for evaluation metrics, reproducible research and baseline algorithms.<o:p></o:p></li></ul>
</li></ul>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Submission deadline: July 28th, 2018 via <a href="https://www.google.com/url?q=https%3A%2F%2Fcmt3.research.microsoft.com%2FOFFLINEEVAL2018%2F&sa=D&sntz=1&usg=AFQjCNEHK77oJ6PBX9gt7KkZIypuZVlIBA" target="_blank">
our CMT website.</a><o:p></o:p></p>
<p class="MsoNormal">---<o:p></o:p></p>
<p class="MsoNormal">Thorsten Joachims<o:p></o:p></p>
<p class="MsoNormal">Professor, Cornell University<o:p></o:p></p>
<p class="MsoNormal">Department of Computer Science<o:p></o:p></p>
<p class="MsoNormal">Department of Information Science<o:p></o:p></p>
<p class="MsoNormal"><a href="http://www.joachims.org/"><span style="color:blue">http://www.joachims.org/</span></a><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>