[CMU AI Seminar] February 28 at 12pm (GHC 6115 & Zoom) -- Steven Jecmen (CMU) -- Off-Policy Evaluation of Peer-Review Assignment Strategies -- AI Seminar sponsored by SambaNova Systems

[Asher Trockman]

Dear all,

We look forward to seeing you *this Tuesday (2/28)* from *1**2:00-1:00 PM
(U.S. Eastern time)* for the next talk of this semester's *CMU AI Seminar*,
sponsored by SambaNova Systems <https://sambanova.ai/>. The seminar will be
held in GHC 6115 *with pizza provided *and will be streamed on Zoom.

To learn more about the seminar series or to see the future schedule,
please visit the seminar website <http://www.cs.cmu.edu/~aiseminar/>.

This Tuesday (2/28), *Steven Jecmen* (CMU) will be giving a talk
titled *"**Off-Policy
Evaluation of Peer-Review Assignment Strategies**".*

*Title*: Off-Policy Evaluation of Peer-Review Assignment Strategies

*Talk Abstract*: Peer review assignment algorithms aim to match research
papers to suitable expert reviewers, working to maximize the quality of the
resulting reviews. A key challenge in designing effective assignment
strategies is evaluating how changes to the assignment algorithm map to
changes in quality. In this talk, I will show how we leverage
recently-proposed strategies that introduce randomness in peer-review
assignment (aiming to mitigate fraud) as a valuable opportunity to evaluate
counterfactual assignment strategies. To address challenges in applying
standard off-policy evaluation techniques, we introduce novel methods for
partial identification based on mild assumptions about the review quality
outcomes. We apply our methods to peer-review data from two computer
science venues and examine the effect on review quality of various changes
to the paper assignment algorithms.

*Speaker Bio:* Steven Jecmen is a fourth-year Ph.D. student in the Computer
Science Department at Carnegie Mellon University, advised by Fei Fang and
Nihar Shah. Steven's recent research focuses primarily on preventing
undesirable behavior in settings with human evaluators, particularly in
academic peer review. His personal website is at https://sjecmen.github.io/.

*In person: *GHC 6115
*Zoom Link*:
https://cmu.zoom.us/j/99510233317?pwd=ZGx4aExNZ1FNaGY4SHI3Qlh0YjNWUT09

Thanks,
Asher Trockman
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.srv.cs.cmu.edu/pipermail/ai-seminar-announce/attachments/20230226/eb0b9250/attachment.html>