Connectionists: CFP: Workshop on Insights from Negative Results (EMNLP 2020)

[Anna Rogers]

*

(apologies for cross-posting)

Call For Papers: Workshop on Insights from Negative Results

(Nov 19 2020, co-located with EMNLP 2020)

Workshop website:https://insights-workshop.github.io/


=======

Overview

=======


Publication of negative results is difficult in most fields, but in NLP 
the problem is exacerbated by the near-universal focus on improvements 
in benchmarks. This situation implicitly discourages hypothesis-driven 
research, and it turns creation and fine-tuning of NLP models into art 
rather than science. Furthermore, it increases the time, effort, and 
carbon emissions spent on developing and tuning models, as the 
researchers have no opportunity to learn what has already been tried and 
failed.


This workshop invites both practical and theoretical unexpected or 
negative results that have important implications for future research, 
highlight methodological issues with existing approaches, and/or point 
out pervasive misunderstandings or bad practices. In particular, the 
most successful NLP models currently rely on different kinds of 
pretrained meaning representations (from word embeddings to 
Transformer-based models like BERT). To complement all the success 
stories, it would be insightful to see where and possibly why they fail. 
Any NLP tasks are welcome: sequence labeling, question answering, 
inference, dialogue, machine translation - you name it.


A successful negative results paper would contribute one of the following:


* experiments on (in)stability of the previously published results due 
to hardware, random initializations, etc.;

* ablation studies of components in previously proposed models, showing 
that their contributions are different from the initially reported;

* datasets or probing tasks showing that previous approaches do not 
generalize to other domains or language phenomena;

* extensions or annotations of existing datasets which show that prior 
successes are due to spurious statistical factors or annotation artifacts;

* the respective contributions of pre-training vs fine-tuning to the end 
result;

* cross-lingual studies showing that a technique X is only successful 
for a certain language or language family;

* broadly applicable recommendations for training/fine-tuning, 
especially if the X that didn’t work is something that many 
practitioners would think reasonable to try, and if the demonstration of 
X’s failure is accompanied by some explanation/hypothesis.


Some examples of insightful negative results papers are listed on the 
workshop website:

https://insights-workshop.github.io/papers


============

Important Dates

============


* Anonymity period begins: July 15, 2020

* Submission deadline: August 15, 2020

* Notification of acceptance: September 29, 2020

* Camera-ready papers due: October 10, 2020

* Workshop: November 19, 2020


All deadlines are 11.59 pm UTC -12h ("anywhere on Earth").


==========

Submissions

==========


Submission is electronic, using the Softconf START conference management 
system.

Submission link:https://www.softconf.com/emnlp2020/insights2020/


Both long and short papers must follow the EMNLP 2020 two-column format.

Official style sheets:https://2020.emnlp.org/files/emnlp2020-templates.zip


Please do not modify these style files, nor should you use templates 
designed for other conferences. Submissions that do not conform to the 
required styles, including paper size, margin width, and font size 
restrictions, will be rejected without review.


The workshop will accept short papers (up to 4 pages), as well as 1-2 
page non-archival abstract submissions for papers published elsewhere 
(e.g. in one of the main conferences or in non-NLP venues). The goal of 
this event is to stimulate a meaningful community-wide discussion of the 
deep issues in NLP methodology, and the authors of both types of 
submissions will be welcome to take part in our virtual get-togethers.


========

Authorship

========


The author list for submissions should include all (and only) 
individuals who made substantial contributions to the work presented. No 
changes to the order or composition of authorship may be made after the 
paper submission deadline.


===================

Citation and Comparison

===================


You are expected to cite all refereed publications relevant to your 
submission, but you may be excused for not knowing about all unpublished 
work (especially work that has been recently posted and/or is not widely 
cited).


In cases where a preprint has been superseded by a refereed publication, 
the refereed publication should be cited instead of the preprint version.


Papers (whether refereed or not) appearing less than 3 months before the 
submission deadline are considered contemporaneous to your submission, 
and you are therefore not obliged to make detailed comparisons that 
require additional experimentation and/or in-depth analysis.


For more information, see the ACL Policies for Submission, Review, and 
Citation.


===================================

Multiple Submission Policy and Fast-Tracking

===================================


We welcome dual submissions, as long as they are specified at the time 
of submission time.


The authors submitting a paper that is under review for EMNLP 2020 
should specify that in the submission form. The organizers would then be 
able to access the reviews and may decide to fast-track the paper to the 
workshop without additional reviews, if the paper is rejected from the 
main conference.


If the paper has been rejected from another venue, the authors will have 
the option to provide the original reviews and the author response. The 
new reviewers will not have access to this information, but the 
organizers will be able to take into account the fact that the paper has 
already been revised and improved.


===============

NEW: Ethics Policy

===============


EMNLP workshops follow the conference guidelines for honouring the ACM 
Code of Ethics:

https://www.acm.org/code-of-ethics

Per conference guidelines, a paper that may raise ethical issues needs 
to explicitly discuss them, and that discussion will be taken into 
account in the review process.

Specific to the topic of negative results is the problem of revisiting 
published papers that cannot be reproduced. In most cases 
irreproducibility comes down to general methodological problems, but if 
you have reason to believe the unreproducible result was deliberately 
fabricated, that should be discussed.


===========

Reproducibility

===========


Publishing negative results is not easy, partly because the author has 
the burden of proof that something truly does not work, rather than is 
caused by a bug.


We encourage the authors to link code repositories in the camera-ready 
versions. At submission time, each submission can be accompanied by one 
PDF appendix for the paper, one PDF for prior reviews and author 
response, one .tgz or .zip archive containing software, and one.tgz or 
.zip archive containing data (all fully anonymized). The appendix can 
document preprocessing decisions, model parameters, feature templates, 
lengthy proofs or derivations, pseudocode, sample system inputs/outputs, 
and other details that are necessary for the exact replication of the 
work (see the official EMNLP reproducibility guidelines). However, the 
paper submissions need to remain fully self-contained, as these 
supplementary materials are completely optional, and reviewers are not 
even asked to review or download them.


=============

Anonymity Period

=============


We follow EMNLP anonymity policy. The anonymity period runs from 1 month 
before the submission deadline (starting July 15, 2020) up to the date 
when your paper is accepted or rejected (September 29, 2020). The papers 
should be properly anonymized and not publicized up until the acceptance 
notifications come out.


==========

Presentation

==========


All accepted papers must be presented at the workshop to appear in the 
proceedings. Authors of accepted papers must notify the program chairs 
by the camera-ready deadline if they wish to withdraw the paper. At 
least one author of each accepted paper must register for the workshop.

Previous presentations of the work (e.g. preprints on arXiv.org) should 
be noted in a footnote in the camera-ready version (but not in the 
anonymized version of the paper).


The workshop will take place online on November 19 2020, incorporating 
the best practices from other online conferences this year (such as 
ICLR). For now we are expecting to have pre-recorded oral presentations 
of the papers, asynchronous Q&A sessions (in which more than one author 
for each paper can take part), and also community discussion sessions 
and/or panels.



*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.srv.cs.cmu.edu/pipermail/connectionists/attachments/20200525/55498298/attachment.html>