Connectionists: Final Call For Papers: Workshop on Insights from Negative Results (EMNLP 2020)

[Anna Rogers]

(apologies for cross-posting)

Final Call For Papers: Workshop on Insights from Negative Results
(Nov 19 2020, co-located with EMNLP 2020)
Workshop website: https://insights-workshop.github.io/

=======
Overview
=======

Publication of negative results is difficult in most fields, but in NLP the
problem is exacerbated by the near-universal focus on improvements in
benchmarks. This situation implicitly discourages hypothesis-driven
research, and it turns creation and fine-tuning of NLP models into art
rather than science. Furthermore, it increases the time, effort, and carbon
emissions spent on developing and tuning models, as the researchers have no
opportunity to learn what has already been tried and failed.

This workshop invites both practical and theoretical unexpected or negative
results that have important implications for future research, highlight
methodological issues with existing approaches, and/or point out pervasive
misunderstandings or bad practices. In particular, the most successful NLP
models currently rely on different kinds of pretrained meaning
representations (from word embeddings to Transformer-based models like
BERT). To complement all the success stories, it would be insightful to see
where and possibly why they fail. Any NLP tasks are welcome: sequence
labeling, question answering, inference, dialogue, machine translation -
you name it.

A successful negative results paper would contribute one of the following:

* experiments on (in)stability of the previously published results due to
hardware, random initializations, etc.;
* ablation studies of components in previously proposed models, showing
that their contributions are different from the initially reported;
* datasets or probing tasks showing that previous approaches do not
generalize to other domains or language phenomena;
* extensions or annotations of existing datasets which show that prior
successes are due to spurious statistical factors or annotation artifacts;
* the respective contributions of pre-training vs fine-tuning to the end
result;
* cross-lingual studies showing that a technique X is only successful for a
certain language or language family;
* broadly applicable recommendations for training/fine-tuning, especially
if the X that didn’t work is something that many practitioners would think
reasonable to try, and if the demonstration of X’s failure is accompanied
by some explanation/hypothesis.

Some examples of insightful negative results papers are listed on the
workshop website:
https://insights-workshop.github.io/papers

============
Important Dates
============

* Anonymity period begins: July 15, 2020
* Submission deadline: August 15, 2020
* Notification of acceptance: September 29, 2020
* Camera-ready papers due: October 10, 2020
* Workshop: November 19, 2020

All deadlines are 11.59 pm UTC -12h ("anywhere on Earth").

==========
Submissions
==========

Submission is electronic, using the Softconf START conference management
system.

Submission link: https://www.softconf.com/emnlp2020/insights2020/

The workshop will accept short papers (up to 4 pages, excluding
references), as well as 1-2 page non-archival abstract submissions for
papers published elsewhere (e.g. in one of the main conferences or in
non-NLP venues). The goal of this event is to stimulate a meaningful
community-wide discussion of the deep issues in NLP methodology, and the
authors of both types of submissions will be welcome to take part in our
virtual get-togethers.

Both research papers and abstracts must follow the EMNLP 2020 two-column
format.
Official style sheets: https://2020.emnlp.org/files/emnlp2020-templates.zip

Please do not modify these style files, nor should you use templates
designed for other conferences. Submissions that do not conform to the
required styles, including paper size, margin width, and font size
restrictions, will be rejected without review.


========
Authorship
========

The author list for submissions should include all (and only) individuals
who made substantial contributions to the work presented. No changes to the
order or composition of authorship may be made after the paper submission
deadline.

===================
Citation and Comparison
===================

You are expected to cite all refereed publications relevant to your
submission, but you may be excused for not knowing about all unpublished
work (especially work that has been recently posted and/or is not widely
cited).

In cases where a preprint has been superseded by a refereed publication,
the refereed publication should be cited instead of the preprint version.

Papers (whether refereed or not) appearing less than 3 months before the
submission deadline are considered contemporaneous to your submission, and
you are therefore not obliged to make detailed comparisons that require
additional experimentation and/or in-depth analysis.

For more information, see the ACL Policies for Submission, Review, and
Citation.

===================================
Multiple Submission Policy and Fast-Tracking
===================================

We welcome dual submissions, as long as they are specified at the time of
submission time.

The authors submitting a paper that is under review for EMNLP 2020 should
specify that in the submission form. The organizers would then be able to
access the reviews and may decide to fast-track the paper to the workshop
without additional reviews, if the paper is rejected from the main
conference.

If the paper has been rejected from another venue, the authors will have
the option to provide the original reviews and the author response. The new
reviewers will not have access to this information, but the organizers will
be able to take into account the fact that the paper has already been
revised and improved.

===============
NEW: Ethics Policy
===============

EMNLP workshops follow the conference guidelines for honouring the ACM Code
of Ethics:
https://www.acm.org/code-of-ethics
Per conference guidelines, a paper that may raise ethical issues needs to
explicitly discuss them, and that discussion will be taken into account in
the review process.
Specific to the topic of negative results is the problem of revisiting
published papers that cannot be reproduced. In most cases irreproducibility
comes down to general methodological problems, but if you have reason to
believe the unreproducible result was deliberately fabricated, that should
be discussed.

===========
Reproducibility
===========

Publishing negative results is not easy, partly because the author has the
burden of proof that something truly does not work, rather than is caused
by a bug.

We encourage the authors to link code repositories in the camera-ready
versions. At submission time, each submission can be accompanied by one PDF
appendix for the paper, one PDF for prior reviews and author response, one
.tgz or .zip archive containing software, and one.tgz or .zip archive
containing data (all fully anonymized). The appendix can document
preprocessing decisions, model parameters, feature templates, lengthy
proofs or derivations, pseudocode, sample system inputs/outputs, and other
details that are necessary for the exact replication of the work (see the
official EMNLP reproducibility guidelines). However, the paper submissions
need to remain fully self-contained, as these supplementary materials are
completely optional, and reviewers are not even asked to review or download
them.

=============
Anonymity Period
=============

We follow EMNLP anonymity policy. The anonymity period runs from 1 month
before the submission deadline (starting July 15, 2020) up to the date when
your paper is accepted or rejected (September 29, 2020). The papers should
be properly anonymized and not publicized up until the acceptance
notifications come out.

==========
Presentation
==========

All accepted papers must be presented at the workshop to appear in the
proceedings. Authors of accepted papers must notify the program chairs by
the camera-ready deadline if they wish to withdraw the paper. At least one
author of each accepted paper must register for the workshop.
Previous presentations of the work (e.g. preprints on arXiv.org) should be
noted in a footnote in the camera-ready version (but not in the anonymized
version of the paper).

The workshop will take place online on November 19 2020, incorporating the
best practices from other online conferences this year (such as ICLR and
ACL). For now we are expecting to have pre-recorded oral presentations of
the papers, live Q&A sessions (in which more than one author for each paper
can take part), and also community discussion sessions and/or panels.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.srv.cs.cmu.edu/pipermail/connectionists/attachments/20200728/927e046e/attachment.html>