Connectionists: [CLEF 2026] Call for Lab Proposals - Deadline 14th of July
Julia Maria Struß
julia.struss at fh-potsdam.de
Thu Jun 26 07:29:54 EDT 2025
***Apologies for cross-posting ***
-------------------------------------------
CLEF 2026
Conference and Labs of the Evaluation Forum
Jena, Germany, September 21-24, 2026
https://clef2026.clef-initiative.eu/ <http://clef2026.clef-initiative.eu>
-------------------------------------------
Call for Lab Proposals
Background
The CLEF Initiative <http://www.clef-initiative.eu/>is a self-organised
body whose main mission is to promote research, innovation, and
development of information access systems with an emphasis on
multilingual information in different modalities - including text and
multimedia - with various levels of structure. CLEF promotes research
and development by providing an infrastructure for:
1.
Independent evaluation of information access systems
2.
Investigation of the use of unstructured, semi-structured,
highly-structured, and semantically enriched data in information access
3.
Creation of reusable test collections for benchmarking
4.
Exploration of new evaluation methodologies and innovative ways of
using experimental data
5.
Discussion of results, comparison of approaches, exchange of ideas,
and transfer of knowledge
Scope of CLEF Labs
We invite submission of proposals for two types of labs:
1.
"Campaign-style" Evaluation Labs for specific information access
problems (during the twelve months period preceding the conference),
similar in nature to the traditional CLEF campaign "tracks". Topics
covered by campaign-style labs can be inspired by any information
access-related domain or task.
2.
Labs that follow a more classical "workshop" pattern, exploring
evaluation methodology, metrics, processes, etc. in information
access and closely related fields, such as natural language
processing, machine translation, and human-computer interaction.
We highly recommend organisers new to the CLEF format of shared task
evaluation campaigns to first consider organising a lab workshop to
discuss the format of their proposed task, the problem space and
practicalities of the shared task. The CLEF 2026 programme will reserve
about half of the conference schedule for lab sessions.
During the conference, the lab organisers will present their overall
results in overview presentations during the plenary scientific paper
sessions to give non-participants insights into where the research
frontiers are moving. Lab organisers are expected to organise separate
sessions for their lab with ample time for general discussion and
engagement with all participants - not just those presenting campaign
results and papers. Organisers should plan time in their sessions for
activities such as panels, demos, poster sessions, etc. as appropriate.
CLEF is always interested in receiving and facilitating innovative lab
proposals.
Potential task proposers unsure of the suitability of their task
proposal or its format for inclusion at CLEF are encouraged to contact
the CLEF 2026 Lab Organizing Committee Chairs to discuss its suitability
or design at an early stage.
Proposal Submission
Lab proposals must provide sufficient information to judge the
relevance, timeliness, scientific quality, benefits for the research
community, and the competence of the proposers to coordinate the lab.
Each lab proposal should identify one or more organisers as responsible
for ensuring the timely execution of the lab. Proposals should be 3 to 4
pageslong and should provide the following information:
1.
Title of the proposed lab.
2.
A brief description of the lab topic and goals, its relevance to
CLEF and the significance for the field.
3.
A brief and clear statement on usage scenarios and domain to which
the activity is intended to contribute, including the evaluation
setup and metrics.
4.
Details on the lab organiser(s), including identifying the task
chair(s) responsible for ensuring the running of the task. This
should include details of any previous involvement in organising or
participating in evaluation tasks at CLEF or similar campaigns.
5.
The planned format of the lab, i.e., campaign-style (“track”) or
workshop.
6.
Is the lab a continuation of an activity from previous year(s) or a
new activity?
1.
For activities continued from previous year(s): Statistics from
previous years (number of participants/runs for each task), a
clear statement on why another edition is needed, an explicit
listing of the changes proposed, and a discussion of lessons to
be learned or insights to be made.
2.
For new activities: A statement on why a new evaluation campaign
is needed and how the community would benefit from the activity.
7.
Details of the expected target audience, i.e., who do you expect to
participate in the task(s), and how do you propose to reach them.
8.
Brief details of tasks to be carried out in the lab. The proposal
should clearly motivate the need for each of the proposed tasks and
provide evidence of its capability of attracting enough
participation. The dataset which will be adopted by the Lab needs to
be described and motivated in the perspective of the goals of the
Labs; also indications on how the dataset will be shared are useful.
It is fine for a lab to have a single task, but labs often contain
multiple closely related tasks, needing a strong motivation for more
than 3 tasks, to avoid useless fragmentation.
9.
Expected length of the lab session at the conference: half-day, one
day, two days. This should include high-level details of planned
structure of the session, e.g. participant presentations, invited
speaker(s), panels, etc., to justify the requested session length.
10.
Arrangements for the organisation of the lab campaign: who will be
responsible for activities within the task; how will data be
acquired or created, what tools or methods will be used, e.g., how
will necessary queries be created or relevance assessment carried
out; any other information which is relevant to the conduct of your lab.
11.
If the lab proposes to set up a steering committee to oversee and
advise its activities, include names, addresses, and homepage links
of people you propose to be involved.
Lab proposals must be submitted via EasyChair. The link will be
distributed, once EasyChair is set up.
Review Process
Each proposal submitted by 14 July 2025will be reviewed by the CLEF 2026
Lab Organising Committee. The acceptance decision will be sent by email
to the responsible organiser by 4 Aug 2025. The final length of the lab
session at the conference will be determined based on the overall
organisation of the conference and the number of participant submissions
received by a lab.
Advertising Labs at CLEF 2025 and ECIR 2026
Organisers of accepted labs are expected to advertise their labs at both
CLEF 2025 (September 9-12, 2025, Madrid, Spain) and ECIR 2026 (March 29
- April 2, 2026, Delft, Netherlands). So, at least one lab
representative should attend these events.
Advertising at CLEF 2025 will consist of displaying a poster describing
the new lab and advertising/announcing it during the closing session.
Advertising at ECIR 2026 will consist of submitting a lab description
(abstract submission deadline TBA by ECIR) to be included in ECIR 2026
proceedings and advertising the lab in a booster session during ECIR 2026.
Lab Proposals from Newcomers
If you have not organised a lab before, do not panic! The CLEF 2026 Lab
Organising Committee Lab is willing to mentor you by offering help,
guidance, and feedback on the writing of your draft lab proposal.
If you are a newcomer interested in receiving guidance, please send an
e-mail with the following tag in the subject “[Mentorship CLEF 2026 Lab
Proposals]” to Sean.MacAvaney at glasgow.ac.uk and julia.struss at
fh-potsdam.de
We also encourage newcomers to refer toFriedberg et al. (2015)
<https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1004150>for
initial guidance on preparing their proposal:
Friedberg I, Wass MN, Mooney SD, Radivojac P. Ten simple rules for a
community computational challenge. PLoS Comput Biol. 2015 Apr
23;11(4):e1004150.
Important Dates
*
14 July 2025:Hard deadline to submit proposal to Easychair
*
4 August 2025:Notification of lab acceptance
*
9-12 September 2025:Advertising Accepted Labs at CLEF 2025, Madrid,
Spain
*
October 2025 (TBA by ECIR):Submission of short lab description for
ECIR 2026
*
April 2026:Advertising labs at ECIR 2026, Delft, Netherlands
*
April-May:Lab evaluation cycle
*
May-June:Review process of participant papers
*
June 2026:Review of the condensed labs overviews
*
July 2026:CEUR-WS Working Notes Preview for Checking by Authors and
Lab Organisers
*
21-24 September, 2026:Labs at CLEF 2026
CLEF 2026 Lab Chairs
*
Julia Maria Struß, Fachhochschule Potsdam University of Applied Sciences
*
Sean MacAvaney, University of Glasgow
--
___________________________
Prof. Dr. Julia Maria Struß
Fachhochschule Potsdam
University of Applied Sciences
Fachbereich Informationswissenschaften
Kiepenheuerallee 5
14469 Potsdam
Telefon: +49 331 580 4532
Zoom:https://fh-potsdam.zoom-x.de/my/juliamstruss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.srv.cs.cmu.edu/pipermail/connectionists/attachments/20250626/b51939e9/attachment.html>
More information about the Connectionists
mailing list