ARQMath Home

**Update, June 2024. A backup for the ARQMath data, runs, evaluation scripts, etc. stored in Google Drive is now available here.

ARQMath is a cooperative evaluation task aiming to advance math-aware search and the semantic analysis of mathematical notation and texts. These web pages are for the third edition of the task, being run as part of the CLEF 2022 conference being held in Bologna, Italy.

To learn about participating, and to access ARQMath data, tools, and paper from previous ARQMath labs, please visit the Resources page. Make sure to join the ARQMath Forum to keep up to date with the lab's activities and announcements.

ARQMath Overview and Participant Papers:

ARQMath-3 (CLEF 2022 CEUR Working Notes)
ARQMath-2 (CLEF 2021 CEUR Working Notes)
ARQMath-1 (CLEF 2020 CEUR Working Notes)

**Corrected** ARQMath-1 Overview Paper (Table A2)
Errata: in the updated version of the ARQMath-1 (2020) overview paper above, Task 2 results were updated to correct a normalization error for the nDCG' metrics. This change did not affect system rankings, but corrected nDCG' scores are higher.
[ Errata explanatory notes ] and accompanying [ spreadsheet ]

LNCS ARQMath Overview Papers

(Please note: Overview papers in the CEUR Working Notes contain additional results and information)

Video. Brief overviews of ARQMath tasks have appeared in ECIR 2020, ECIR 2021, and will be presented at ECIR 2022 this year. A video from Behrooz Mansouri's presentation at ECIR 2021 is provided above -- please note that there is a new Open Domain QA task for ARQMath-3 this year (Task 3).

(July 11) Looking forward to meeting up with participants at CLEF this Septmber in Bologna! Registration is open through the end of July for both in-person and virtual attendance. Early registration closes July 20.

(July 11) The evaluation portion of ARQMath-3 is now complete. A big thanks from the lab organizers to all participants, and to our student assessors from RIT. In the coming weeks we will compile and release additional topics and other data.

(Apr 18) pt-arqmath, a new text-based baseline for Task 1 (Answer Retrieval) is available (built on PyTerrier).

(Mar 1) Topics for ARQMath-3 along with an updated collection are now available.

Run file formats are available through the ARQMath Google Drive, including for the new Task 3 (Open QA)

We welcome Vít Novotný to the organizing team. Vít will lead Task 3.

ARQMath-2 Best Paper Award. We congratulate Anja Reusch, Maik Thiele, and Wolfgang Lehner on having their paper "TU_DBS in the ARQMath Lab 2021" selected as the Best Paper of ARQMath-2. A revised version of their paper will be included as part of a special session for Best Papers at CLEF 2022.

ARQMath-1 Best Paper Award. The Best Paper for ARQMath-1 was awarded to Yin Ki Ng, Dallas Fraser, Besat Kassaie and Frank Tompa for their paper "Dowsing for Math Answers" (original version). A revised version was presented in the CLEF 2021 Best Paper track.

~~Early Feb~~ Mar 1 - Evaluation topics released (Note: previous topics from 2020 / 2021 are available)

April 22 - Registration closes

May 6 - Last day to submit runs for CLEF 2022 labs (including ARQMath-3)

May 27 - Participant paper deadline

~~June 6~~ June 11 - Condensed LNCS lab overview due (written by organizers)
June 13 - Notification of acceptance for participant papers

~~June 20~~ June 22 - Camera Ready LNCS Overview Paper due

July 1 - Camera Ready Copy of Participant Papers and Overview Paper for CEUR-WS working notes

July 18-22 - CEUR-WS preview available for checking by participants and lab organizers

Richard Zanibbi (rxzvcs@rit.edu) Department of Computer Science Rochester Institute of Technology	Behrooz Mansouri (bm3302@rit.edu) Department of Computer Science Rochester Institute of Technology
Douglas W. Oard (oard@umd.edu) College of Information Studies, and Institute for Advanced Computer Studies (UMIACS) University of Maryland, College Park	Anurag Agarwal (axasma@rit.edu) School of Mathematical Sciences Rochester Institute of Technology Vít Novotný (witiko@mail.muni.cz) Faculty of Informatics Masaryk University

Answer Retrieval for Questions on Math (2022)

ARQMath Overview and Participant Papers

ARQMath Overview Video (Tasks 1 and 2)

News

Key Dates ( 2022 )

Organizers

Questions: rxzvcs@rit.edu