NOTE -- Please cite the ARQMath-3 Overview paper if you use ARQMath data or tools in your work:
Mansouri, B., Zanibbi, R., Oard, D.W., and Agarwal, A. Overview of ARQMath-3 (2022): Third CLEF Lab on Answer Retrieval for Questions on Math (Working Notes Version)
. CLEF 2022 Working Notes, pp. 1-27. Baseline Systems.
Baseline systems for Tasks 1 and 2 are available through the ARQMath Google Drive
. Note that Task 1 baselines can be converted to a Task 3 baseline simply by returning only the first hit obtained by the Task 1 system. New: for Task 1/3, a text-based baseline system built on top of PyTerrier is now available (pt-arqmath). Evaluation Tools, Data, and Previous Runs.
Tools for evaluation are available, along with qrel files (i.e., relevance scores), a detailed record of assessment data that produced the qrels, and system runs from previous ARQMath participants. See the Evaluation
directory on the ARQMath Google Drive for more information.Formula Index Files. Formula index files
are provided in three encodings -- LaTeX
(from MSE posts) and Presentation MathML
represent formula appearance, while formula syntax is represented using Content MathML
. Both the appearance and math syntax encodings are trees: Symbol Layout Trees (SLTs) for appearance, and Operator Trees (OPTs) for formula operation syntax. Formulas are grouped by appearance into 'visually distinct' groups prior to assessment. For ARQMath 2021 and 2022, we have pre-computed and enumerated these groups, and provided the unique 'visual group' identifier in the formula index files.
Visually distinct formula groups are computed using their Tangent-S Symbol Layout Tree (SLT) representations, falling back to LaTeX strings where SLT construction fails. See the previous task overview papers for more details. Our thanks to Frank Tompa for suggesting including formula appearance groups in the provided index files.
Collection and Topics.
The test collection
is built from Mathematics Stack Exchange
, an online Question Answering (QA) site. There were approximately 1.1 million questions on the forum when the main collection was created. Please see the Guidelines document and README files in the collection for additional details. After ARQMath03, over 200 annotated topics are available for each of Task 1 and Task 2,
including qrel files for use in evaluation.
Previously submitted runs from participants are available in the Runs
directory. Please consult the README files in the collection for additional details.
Question Threads (HTML). Within the collection, tools are provided for generating readable question threads from the raw collection snapshot (in Python), along with the question thread HTML pages that the tool produces (provided with collection and topic data mentioned above). Threads are intended for use in studying the collection, sanity checking results during development, and are also used for the relevance assessment (by human assessors) after the submission deadline has passed. Formulas that are indexed for ARQMath are placed in span tags with the class 'math-container' that include the integer identifier for the formula in the ARQMath formula index, e.g.,
<span class="math-container" id="844">...</span>