Given a question post with an identified formula as a query, search all question and answer posts and return relevant formulas with their posts.
Above is an example query, with a formula taken from the example search for Task 1 at left, along with formulas with their associated posts (i.e., in-context) returned in search results at right. Relevant formulas are shown in green.
Formulas provided in LaTeX, Presentation MathML, and Content MathML
Manual and automatic runs will be collected.
Evaluation
Top-k formulas from participants + additional manual runs by organizers will be pooled. Assessors can use formula hits + pools from Task 1 to identify similar formulas.
Most topics will be assessed once, some doubly-assessed to check agreement. Assessors include volunteers from teams along with hired assessors.
(Mar 23) Update: we will be using nDCG' (i.e., nDCG after ignoring hits missing from the evaluation pool) to promote fair comparison with systems that do not participate in the task.