Task 2:  Formula Search

Given a question post with an identified formula as a query, search all question and answer posts and return relevant formulas with their posts.

Above is an example query, with a formula taken from the example search for Task 1 at left, along with formulas with their associated posts (i.e., in-context) returned in search results at right. Relevant formulas are shown in green.

Topics and Runs

  • (Apr. 11) Topics for Task 2 are now available.
  • The corpus for both tasks is available online.
  • Formulas provided in LaTeX, Presentation MathML, and Content MathML
  • Manual and automatic runs will be collected.


  • Top-k formulas from participants + additional manual runs by organizers will be pooled. Assessors can use formula hits + pools from Task 1 to identify similar formulas.
  • Most topics will be assessed once, some doubly-assessed to check agreement. Assessors include volunteers from teams along with hired assessors.
  • (Mar 23) Update: we will be using nDCG' (i.e., nDCG after ignoring hits missing from the evaluation pool) to promote fair comparison with systems that do not participate in the task.

Answer Retrieval for Questions on Math