Task 3 ( New! ):  Open Domain Question Answering

Given a posted question as a query, return a single answer. The answer may be automatically generated, and may contain passages from outside the ARQMath collection.

Above is an example query (question post) at left, with a single answer returned at right.


  • Participating systems may use any knowledge sources to train systems and generate responses, except for Math Stack Exchange (MSE) data from 2019 to the present. 
  • Answers may produced using (1) extraction, i.e., passages from sources (including the ARQMath MSE collection from 2010-2018), (2) generated text, or (3) a combination of both.
  • Queries will be identical to those used for Task 1.
  • Manual and automatic runs will be collected. See the 'Guidelines' document for details.
  • For relevance assessment, submitted answers will be included in the same pools as responses for Task 1. We will prefer primary runs if there is a large number of submissions.
  • As part of the task, we will compare human relevance assessments against automated answer quality measures (e.g., such as BLEU).