MathDeck
MathDeck is a first-of-its-kind search interface designed to make it easier to edit, reuse, annotate, share, and search formulas. The system is based around a formula 'chip and card' metaphor inspired by board games. Recently the system was extended to support search of formulas extracted from PDF files in the ACL Anthology, with matches shown in-place within search results.
(funded by NSF & Alfred P. Sloan Foundation)
SIGIR 2023 (demo page - poster - video); CHI 2021 demo video; ECIR 2020 demo video - online system (Google Chrome strongly recommended)
Formula Search Engines
We have created a number of state-of-the-art formula search engines, including models based on graphs representing operations and symbol placement on writing lines (Tangent-S and Approach0), on symbol locations in rendered formulas (PHOC), and on vectors produced by neural networks (Tangent-CFT ).
(Funded by NSF and the Alfred P. Sloan Foundation)
ARQMath
(Video overview ECIR 2021) (ARQMath web page)
ARQMath is a cooperative evaluation exercise aiming to advance math-aware search and the semantic analysis of mathematical notation and texts. The lab has been run for three years, for CLEF 2020, 2021, and 2022.
ARQMath has two main tasks, one for answer retrieval, and one for formula retrieval. In the third year a task for open-domain question answering was added, where answers may come from any source, or may be generated (e.g., by GPT-3). The ARQMath collection has become the largest collection of its kind, with over 200 search topics annotated for the answer retrieval and formula retrieval tasks.
(funded by NSF & Alfred P. Sloan Foundation)
AccessMath
AccessMath tracks whiteboard contents in lecture videos through time and generates keyframes. 'Ink' in keyframes can be used to jump to where that 'ink' is drawn in the video, or searched using image queries through the Tangent-V search engine. (NSF-funded project - Video Demo)
min
The first multi-modal equation editing prototype, with support for handwriting, typing, and image input for formulas, along with support for math + keyword search.
(NSF-funded project; Video Demo)
Audio Search in Lectures
A prototype that supports searching lecture videos for spoken keyterms (within-speaker), using a modified Dynamic Time Warping algorithm.
(NSF-funded project; Example Results)
Video CAPTCHAs
Back in 2008, Kurt Kluever created the first video captcha, where users enter three words describing a video.
(Funded by Xerox Corporation)
CROHME
The dprl has participated in every CROHME competition since its inception in 2011, and co-organized the competition since 2013. CROHME has become a standard benchmark for handwritten formula recognition. The last CROHME the lab helped organize was held in 2019, adding a typeset formula detection task. (CROHME 2023 was next held for ICDAR 2023)
ChemScraper
ChemScraper is an online tool for extracting molecules from PDFs. Unlike previous molecule extraction tools, it locates and recognizes diagrams that were created with vector graphics (i.e., explicit drawing instructions in the PDF) or images, and generates CDXML (ChemDraw) and SMILES files as output.
The first 'alpha' version was released as part of the AlphaSynthesis platform in June 2023, and we have continued to improve the system. ChemScraper was created through a collaborative effort between the dprl, the Denmark Lab at UIUC, and NCSA. (ICDAR 2024 journal paper preprint -- tutorial video by Ayush Kumar Shah)
(funded by NSF through the MMLI AI Center)
ReactionMiner
ReactionMiner searches reaction descriptions in PDF documents using text and chemical structures provided as SMILES/SMARTS strings. Extracted reaction records associated with passages can also be browsed and exported. This tool combines LLM-based reaction extraction tools with our lab's ChemScaper tools, linking molecule names in text with their appearance in diagrams, and mutli-modal search for molecules and/or text. Users are able to upload their own PDF for extraction and search.
This is one of the newer AlphaSynthesis tools released in late Spring 2025. This tool was created through a collaborative effort between the dprl, the Han Lab and Jackson Lab at UIUC, and NCSA. (SIGIR 2025 demo paper and demo web page (*online system is newer); tutorial video by Leo Luo)
(funded by NSF through the MMLI AI Center)
UniChemFinder
UniChemFinder is a multi-modal text + diagram passage retrieval tool for chemical patents. The first version of the system supports queries combining text and molecules represented in SMILES strings. The demo searches a collection of 131 patents for compounds used to bind and inhibit specific human genes.
A simple but innovative text/diagram linking technique allows users to browse and navigate through passages and associated diagrams together. These passages and diagrams may be hundreds of pages apart in the original patent.
This tool was developed in collaboration with Nathaniel Stanley at Insitro, a drug discovery company.
(SIGIR 2025 demo paper -- video demonstration by Abhisek Dey)
(funded through the NSF MMLI AI Center)
