Project Summary

This project aims to produce new methods for retrieving math in documents, using mathematical expressions as queries (query-by-expression). We are working to develop retrieval tools that are intuitive to use, both for math experts and (perhaps more importantly) non-experts. Methods developed for the project might later be adapted for retrieving other non-textual document elements such as chemical diagrams, tables, and figures. Source code and experimental data developed for the project will be made public via the project web site (http://www.cs.rit.edu/~dprl/msearch.html). To promote mathematical literacy, the principal investigator and graduate students working on the project will visit middle schools and talk about the history, recognition and retrieval of mathematical notation. The project will also provide opportunties for students to obtain research experience, including students in the McNair Scholars program at RIT. The McNair Scholars program seeks to provide research experiences to low-income, first-generation college students that are interested in pursuing doctoral studies.

News (2011-2012)

• (June 2012) Prof. Zanibbi will be giving a talk entitled Some Recent Developments in Math Retrieval at the Université de Nantes, France on June 14.
• (June 2012) Prof. Zanibbi is spending the month of June as a Visiting Professor at the IRCCyN/IVC research center located within Polytech Nantes, France.
• (May 2012) The DPRL lab will be demonstrating the min math search interface at ImagineRIT on Saturday, May 5.
• (April 2012) The lab welcomes Christopher Sasarak. Christopher will be working on the min interface and math recognition algorithms over the summer as an NSF REU student.
• (March 2012) Prof. Zanibbi gave a seminar on current math recognition and retrieval research in the DPRL at Lehigh University.
• (Jan 2012) The lab welcomes Meridangela Gutierrez Jhong. Meridangela is an undergraduate research assistant, who will be carrying out work on math retrieval in LaTeX documents.
• (Dec 2011) The lab welcomes David Stalnaker, a BSc/MSc student in Computer Science who will be working as a research assistant on the project.
• (Dec 2011) The lab welcomes Robert LiVolsi, a BSc/MSc student in Computer Engineering, who will be working on machine learning algorithms for handwritten math recognition.
• (Nov 2011) Thomas Schellenberg has successfully defended his MSc thesis, Layout-Based Substitution Tree Indexing and Retrieval for Mathematical Expressions.
• (Oct/Nov 2011) Prof. Zanibbi gave talks on math recognition and retrieval research in the DPRL at the Rochester chapter meetings for the SPIE and IEEE Computer Society (slides are available from the IEEE web page).
• (Sept 2011) The lab welcomes Siyu Zhu to the lab. Siyu is a PhD student in Imaging Science, who will be working on math retrieval using images.
• (Sept 2011) Thomas Schellenberg, Prof. Yuan and Prof. Zanibbi's paper Layout-based substitution tree indexing and retrieval for mathematical expresions has been accepted for presentation at Document Recognition and Retrieval 2012.
• (Sept 2011) Lei Hu and Prof. Zanibbi presented research papers on HMM-based math symbol recognition and image-based math retrieval, along with a poster describing a new metric for evaluating math recognition systems at ICDAR 2011 in Beijing. The metric was developed in collaboration with researchers from Université de Nantes (France) and Queen's University (Canada).
• (Aug 2011) Ben Holm has successfully defended his MSc thesis, Evaluation of RSL History as a Tool for Assistance in the Development and Evaluation of Computer Vision Algorithms.
• (July 2011) Prof. Zanibbi and Prof. Dorothea Blostein's survey on recognition and retrieval of mathematical notation has been accepted for publication in the International Journal on Document Analysis and Recognition.
• (May 2011) A first prototype of a web-based system for pen/finger-based math entry will be presented at ImagineRIT on May 7th.
• (April 2011) We welcome Kevin Hart, who will be working on the project as part of an NSF Research Experience for Undergraduates (NSF REU) project.

Project Team

Richard Zanibbi, PhD, Principal Investigator
Lei Hu, PhD Student (Computer Science, RIT), Sept. 2010-
Siyu Zhu, PhD Student (Imaging Science, RIT) Sept. 2011-
Robert Li Volsi, MSc Student (Computer Engineering, RIT), Jan. 2011-
Christopher Sasarak, NSF REU Student, Summer 2012 (BSc Computer Science Student, RIT)
David Stalnaker, BSc/MSc Student (Computer Science, RIT), Dec. 2011-May 2012
Meridangela Gutierrez Jhong, BSc Student and McNair Scholar (Computer Science), Jan.-May 2011
Kevin Hart, NSF REU Summer 2011 (BSc Computer Science, RIT), RA Sept. 2011-Feb. 2012
Thomas Schellenberg, MSc Student (Computer Science, RIT), Sept. 2010-Nov. 2011
Benjamin Holm, MSc Student (Computer Science, RIT), April 2010-Aug. 2011
Richard Pospesel, Research Programmer (Master's in Game Design and Development Student, RIT), Sept. 2010-July 2011

Collaborators

Dorothea Blostein (Queen's University at Kingston, Canada)
Matthew Fluet (RIT)
Harold Mouchère (IRCCyN/IVC, Nantes, France)
George Nagy, (Prof. Emeritus, RPI)
Christian Viard-Gaudin (IRCCyN/IVC, Nantes, France)
Bo Yuan (RIT)

Software

Please Note: the software below is currently under active development.

Web-based math entry interface (click image at left; works best with Firefox 4; iPad compatiable)

• Additional tools and software for the ICDAR 2011 online math entry recognition competition

• A number of other pieces of software have been created for the project, many of them under active development. We intend to make (useful) portions of this software publicly available as time permits.

Publications

T. Schellenberg, B. Yuan and R. Zanibbi (2012). Layout-based substitution tree indexing and retrieval for mathematical expressions, Proc. Document Recognition and Retrieval XIX, pp. 8297OI-1 - 8297OI-8, San Francisco.

R. Zanibbi and B. Yuan. (2011) Keyword and image-based retrieval of mathematical expressions. Proc. Document Recognition and Retrieval XVIII, vol. 7874 Proc. SPIE, pp. OI1-OI9, San Francisco, CA.

R. Zanibbi and L. Yu. (2011) Math Spotting: Retrieving Math in Technical Documents Using Handwritten Query Images. Proc. Int'l Conf. Document Analysis and Recognition, pp. 446-451, Beijing.

Zanibbi, R. and Blostein, D. (2011) Recognition and Retrieval of Mathematical Notation, International Journal of Document Analysis and Recognition, to appear, available online.

L. Hu and R. Zanibbi. (2011) HMM-Based Recognition of Online Handwritten Mathematical Symbols Using Segmental K-means Initialization and a Modified Pen-up/down Feature. Proc. Int'l. Conf. Document Analysis and Recognition, pp. 457-462, Beijing.

R. Zanibbi, A. Pillay, H. Mouchere, C. Viard-Gaudin, and D. Blostein. (2011) Stroke-Based Performance Metrics for Handwritten Mathematical Expressions. Int'l Conf. Document Analysis and Recognition, pp. 334-338, Beijing.