Combining Algorithms for
Recognition and Retrieval of Mathematics
(Sept. 2010 - May 2015)

NSF Award Summary (Grant No. IIS-1016815)


Project Summary

This project aims to produce new methods for retrieving math in documents, using mathematical expressions as queries (query-by-expression). We are working to develop retrieval tools that are intuitive to use, both for math experts and (perhaps more importantly) non-experts. Methods developed for the project might later be adapted for retrieving other non-textual document elements such as chemical diagrams, tables, and figures. Source code and experimental data developed for the project will be made public via the project web site ( To promote mathematical literacy, the principal investigator and graduate students working on the project will visit middle schools and talk about the history, recognition and retrieval of mathematical notation. The project will also provide opportunties for students to obtain research experience, including students in the McNair Scholars program at RIT. The McNair Scholars program seeks to provide research experiences to low-income, first-generation college students that are interested in pursuing doctoral studies.

Project Team

Richard Zanibbi, PhD, Principal Investigator
Lei Hu, PhD Student (Computer Science, RIT), Sept. 2010-May 2015
Siyu Zhu, PhD Student (Imaging Science, RIT) Sept. 2011-May 2015
Kevin Talmadge, MSc Student (Computer Science, RIT), Summer 2014
Nidhin Pattaniyil, MSc Student (Computer Science, RIT), Spring 2014
Kenny Davila Castellanos, MSc (May, 2013) Jan. - May 2013
Keita Del Valle, MSc Student (Human-Computer Interaction, RIT) Sept. 2012- Dec. 2013
Justin Kelly, NSF REU Student, June 2013 - May 2014
Awelemdy Orakwue, NSF REU Student, Sept. 2012 - May 2014
Matthias Reichenbach, MSc Student (Human-Computer Interaction, RIT) Sept. 2012-May 2013
Robert Li Volsi, MSc Student (Computer Engineering, RIT), Jan. 2011-Aug. 2012
Christopher Sasarak, NSF REU Student, Summer 2012, RA Summer 2013, MSc Project Spring 2014 (BSc/Msc Computer Science Student, RIT)
David Stalnaker, BSc/MSc Student (Computer Science, RIT), Dec. 2011-Aug 2013
Meridangela Gutierrez Jhong, BSc Student and McNair Scholar (Computer Science), Jan.-May 2011
Kevin Hart, NSF REU Summer 2011 (BSc Computer Science, RIT), RA Sept. 2011-Feb. 2012
Thomas Schellenberg, MSc Student (Computer Science, RIT), Sept. 2010-Nov. 2011
Benjamin Holm, MSc Student (Computer Science, RIT), April 2010-Aug. 2011
Richard Pospesel, Research Programmer (Master's in Game Design and Development Student, RIT), Sept. 2010-July 2011


Francisco Alvaro (PhD Student, Univ. Valencia, Spain)
Dorothea Blostein (Queen's University at Kingston, Canada)
Houssem Chatbri (PhD Candidate, Univ. Tsukuba, Japan)
Matthew Fluet (RIT)
Keisuke Kameyama (Univ. Tsukuba, Japan)
Andrew Kane (Postdoctoral Fellow, Univ. Waterloo, Canada)
Harold Mouchère (IRCCyN/IVC, Nantes, France)
George Nagy, (Prof. Emeritus, RPI)
Frank Tompa (Univ. Waterloo, Canada)
Christian Viard-Gaudin (IRCCyN/IVC, Nantes, France)
Bo Yuan (RIT)



(Note: In June 2015, two additional papers for the project were under review)


Zanibbi, R. and Orakwue, A. (2015) Math Search for the Masses: Multimodal Search Interfaces and Appearance-Based Retrieval. Proc. Conference on Intelligent Computer Mathematics, Washington, DC (to appear, July 2015).

Stalnaker, D. and Zanibbi, R. (2015) Math expression retrieval using an inverted index over symbol pairs. Proc. Document Recognition and Retrieval, Vol. 9402 of Proc. SPIE, pp. 940207:1-12,San Francisco.


Pattaniyil, N. and Zanibbi, R. (2014) Combining TF-IDF Text Retrieval with an Inverted Index over Symbol Pairs in Math Expressions: The Tangent Math Search Engine at NTCIR 2014. Proc. 11th NII Testbeds and Community for Information access Research (NTCIR), Tokyo, Japan (online).

Reichenbach, M.S., Agarwal, A. and Zanibbi, R. (2014) Rendering expressions to improve accuracy of relevance assessment for math search. Proc. ACM SIGIR, Gold Coast, Australia, pp. 851-854.

Del Valle Wangari, K., Zanibbi, R. and Agarwal, A. (2014) Discovering real-world use cases for a multimodal math search interface. Proc. ACM SIGIR, Gold Coast, Australia, pp. 947-950.

Mouchere, H., Viard-Gaudin, C., Zanibbi, R. and Garain, U. (2014) ICFHR 2014 Competition on Recognition of On-line Handwritten Mathematical Expressions (CROHME 2014). Proc. Int'l Conf. Frontiers in Handwriting Recognition, pp. 791-796, Crete, Greece.

D. Blostein and R. Zanibbi. (2014) Processing Mathematical Notation, in Handbook of Document Image Processing and Recognition, pp. 679-702, Springer-Verlag.


F. Alvaro and R. Zanibbi (2013) A Shape-Based Layout Descriptor for Classifying Spatial Relationships in Handwritten Math. ACM Symp. Document Engineering, Florence, pp. 123-126, Italy.

H. Mouchere, C. Viard-Gaudin, R. Zanibbi, U. Garain, D.H. Kim and J.H. Kim (2013) ICDAR 2013 CROHME: Third International Competition on Recognition of Online Handwritten Mathematical Expressions. Proc. Int'l Conf. Document Analysis and Recognition, Washington, DC, 1428-1432.

S. Zhu and R. Zanibbi (2013) Label Detection and Recognition for USPTO Images using Convolutional K-means Feature Quantization and AdaBoost. Proc. Int'l Conf. Document Analysis and Recognition, pp. 633-637, Washington, DC.

L. Hu and R. Zanibbi (2013) Segmenting Handwritten Math Symbols Using AdaBoost and Multi-Scale Shape Context Features. Proc. Int'l Conf. Document Analysis and Recognition, pp. 1180-1184, Washington, DC.

R. Zanibbi, H. Mouchere, and C. Viard-Gaudin (2013) Evaluating Structural Pattern Recognition for Handwritten Math via Primitive Label Graphs Proc. Document Recognition and Retrieval, Proc. SPIE vol. 8658, pp. 17-1 - 17-11, San Francisco, CA.

S. Zhu, L. Hu and R. Zanibbi (2013) Rotation-Robust Math Symbol Recognition and Retrieval Using Outer Contours and Image Subsampling Proc. Document Recognition and Retrieval, Proc. SPIE vol. 8658, pp. 05-1 - 05-12, San Francisco, CA.


C. Sasarak, K. Hart, R. Pospesel, D. Stalnaker, L. Hu, R. LiVolsi, S. Zhu, and R. Zanibbi. (2012) min: A Multimodal Web Interface for Math Search. Symp. Human-Computer Interaction and Information Retrieval, Cambridge, MA (online, 4pp).

L. Hu, K. Hart, R. Pospesel, and R. Zanibbi. (2012) Baseline extraction-driven parsing of handwritten mathematical expressions Proc. Int'l Conf. Pattern Recognition, Tsukuba Science City, Japan.

R. Zanibbi and D. Blostein (2012) Recognition and Retrieval of Mathematical Expressions, Int'l. Journal on Document Analysis and Recognition 15(4): 331-357. (original publication available from

T. Schellenberg, B. Yuan and R. Zanibbi (2012). Layout-based substitution tree indexing and retrieval for mathematical expressions, Proc. Document Recognition and Retrieval XIX, pp. 8297OI-1 - 8297OI-8, San Francisco.


R. Zanibbi and B. Yuan. (2011) Keyword and image-based retrieval of mathematical expressions. Proc. Document Recognition and Retrieval XVIII, vol. 7874 Proc. SPIE, pp. OI1-OI9, San Francisco, CA.

R. Zanibbi and L. Yu. (2011) Math Spotting: Retrieving Math in Technical Documents Using Handwritten Query Images. Proc. Int'l Conf. Document Analysis and Recognition, pp. 446-451, Beijing.

L. Hu and R. Zanibbi. (2011) HMM-Based Recognition of Online Handwritten Mathematical Symbols Using Segmental K-means Initialization and a Modified Pen-up/down Feature. Proc. Int'l. Conf. Document Analysis and Recognition, pp. 457-462, Beijing.

R. Zanibbi, A. Pillay, H. Mouchere, C. Viard-Gaudin, and D. Blostein. (2011) Stroke-Based Performance Metrics for Handwritten Mathematical Expressions. Int'l Conf. Document Analysis and Recognition, pp. 334-338, Beijing.

News (Sept. 2010 - May 2015)

Last updated: June 4, 2015