Combining Algorithms for
Recognition and Retrieval of Mathematics
(Sept. 2010  May 2015)
NSF Award Summary (Grant No. IIS1016815)
Project Summary
This project aims to produce new methods for retrieving math in documents, using mathematical expressions as queries (querybyexpression). We are working to develop retrieval tools
that are intuitive to use, both for math experts and (perhaps more importantly) nonexperts. Methods developed for the project might later be adapted for retrieving other nontextual document elements such as chemical diagrams, tables, and figures. Source code and experimental data developed for the project will be made public via the project web site (http://www.cs.rit.edu/~dprl/msearch.html). To promote mathematical literacy, the principal investigator and graduate students working on the project will visit middle schools and talk about the history, recognition and retrieval of mathematical notation.
The project will also provide opportunties for students to obtain research experience, including students in the McNair Scholars program at RIT. The McNair Scholars program seeks to provide research experiences to lowincome, firstgeneration college students that are interested in pursuing doctoral studies.
Project Team
Richard Zanibbi, PhD, Principal Investigator
Lei Hu, PhD Student (Computer Science, RIT), Sept. 2010May 2015
Siyu Zhu, PhD Student (Imaging Science, RIT) Sept. 2011May 2015
Kevin Talmadge, MSc Student (Computer Science, RIT), Summer 2014
Nidhin Pattaniyil, MSc Student (Computer Science, RIT), Spring 2014
Kenny Davila Castellanos, MSc (May, 2013) Jan.  May 2013
Keita Del Valle, MSc Student (HumanComputer Interaction, RIT) Sept. 2012 Dec. 2013
Justin Kelly, NSF REU Student, June 2013  May 2014
Awelemdy Orakwue, NSF REU Student, Sept. 2012  May 2014
Matthias Reichenbach, MSc Student (HumanComputer Interaction, RIT) Sept. 2012May 2013
Robert Li Volsi, MSc Student (Computer Engineering, RIT), Jan. 2011Aug. 2012
Christopher Sasarak, NSF REU Student, Summer 2012, RA Summer 2013, MSc Project Spring 2014 (BSc/Msc Computer Science Student, RIT)
David Stalnaker, BSc/MSc Student (Computer Science, RIT), Dec. 2011Aug 2013
Meridangela Gutierrez Jhong, BSc Student and McNair Scholar (Computer Science), Jan.May 2011
Kevin Hart, NSF REU Summer 2011 (BSc Computer Science, RIT), RA Sept. 2011Feb. 2012
Thomas Schellenberg, MSc Student (Computer Science, RIT), Sept. 2010Nov. 2011
Benjamin Holm, MSc Student (Computer Science, RIT), April 2010Aug. 2011
Richard Pospesel, Research Programmer (Master's in Game Design and Development Student, RIT), Sept. 2010July 2011
Collaborators
Francisco Alvaro (PhD Student, Univ. Valencia, Spain)
Dorothea Blostein (Queen's University at Kingston, Canada)
Houssem Chatbri (PhD Candidate, Univ. Tsukuba, Japan)
Matthew Fluet (RIT)
Keisuke Kameyama (Univ. Tsukuba, Japan)
Andrew Kane (Postdoctoral Fellow, Univ. Waterloo, Canada)
Harold Mouchère (IRCCyN/IVC, Nantes, France)
George Nagy, (Prof. Emeritus, RPI)
Frank Tompa (Univ. Waterloo, Canada)
Christian ViardGaudin (IRCCyN/IVC, Nantes, France)
Bo Yuan (RIT)
Software
The new min math search interface, and the Tangent math search engine (for which expressions from Wikipedia have been indexed) may be used by clicking on their images at the top of this page. Both work with desktop machines and tablets.

 The CROHMELib and LgEval libraries for evaluating structural pattern recognition systems and the CROHME handwritten math competition have been publicly released under a Creative Commons noncommercial license.

The DPRL Lab GitHub Page provides source code for the following.
 Tangent math search engine, versions 0.1 (D. Stalnaker, 2013) and 0.2 (N. Pattaniyil, 2014)
 DPRL lab entries for CROHME 20112014 (all four systems)
 HMMbased (L. Hu, 2011) and offline feature/SVMbased (K. Davila, 2014) handwritten math symbol classifiers
 Imagebased classifier for typeset symbols (S. Zhu and L. Hu, 2013)
 The min multimodal math search interface, supporting mouse/touch, keyboard and image input, with integration for multiple search engines (e.g. DLMF, Wolfram Alpha, Google, and our Tangent search engine for formulas in a Wikipedia snapshot)
Additional software is available from the DPRL software web page (e.g. the SMLbased Recognition Strategy Language, version 0.2) and the ICDAR 2011 tools web page.
Publications
(Note: In June 2015, two additional papers for the project were under review)
2015
Zanibbi, R. and Orakwue, A. (2015) Math Search for the Masses: Multimodal Search Interfaces and AppearanceBased Retrieval. Proc. Conference on Intelligent Computer Mathematics, Washington, DC (to appear, July 2015).
Stalnaker, D. and Zanibbi, R. (2015) Math expression retrieval using an inverted index over
symbol pairs. Proc. Document Recognition and Retrieval, Vol. 9402 of Proc. SPIE, pp. 940207:112,San Francisco.
2014
Pattaniyil, N. and Zanibbi, R. (2014) Combining TFIDF Text Retrieval with an Inverted Index over Symbol Pairs in Math Expressions: The Tangent Math Search Engine at NTCIR 2014. Proc. 11th NII Testbeds and Community for Information access Research (NTCIR), Tokyo, Japan (online).
Reichenbach, M.S., Agarwal, A. and Zanibbi, R. (2014) Rendering expressions to improve accuracy of relevance assessment for math search. Proc. ACM SIGIR, Gold Coast, Australia, pp. 851854.
Del Valle Wangari, K., Zanibbi, R. and Agarwal, A. (2014) Discovering realworld use cases for a multimodal math search interface. Proc. ACM SIGIR, Gold Coast, Australia, pp. 947950.
Mouchere, H., ViardGaudin, C., Zanibbi, R. and Garain, U. (2014) ICFHR 2014 Competition on Recognition of Online Handwritten Mathematical Expressions (CROHME 2014). Proc. Int'l Conf. Frontiers in Handwriting Recognition, pp. 791796, Crete, Greece.
D. Blostein and R. Zanibbi. (2014) Processing Mathematical Notation, in Handbook of Document Image Processing and Recognition, pp. 679702, SpringerVerlag.
2013
F. Alvaro and R. Zanibbi (2013) A ShapeBased Layout Descriptor for Classifying Spatial Relationships in Handwritten Math. ACM Symp. Document Engineering, Florence, pp. 123126, Italy.
H. Mouchere, C. ViardGaudin, R. Zanibbi, U. Garain, D.H. Kim and J.H. Kim (2013) ICDAR 2013 CROHME: Third International Competition on Recognition of Online Handwritten Mathematical Expressions. Proc. Int'l Conf. Document Analysis and Recognition, Washington, DC, 14281432.
S. Zhu and R. Zanibbi (2013) Label Detection and Recognition for USPTO Images using Convolutional Kmeans Feature Quantization and AdaBoost. Proc. Int'l Conf. Document Analysis and Recognition, pp. 633637, Washington, DC.
L. Hu and R. Zanibbi (2013) Segmenting Handwritten Math Symbols Using AdaBoost and MultiScale Shape Context Features. Proc. Int'l Conf. Document Analysis and Recognition, pp. 11801184, Washington, DC.
R. Zanibbi, H. Mouchere, and C. ViardGaudin (2013) Evaluating Structural Pattern Recognition for Handwritten Math via Primitive Label Graphs Proc. Document Recognition and Retrieval, Proc. SPIE vol. 8658, pp. 171  1711, San Francisco, CA.
S. Zhu, L. Hu and R. Zanibbi (2013) RotationRobust Math Symbol Recognition and Retrieval Using Outer Contours and Image Subsampling Proc. Document Recognition and Retrieval, Proc. SPIE vol. 8658, pp. 051  0512, San Francisco, CA.
2012
C. Sasarak, K. Hart, R. Pospesel, D. Stalnaker, L. Hu, R. LiVolsi, S. Zhu, and R. Zanibbi. (2012) min: A Multimodal Web Interface for Math Search. Symp. HumanComputer Interaction and Information Retrieval, Cambridge, MA (online, 4pp).
L. Hu, K. Hart, R. Pospesel, and R. Zanibbi. (2012) Baseline extractiondriven parsing of handwritten mathematical expressions Proc. Int'l Conf. Pattern Recognition, Tsukuba Science City, Japan.
R. Zanibbi and D. Blostein (2012) Recognition and Retrieval of Mathematical Expressions, Int'l. Journal on Document Analysis and Recognition 15(4): 331357. (original publication available from www.springerlink.com).
T. Schellenberg, B. Yuan and R. Zanibbi (2012). Layoutbased substitution tree indexing and retrieval for mathematical expressions, Proc. Document Recognition and Retrieval XIX, pp. 8297OI1  8297OI8, San Francisco.
2011
R. Zanibbi and B. Yuan. (2011) Keyword and imagebased retrieval of mathematical expressions. Proc. Document Recognition and Retrieval XVIII, vol. 7874 Proc. SPIE, pp. OI1OI9, San Francisco, CA.
R. Zanibbi and L. Yu. (2011) Math Spotting: Retrieving Math in Technical Documents Using Handwritten Query Images. Proc. Int'l Conf. Document Analysis and Recognition, pp. 446451, Beijing.
L. Hu and R. Zanibbi. (2011) HMMBased Recognition of Online Handwritten Mathematical Symbols Using Segmental Kmeans Initialization and a Modified Penup/down Feature. Proc. Int'l. Conf. Document Analysis and Recognition, pp. 457462, Beijing.
R. Zanibbi, A. Pillay, H. Mouchere, C. ViardGaudin, and D. Blostein. (2011) StrokeBased Performance Metrics for Handwritten Mathematical Expressions. Int'l Conf. Document Analysis and Recognition, pp. 334338, Beijing.
News (Sept. 2010  May 2015)
2014
 (Dec.) Prof. Zanibbi will be an invited speaker at the 2015 Conferences on Intelligent Computer Mathematics (CICM).
 (Dec.) Nidhin Pattaniyil has received a travel award from the NTCIR11 conference organizers to present his work in Tokyo.
 (Oct.) David Stalnker's paper Math expression retrieval using an inverted index over symbol pairs has been accepted for oral presentation at Document Recognition and Retrieval XXII (2015) in San Francisco.
 (Oct.) Nidhin Pattaniyil extended David's work for his MSc project (paper), adding support for matrices and combining expression retrieval with text search to create an entry for the international NTCIR Math Retrieval Competition. The competition was held for the upcoming NTCIR11 conference in Japan. David and Nidhin's system performed best for retrieving Wikipedia expressions, and had the best overall top5 results and secondhighest 'highly relevant' top5 results for combined text and math queries for 100,000 documents from the arXiv.
 (Sept.) Prof. Zanibbi gave a talk in the HumanComputer Interaction Seminar at the University of Waterloo on Sept. 26th, entitled "Creating UserFriendly Math Search Engines." The talk covered search interface designs and math retrieval algorithms developed in the DPRL. Our thanks to Dr. Edward Lank for the invitation to give this talk.
 (Sept.) The successful ICFHR 2018 Rochester bid has been announced on the GCCIS Web Pages.
 (Sept.) Prof. Zanibbi submitted a successful bid to host the 2018 International Conference on Frontiers in Handwriting Recognition in Rochester, at RIT. The bid is available here, and the presentation slides (presented by Umapada Pal at ICFHR 2014 in Crete) are available here. If you work in handwriting recognition, we hope to see you in 2018!
 (Aug.) Kevin Talmadge has succesfully completed his Master's Project on applying the Recognition Strategy Library to analyzing a math recognition system.
 (June) As part of an Image and document processing, recognition and interaction seminar held at IUT/Univ. Nantes (France, on June 30), Prof. Zanibbi gave a talk on human math retrieval studies and math search engines (work done with Keita Wangari, Matthias Reichenbach, David Stalnaker and Nidhin Pattaniyil.) Other presenters were researchers from Australia, Brazil, and China. (program; slides)
 (June) Amit Pillay has successfully defended his MSc thesis, Intelligent Combination of Structural Analysis Algorithms: Application to Mathematical Expression Recognition, which presents a method for combining DRACULAE math notation parsers using a Graph Transformer Network (GTN). This technique was successful in increasing spatial relationship detection accuracy.
 (May) Congratulations to Kenny Davila, who has succesfully completed his Research Potential Assessment, which is required for PhD students at the end of their first year.
 (May) The lab welcomes Kevin Talmadge, who will be working on the new Pythonbased Recognition Strategy Library for his MSc Project in the summer.
 (May) Congratulations to Zack Miller, Nidhin Pattaniyil and Christopher Sasarak, who have all successfully completed their MS Projects (their presentation posters are available from the "Publications" link above).
 (Apr.) Kenny Davila's paper describing a stateoftheart classifier for handwritten mathematical symbols has been accepted for oral presentation at ICFHR 2014 being held in Greece this coming September.
 (Apr.) Keita Del Valle's paper describing a user study of the min math entry and search interface has been accepted for publication at ACM SIGIR 2014. SIGIR is being held in the Gold Coast, Australia in July.
 (Apr.) Matthias Reichenbach's paper on the effect of rendering math expression in search hits has been accepted for publication at ACM SIGIR 2014.
 (Mar.) The lab has released a number of systems for math recognition and retrieval as open source on GitHub (DPRL GitHub Page). We hope to add some more information to these pages soon.
 (Mar.) Kenny Davila has released his collection of isolated handwritten math symbol classifiers under GNU GPL v.3 (available here). In benchmarking experiments, the best classifiers (using an SVM with RBF kernel) have recognition rates that match or exceed the best published rates for the MathBrush handwritten symbol data set.
 (Mar.) As part of his upcoming sabbatical, Prof. Zanibbi will be a Visiting Professor at IRCCyN/IVC (Nantes, France) during June and July, and at the University of Waterloo Computer Science Department (Canada) in October.
2013
 (Aug 2013) David Stalnaker has successfully defended his MSc thesis, Math Expression Retrieval Using Symbols Pairs in Layout Trees.
 (July 2013) Christopher Sasarak will be giving a presentation on his work with recognizing ASL in video using the Recognition Strategy Language at the Undergraduate Research Symposium on Friday August 2nd at 2:15pm (at the RIT Inn).
 (June 2013) Lei Hu and Siyu Zhu will be participating in the Doctoral Consortium at the International Conference on Document Analysis and Recognition in August.
 (June 2013) The lab welcomes Justin Kelly to the lab. Justin is a BSc (Computer Science) NSF REU student who is working over the summer on recognizing math in images.
 (June 2013) The lab welcomes Dr. Harold Mouchere from Univ. Nantes, France, who is visiting the lab and collaborating with Prof. Zanibbi during the month of June.
 (May 2013) Congratulations to Matthias Reichenbach, who has successfully defended his MSc project on the effect of rendering math expressions on relevance assessment for math search.
 (May 2013) Congratulations to Kenny Davila Castellanos on successfully defending his MSc project on imagebased search of whiteboard notes in math lecture videos. Kenny will start his PhD in the DPRL for the NSFfunded AccessMath project this summer, cosupervised by Prof. Zanibbi and Prof. Ludi.
 (May 2013) Prof. Zanibbi has been awarded an NSF REU grant, which will be used to fund undergraduate researchers in the DPRL lab during summer 2013.
 (May 2013) ImagineRIT will be held on Saturday, May 4. The lab will be demonstrating the
${m}_{\mathrm{in}}$ math search interface in the atrium of the Golisano college.
 (April 2013) Congratulations to Matthias Reichenbach, who has secured a position with Square in San Francisco.
 (Mar. 2013) Prof. Zanibbi gave a talk on the dprl's work in math recognition and retrieval Creating UserFriendly Systems for Math Search as part of the RIT Imaging Science Seminar Series.
 (Feb. 2013) CROHMELib v. 0.1 and LgEval v. 0.2 have been released.
 (Jan. 2013) The
${m}_{\mathrm{in}}$ math search interface will be demonstrated at the 20th Document Recognition and Retrieval Conference being held in San Francisco Feb. 57th.
 (Jan. 2013) The 3rd international Competition on Recognition of Handwritten Mathematical Expressions (CROHME) has begun! Data and tools are available. The registration deadline is Feb. 15th.
 (Jan. 2013) The lab welcomes Francisco Alvaro, a PhD student from the University of Valencia, Spain, who will visit the DPRL between January and May of this year, as he carries out his research in mathematical notation recognition.
 (Jan. 2013) The LgEval library (a python library for evaluating structural pattern recognition algorithms by Profs. Zanibbi and Mouchere) has been posted on the Software page.
2012
 (Dec. 2012) The lab welcomes Alex Canter to the lab, who will be looking at applying OCR to text extracted from natural scenes.
 (Nov. 2012) Siyu Zhu and Lei Hu have received RIT Graduate Research and Creativity Awards ($500 each), to be used in support of their travel to DRR in San Francisco, and ICPR in Japan, respectively.
 (Sept 2012) The lab welcome Matthias Reichenbach to the lab. Matthias is an MSc in HCI student, who will be completing a project on human relevance assessments for math search.
 (Sept 2012) Christopher Sasarak has received a student travel award in support of his poster presentation at HCIR 2012.
 (Sept 2012) The lab welcomes Awelemdy Orakwue (NSF REU Student) and Keita Del Valle (MSc Student, HumanComputer Interaction), both of whom will be working on HCI aspects of math recognition and retrieval.
 (Sept 2012) David Stalnaker has been offered a position with Google.
 (June 2012) Prof. Zanibbi will be giving a talk entitled Some Recent Developments in Math Retrieval at the Université de Nantes, France on June 14.
 (June 2012) Prof. Zanibbi is spending the month of June as a Visiting Professor at the IRCCyN/IVC research center located within Polytech Nantes, France.
 (May 2012) The DPRL lab will be demonstrating the min math search interface at ImagineRIT on Saturday, May 5.
 (April 2012) The lab welcomes Christopher Sasarak. Christopher will be working on the min interface and math recognition algorithms over the summer as an NSF REU student.
 (March 2012) Prof. Zanibbi gave a seminar on current math recognition and retrieval research in the DPRL at Lehigh University.
 (Jan 2012) The lab welcomes Meridangela Gutierrez Jhong. Meridangela is an undergraduate research assistant, who will be carrying out work on math retrieval in LaTeX documents.
2011
 (Dec 2011) The lab welcomes David Stalnaker, a BSc/MSc student in Computer Science who will be working as a research assistant on the project.
 (Dec 2011) The lab welcomes Robert LiVolsi, a BSc/MSc student in Computer Engineering, who will be working on machine learning algorithms for handwritten math recognition.
 (Nov 2011) Thomas Schellenberg has successfully defended his MSc thesis, LayoutBased Substitution Tree Indexing and Retrieval for Mathematical Expressions.

(Oct/Nov 2011) Prof. Zanibbi gave talks on math recognition and retrieval research in the DPRL at the Rochester chapter meetings for the SPIE and IEEE Computer Society (slides are available from the IEEE web page).
 (Sept 2011) The lab welcomes Siyu Zhu to the lab. Siyu is a PhD student in Imaging Science, who will be working on math retrieval using images.
 (Sept 2011) Thomas Schellenberg, Prof. Yuan and Prof. Zanibbi's paper Layoutbased substitution tree indexing and retrieval for mathematical expresions has been accepted for presentation at Document Recognition and Retrieval 2012.
 (Sept 2011) Lei Hu and Prof. Zanibbi presented research papers on HMMbased math symbol recognition and imagebased math retrieval, along with a poster describing a new metric for evaluating math recognition systems at ICDAR 2011 in Beijing. The metric was developed in collaboration with researchers from Université de Nantes (France) and Queen's University (Canada).
 (Aug 2011) Ben Holm has successfully defended his MSc thesis, Evaluation of RSL History as a Tool for Assistance in the Development and Evaluation of Computer Vision Algorithms.
 (July 2011) Prof. Zanibbi and Prof. Dorothea Blostein's survey on recognition and retrieval of mathematical notation has been accepted for publication in the International Journal on Document Analysis and Recognition.
 (May 2011) A first prototype of a webbased system for pen/fingerbased math entry will be presented at ImagineRIT on May 7th.
 (April 2011) We welcome Kevin Hart, who will
be working on the project as part of an NSF Research Experience for Undergraduates (NSF REU) project.
Last updated: June 4, 2015