amp template

Software and Data

Data and source code from our work are provided below. We use licenses that do not restrict non-commercial use. Associated publications may be found here.


SymbolScraper: PDF symbol extractor

(Ritvik Joshi et al., April 2019) SymbolScraper is an Apache PDFBox extension (in Java) that provides exact locations and identities for characters and symbols in PDF. The system was used in Kenny Davila and Ritvik Joshi's ECIR 2019 paper on visual formula search in images.

Approach0 Formula Search Engine

(W. Zhong, Jan.2019) A new formula search engine using paths in operator trees (representing operations in a formula), with support for multiple subexpression matches. Released as a companion to Wei's ECIR 2019 paper describing the system. The systems obtains state-of-the-art results for queries without wildcards in the NTCIR-12 Formula Browsing Task.

  • The ECIR 2019 version of Approach0 is available from GitHub.



AccessMath: Whiteboard Video Summarization

(K. Davila, Nov. 2017) Generating keyframe summaries of lecture videos containing only whiteboard contents. The system works with single-shot recordings of lecture videos. Released as a companion to Kenny's ICDAR 2017 paper on the same work. This work was later used to support keyframe-based video navigation, and cross-modal visual math search (for the Tangent-V (visual) search engine; details: K. Davila's PhD dissertation).

  • Demos: We have one demo for navigating lecture videos using keyframe summaries, and a more detailed demonstration of the Tangent-V visual formula search engine being applied to keyframe summaries. This second demo includes:
    • Examples of keyframe video summaries
    • Video navigation tool that allows traversal by clicking on 'ink' in keyframes
    • Two binary image versions of the video, one with the speaker, and one with the speaker removed. This allows only the whiteboard contents to be viewed throughout the video, for example.
    • Visual, cross-modal formula search. Tangent-V can search formulas within video summary keyframes and lecture notes (in LaTeX), as well as between rendered LaTeX and handwritten formula images in generated keyframes.
  • Source code (analysis + video frame ground truth creation tools): AccessMath_ICDAR_2017_code.zip
  • Video annotations/data (256MB): AccessMath_ICDAR_2017_data.zip
  • Original videos: Video Recordings
  • Extensions (Fall 2018): A newer version of the source code may be found on github. Re-encoded versions of the lecture videos are now available in MP4 format from CUBS at the University at Buffalo.




DPRL Natural Scene Text Detector

(S. Zhu, Apr. 2016) The code below was used to produce the results published in Siyu Zhu's 2016 CVPR paper, A Text Detection System for Natural Scenes with Convolutional Feature Learning and Cascaded Classification, which obtained state-of-the-art results on the ICDAR 2015 Focused Scene Text Detection task at the time of publication.




The Tangent Math Search Engine

  • Tangent-v (ECIR 2019 version). Visual formula search engine for .png (raster) and .pdf (vector) formula images. Results from our related ECIR 2019 paper are included in the package.
  • Tangent-V and AccessMath (July 2018, by K. Davila) This is a generalization of the Tangent formula search engine for more general Visual Search based on appearance alone. This version can be used for indexing and retrieval of binary images. It has been described in our ICFHR 2018 paper. Note: this version also includes code for indexing keyframes in lecture videos that are used for search (from the lab's AccessMath project).
  • Tangent-S (July 2017, by K. Davila, R. Zanibbi, A. Kane and F. Wm. Tompa). This version of the Tangent formula search engine supports individual and parallel search of formula appearance and semantics. This version extends Tangent v. 0.3.1 below, and is described in our SIGIR 2017 paper.
  • Tangent 0.3.1 (May 2016, by K. Davila, R. Zanibbi, A. Kane and F. Wm. Tompa). This is the version described in our NTCIR-12 competition paper, with wildcard support for full subexpressions, and better separation of code for scoring metrics and locating subexpressions with the best match.

  • Tangent 0.3 (July 2015, by R. Zanibbi, K. Davila, A. Kane and F. Wm. Tompa). You can download the source code and sample results (including .html pages with highlighted hits) below. This is the version described in our SIGIR 2016 paper.


  • Tangent 0.2 (2014). Nidhin Pattaniyil implemented this extension of the Tangent system to support matrices and prefix subscripts and superscripts. This updated Tangent combines math expression retrieval with a Solr/Lucene text retrieval system, supporting mixed math and text queries.

    Please Note: the files below are quite large, in part so that others have a better chance to replicate our results at NTCIR-11 (2014; NTCIR-11 paper)


  • Tangent 0.1 demo (2013). A math expression search engine create by David Stalnaker. This online demo searches math expressions in an earlier version of English Wikipedia.




Math Entry and Search Interfaces

  • min multimodal math search interface (2011-2015, demo). Supports mouse/touch, keyboard, mouse and (limited) image input). The program runs on tablets, desktops and laptops.
    • Interface source code: GitHub Page
    • Source code for recognition and other server applications used:
      git clone http://saskatoon.cs.rit.edu:10001/root/min-server-apps.git
    • The handwritten symbol recognizer used by min is available below.
    • The image-based symbol recognizer source code is available from GitHub

  • Freehand Formula Entry System (FFES) and DRACULAE handwritten math parser (1999 - 2007); early pen-based equation editor (last release: Aug. 10, 2007)



CROHME Handwritten Math Recognition Competitions (web page)



US Patent Office (USPTO) Figure and Part Label Detection Competition

  • Paper co-authored by Chris Riedl (Northeastern, former Harvard post-doc), Marti Hearst (UC Berkeley), Siyu Zhu, Richard Zanibbi and researchers from the Harvard-NASA Tournament Lab (Karim Lakhani et al.) describing an online competition for labeling parts in US patent diagram images has been posted on the arXiv.
     
  • The data and source code for the top-5 placing systems in the competition are available through the UCI Machine Learning Repository.



Evaluation Tools

  • LgEval: the Label Graph Evaluation library (by R. Zanibbi and H. Mouchere). The library uses labeled directed graphs to represent results for structural pattern recognition tasks. To obtain the current version, issue the following command using git:
    git clone http://saskatoon.cs.rit.edu:10001/root/lgeval.git
  • CROHMELib translation and file viewing utilities for CROHME InkML/MathML files (by R. Zanibbi and H. Mouchere). To obtain the current version, issue the following command using git:
    git clone http://saskatoon.cs.rit.edu:10001/root/crohmelib.git
  • Earlier overview for CROHMELib and LgEval is available (from CROHME 2013/2014 version of the tools)
     
  • Recognition Strategy Language (version 2.0, implemented in Standard ML; Programmer's Guide to RSL). Ben Holm wrote this code along with a re-implementation of an American Sign Language video interpretation program using OpenCV for his MSc thesis in 2011 (with contributions to the RSL compiler by Matthew Fluet and Richard Zanibbi), and Chris Sasarak made modifications and extensions in 2012-2013. To obtain the source, issue the following commands using git:
    git clone http://saskatoon.cs.rit.edu:10001/root/bholm-thesis.git
    git clone http://saskatoon.cs.rit.edu:10001/root/rsl.git

Address

Room 70-3500, GCCIS
Dept. Computer Science
Rochester Inst. Technology
Rochester, NY, 14623-5608
USA

Contacts

Email: rxzvcs@rit.edu
Phone: +1 (585) 475-4536
Fax: +1 (585) 475-4935