Room 70-3500 (Golisano College) Department of Computer Science Rochester Institute of TechnologyPhone: (585) 475-4536

 [ Home ] [ News ] [ Members ] [ Projects ] [ Publications ] [ Software ] [ Support ]

### Software: Demos and Source Code

Please Note: whenever possible, we use licenses that impose no restrictions for non-commercial use of our systems.

#### AccessMath: Whiteboard Video Summarization

(K. Davila, Nov. 2017) Generating keyframe summaries of lecture videos containing only whiteboard contents. The system works with single-shot recordings of lecture videos. Released as a companion to Kenny's ICDAR 2017 paper on the same work. This work was later used to support keyframe-based video navigation, and cross-modal visual math search (for the Tangent-V (visual) search engine; details: K. Davila's PhD dissertation).

• Demos: We have one demo for navigating lecture videos using keyframe summaries, and a more detailed demonstration of the Tangent-V visual formula search engine being applied to keyframe summaries. This second demo includes:
• Examples of keyframe video summaries
• Video navigation tool that allows traversal by clicking on 'ink' in keyframes
• Two binary image versions of the video, one with the speaker, and one with the speaker removed. This allows only the whiteboard contents to be viewed throughout the video, for example.
• Visual, cross-modal formula search. Tangent-V can search formulas within video summary keyframes and lecture notes (in LaTeX), as well as between rendered LaTeX and handwritten formula images in generated keyframes.
• Source code (analysis + video frame ground truth creation tools): AccessMath_ICDAR_2017_code.zip
• Video annotations/data (256MB): AccessMath_ICDAR_2017_data.zip
• Original videos: Video Recordings
• Extensions (Fall 2018): A newer version of the source code may be found on github. Re-encoded versions of the lecture videos are now available in MP4 format from CUBS at the University at Buffalo.

#### DPRL Natural Scene Text Detector

(S. Zhu, Apr. 2016) The code below was used to produce the results published in Siyu Zhu's 2016 CVPR paper, A Text Detection System for Natural Scenes with Convolutional Feature Learning and Cascaded Classification, which obtained state-of-the-art results on the ICDAR 2015 Focused Scene Text Detection task at the time of publication.

#### The Tangent Math Search Engine

• Tangent-V and AccessMath (July 2018, by K. Davila) This is a generalization of the Tangent formula search engine for more general Visual Search based on appearance alone. This version can be used for indexing and retrieval of binary images. It has been described in our ICFHR 2018 paper. Note: this version also includes code for indexing keyframes in lecture videos that are used for search (from the lab's AccessMath project).

• Tangent-S (July 2017, by K. Davila, R. Zanibbi, A. Kane and F. Wm. Tompa). This version of the Tangent formula search engine supports individual and parallel search of formula appearance and semantics. This version extends Tangent v. 0.3.1 below, and is described in our SIGIR 2017 paper.

• Tangent 0.3.1 (May 2016, by K. Davila, R. Zanibbi, A. Kane and F. Wm. Tompa). This is the version described in our NTCIR-12 competition paper, with wildcard support for full subexpressions, and better separation of code for scoring metrics and locating subexpressions with the best match.

• Tangent 0.3 (July 2015, by R. Zanibbi, K. Davila, A. Kane and F. Wm. Tompa). You can download the source code and sample results (including .html pages with highlighted hits) below. This is the version described in our SIGIR 2016 paper.

• Tangent 0.2 (2014). Nidhin Pattaniyil implemented this extension of the Tangent system to support matrices and prefix subscripts and superscripts. This updated Tangent combines math expression retrieval with a Solr/Lucene text retrieval system, supporting mixed math and text queries.

Please Note: the files below are quite large, in part so that others have a better chance to replicate our results at NTCIR-11 (2014; NTCIR-11 paper)

• Tangent 0.1 demo (2013). A math expression search engine create by David Stalnaker. This online demo searches math expressions in an earlier version of English Wikipedia.

#### CROHME Handwritten Math Recognition Competitions (web page)

• IAPR TC11 CROHME Web Page (datasets and evaluation tools)
• CROHME InkML file viewer (source code provided with CROHMELib below)
• Handwritten math symbol recognizers (source code)
• Kenny Davila's System (SVM/Random Forest-based using offline-style features, 2014)
• Lei Hu's System (HMM-based using online features, 2011) - currently used for ${m}_{\mathrm{in}}$
• Complete systems submitted by the dprl (the 'RIT' team) for the competition:
• Early tools (2011) developed during the lab's participation in the first CROHME (R. Pospesel and K. Hart)

#### US Patent Office (USPTO) Figure and Part Label Detection Competition

• Paper co-authored by Chris Riedl (Northeastern, former Harvard post-doc), Marti Hearst (UC Berkeley), Siyu Zhu, Richard Zanibbi and researchers from the Harvard-NASA Tournament Lab (Karim Lakhani et al.) describing an online competition for labeling parts in US patent diagram images has been posted on the arXiv.

• The data and source code for the top-5 placing systems in the competition are available through the UCI Machine Learning Repository.

#### Evaluation Tools

• LgEval: the Label Graph Evaluation library (by R. Zanibbi and H. Mouchere). The library uses labeled directed graphs to represent results for structural pattern recognition tasks. To obtain the current version, issue the following command using git: