Project Summary

This project aims to produce new methods for retrieving math in documents, using mathematical expressions as queries (query-by-expression). We are working to develop retrieval tools that are intuitive to use, both for math experts and (perhaps more importantly) non-experts. Methods developed for the project might later be adapted for retrieving other non-textual document elements such as chemical diagrams, tables, and figures. Source code and experimental data developed for the project will be made public via the project web site (http://www.cs.rit.edu/~dprl/msearch.html). To promote mathematical literacy, the principal investigator and graduate students working on the project will visit middle schools and talk about the history, recognition and retrieval of mathematical notation. The project will also provide opportunties for students to obtain research experience, including students in the McNair Scholars program at RIT. The McNair Scholars program seeks to provide research experiences to low-income, first-generation college students that are interested in pursuing doctoral studies.

Project Team

Richard Zanibbi, PhD, Principal Investigator
Lei Hu, PhD Student (Computer Science, RIT), Sept. 2010-May 2015
Siyu Zhu, PhD Student (Imaging Science, RIT) Sept. 2011-May 2015
Kevin Talmadge, MSc Student (Computer Science, RIT), Summer 2014
Nidhin Pattaniyil, MSc Student (Computer Science, RIT), Spring 2014
Kenny Davila Castellanos, MSc (May, 2013) Jan. - May 2013
Keita Del Valle, MSc Student (Human-Computer Interaction, RIT) Sept. 2012- Dec. 2013
Justin Kelly, NSF REU Student, June 2013 - May 2014
Awelemdy Orakwue, NSF REU Student, Sept. 2012 - May 2014
Matthias Reichenbach, MSc Student (Human-Computer Interaction, RIT) Sept. 2012-May 2013
Robert Li Volsi, MSc Student (Computer Engineering, RIT), Jan. 2011-Aug. 2012
Christopher Sasarak, NSF REU Student, Summer 2012, RA Summer 2013, MSc Project Spring 2014 (BSc/Msc Computer Science Student, RIT)
David Stalnaker, BSc/MSc Student (Computer Science, RIT), Dec. 2011-Aug 2013
Meridangela Gutierrez Jhong, BSc Student and McNair Scholar (Computer Science), Jan.-May 2011
Kevin Hart, NSF REU Summer 2011 (BSc Computer Science, RIT), RA Sept. 2011-Feb. 2012
Thomas Schellenberg, MSc Student (Computer Science, RIT), Sept. 2010-Nov. 2011
Benjamin Holm, MSc Student (Computer Science, RIT), April 2010-Aug. 2011
Richard Pospesel, Research Programmer (Master's in Game Design and Development Student, RIT), Sept. 2010-July 2011

Collaborators

Francisco Alvaro (PhD Student, Univ. Valencia, Spain)
Dorothea Blostein (Queen's University at Kingston, Canada)
Houssem Chatbri (PhD Candidate, Univ. Tsukuba, Japan)
Matthew Fluet (RIT)
Keisuke Kameyama (Univ. Tsukuba, Japan)
Andrew Kane (Postdoctoral Fellow, Univ. Waterloo, Canada)
Harold Mouchère (IRCCyN/IVC, Nantes, France)
George Nagy, (Prof. Emeritus, RPI)
Frank Tompa (Univ. Waterloo, Canada)
Christian Viard-Gaudin (IRCCyN/IVC, Nantes, France)
Bo Yuan (RIT)

Software

The new min math search interface, and the Tangent math search engine (for which expressions from Wikipedia have been indexed) may be used by clicking on their images at the top of this page. Both work with desktop machines and tablets.

The CROHMELib and LgEval libraries for evaluating structural pattern recognition systems and the CROHME handwritten math competition have been publicly released under a Creative Commons non-commercial license.

The DPRL Lab GitHub Page provides source code for the following.
• Tangent math search engine, versions 0.1 (D. Stalnaker, 2013) and 0.2 (N. Pattaniyil, 2014)
• DPRL lab entries for CROHME 2011-2014 (all four systems)
• HMM-based (L. Hu, 2011) and offline feature/SVM-based (K. Davila, 2014) handwritten math symbol classifiers
• Image-based classifier for typeset symbols (S. Zhu and L. Hu, 2013)
• The min multimodal math search interface, supporting mouse/touch, keyboard and image input, with integration for multiple search engines (e.g. DLMF, Wolfram Alpha, Google, and our Tangent search engine for formulas in a Wikipedia snapshot)

Additional software is available from the DPRL software web page (e.g. the SML-based Recognition Strategy Language, version 0.2) and the ICDAR 2011 tools web page.

Publications

(Note: In June 2015, two additional papers for the project were under review)

2015

Zanibbi, R. and Orakwue, A. (2015) Math Search for the Masses: Multimodal Search Interfaces and Appearance-Based Retrieval. Proc. Conference on Intelligent Computer Mathematics, Washington, DC (to appear, July 2015).

Stalnaker, D. and Zanibbi, R. (2015) Math expression retrieval using an inverted index over symbol pairs. Proc. Document Recognition and Retrieval, Vol. 9402 of Proc. SPIE, pp. 940207:1-12,San Francisco.

2014

Pattaniyil, N. and Zanibbi, R. (2014) Combining TF-IDF Text Retrieval with an Inverted Index over Symbol Pairs in Math Expressions: The Tangent Math Search Engine at NTCIR 2014. Proc. 11th NII Testbeds and Community for Information access Research (NTCIR), Tokyo, Japan (online).

Reichenbach, M.S., Agarwal, A. and Zanibbi, R. (2014) Rendering expressions to improve accuracy of relevance assessment for math search. Proc. ACM SIGIR, Gold Coast, Australia, pp. 851-854.

Del Valle Wangari, K., Zanibbi, R. and Agarwal, A. (2014) Discovering real-world use cases for a multimodal math search interface. Proc. ACM SIGIR, Gold Coast, Australia, pp. 947-950.

Mouchere, H., Viard-Gaudin, C., Zanibbi, R. and Garain, U. (2014) ICFHR 2014 Competition on Recognition of On-line Handwritten Mathematical Expressions (CROHME 2014). Proc. Int'l Conf. Frontiers in Handwriting Recognition, pp. 791-796, Crete, Greece.

D. Blostein and R. Zanibbi. (2014) Processing Mathematical Notation, in Handbook of Document Image Processing and Recognition, pp. 679-702, Springer-Verlag.

2013

F. Alvaro and R. Zanibbi (2013) A Shape-Based Layout Descriptor for Classifying Spatial Relationships in Handwritten Math. ACM Symp. Document Engineering, Florence, pp. 123-126, Italy.

H. Mouchere, C. Viard-Gaudin, R. Zanibbi, U. Garain, D.H. Kim and J.H. Kim (2013) ICDAR 2013 CROHME: Third International Competition on Recognition of Online Handwritten Mathematical Expressions. Proc. Int'l Conf. Document Analysis and Recognition, Washington, DC, 1428-1432.

S. Zhu and R. Zanibbi (2013) Label Detection and Recognition for USPTO Images using Convolutional K-means Feature Quantization and AdaBoost. Proc. Int'l Conf. Document Analysis and Recognition, pp. 633-637, Washington, DC.

L. Hu and R. Zanibbi (2013) Segmenting Handwritten Math Symbols Using AdaBoost and Multi-Scale Shape Context Features. Proc. Int'l Conf. Document Analysis and Recognition, pp. 1180-1184, Washington, DC.

R. Zanibbi, H. Mouchere, and C. Viard-Gaudin (2013) Evaluating Structural Pattern Recognition for Handwritten Math via Primitive Label Graphs Proc. Document Recognition and Retrieval, Proc. SPIE vol. 8658, pp. 17-1 - 17-11, San Francisco, CA.

S. Zhu, L. Hu and R. Zanibbi (2013) Rotation-Robust Math Symbol Recognition and Retrieval Using Outer Contours and Image Subsampling Proc. Document Recognition and Retrieval, Proc. SPIE vol. 8658, pp. 05-1 - 05-12, San Francisco, CA.

2012

C. Sasarak, K. Hart, R. Pospesel, D. Stalnaker, L. Hu, R. LiVolsi, S. Zhu, and R. Zanibbi. (2012) min: A Multimodal Web Interface for Math Search. Symp. Human-Computer Interaction and Information Retrieval, Cambridge, MA (online, 4pp).

L. Hu, K. Hart, R. Pospesel, and R. Zanibbi. (2012) Baseline extraction-driven parsing of handwritten mathematical expressions Proc. Int'l Conf. Pattern Recognition, Tsukuba Science City, Japan.

R. Zanibbi and D. Blostein (2012) Recognition and Retrieval of Mathematical Expressions, Int'l. Journal on Document Analysis and Recognition 15(4): 331-357. (original publication available from www.springerlink.com).

T. Schellenberg, B. Yuan and R. Zanibbi (2012). Layout-based substitution tree indexing and retrieval for mathematical expressions, Proc. Document Recognition and Retrieval XIX, pp. 8297OI-1 - 8297OI-8, San Francisco.

2011

R. Zanibbi and B. Yuan. (2011) Keyword and image-based retrieval of mathematical expressions. Proc. Document Recognition and Retrieval XVIII, vol. 7874 Proc. SPIE, pp. OI1-OI9, San Francisco, CA.

R. Zanibbi and L. Yu. (2011) Math Spotting: Retrieving Math in Technical Documents Using Handwritten Query Images. Proc. Int'l Conf. Document Analysis and Recognition, pp. 446-451, Beijing.

L. Hu and R. Zanibbi. (2011) HMM-Based Recognition of Online Handwritten Mathematical Symbols Using Segmental K-means Initialization and a Modified Pen-up/down Feature. Proc. Int'l. Conf. Document Analysis and Recognition, pp. 457-462, Beijing.

R. Zanibbi, A. Pillay, H. Mouchere, C. Viard-Gaudin, and D. Blostein. (2011) Stroke-Based Performance Metrics for Handwritten Mathematical Expressions. Int'l Conf. Document Analysis and Recognition, pp. 334-338, Beijing.

News (Sept. 2010 - May 2015)

2015

• (May) The paper for Prof. Zanibbi's upcoming invited talk at CICM 2015 is now available: Math Search for the Masses: Multimodal Search Interfaces and Appearance-Based Retrieval. Awelemdy Orakwue (BSc. student) is a co-author on the paper. The paper has also been indexed in the arXiv.
• (Mar.) Project alumnus Keita Wangari (MSc HCI, 2014) has accepted a position with Google Research in Mountain View, CA as a User Experience (UX) Researcher.
• (Mar.) Additional stories about the Tangent math search engine have appeared online: Making Search Engines Understand Math (EE Times), RIT Takes Lead in Formula Search Engine Racing (Campus Technology), and Demystifying Mathematics with Tangent Search Engine (Technology.org). The EE Times story was later picked up by ACM TechNews.
• (Mar.) Houssem Chatbri will be giving a talk on sketch-based retrieval in lecture videos on Friday March 4th at 12pm in GOL-1455.
• (Feb.) Today (Feb. 4) an RIT University News story on the Tangent math search engine appeared. The story features David Stalnaker, Nidhin Pattaniyil, and Kenny Davila.
• (Feb.) The research methods and writing text Writing for Computer Science (3rd ed.) by Justin Zobel will be published in the coming days. Prof. Zanibbi contributed some of the exercises in the new edition.
• (Feb.) The lab welcomes Houssem Chatbri (PhD student, Univ. Tsukuba, Japan), who will be visiting during Feb.-March. He will collaborate with Prof. Zanibbi on math retrieval research.

2014

• (Dec.) Prof. Zanibbi will be an invited speaker at the 2015 Conferences on Intelligent Computer Mathematics (CICM).
• (Dec.) Nidhin Pattaniyil has received a travel award from the NTCIR-11 conference organizers to present his work in Tokyo.
• (Oct.) David Stalnker's paper Math expression retrieval using an inverted index over symbol pairs has been accepted for oral presentation at Document Recognition and Retrieval XXII (2015) in San Francisco.
• (Oct.) Nidhin Pattaniyil extended David's work for his MSc project (paper), adding support for matrices and combining expression retrieval with text search to create an entry for the international NTCIR Math Retrieval Competition. The competition was held for the upcoming NTCIR-11 conference in Japan. David and Nidhin's system performed best for retrieving Wikipedia expressions, and had the best overall top-5 results and second-highest 'highly relevant' top-5 results for combined text and math queries for 100,000 documents from the arXiv.
• (Sept.) Prof. Zanibbi gave a talk in the Human-Computer Interaction Seminar at the University of Waterloo on Sept. 26th, entitled "Creating User-Friendly Math Search Engines." The talk covered search interface designs and math retrieval algorithms developed in the DPRL. Our thanks to Dr. Edward Lank for the invitation to give this talk.
• (Sept.) The successful ICFHR 2018 Rochester bid has been announced on the GCCIS Web Pages.
• (Sept.) Prof. Zanibbi submitted a successful bid to host the 2018 International Conference on Frontiers in Handwriting Recognition in Rochester, at RIT. The bid is available here, and the presentation slides (presented by Umapada Pal at ICFHR 2014 in Crete) are available here. If you work in handwriting recognition, we hope to see you in 2018!
• (Aug.) Kevin Talmadge has succesfully completed his Master's Project on applying the Recognition Strategy Library to analyzing a math recognition system.
• (June) As part of an Image and document processing, recognition and interaction seminar held at IUT/Univ. Nantes (France, on June 30), Prof. Zanibbi gave a talk on human math retrieval studies and math search engines (work done with Keita Wangari, Matthias Reichenbach, David Stalnaker and Nidhin Pattaniyil.) Other presenters were researchers from Australia, Brazil, and China. (program; slides)
• (June) Amit Pillay has successfully defended his MSc thesis, Intelligent Combination of Structural Analysis Algorithms: Application to Mathematical Expression Recognition, which presents a method for combining DRACULAE math notation parsers using a Graph Transformer Network (GTN). This technique was successful in increasing spatial relationship detection accuracy.
• (May) Congratulations to Kenny Davila, who has succesfully completed his Research Potential Assessment, which is required for PhD students at the end of their first year.
• (May) The lab welcomes Kevin Talmadge, who will be working on the new Python-based Recognition Strategy Library for his MSc Project in the summer.
• (May) Congratulations to Zack Miller, Nidhin Pattaniyil and Christopher Sasarak, who have all successfully completed their MS Projects (their presentation posters are available from the "Publications" link above).
• (Apr.) Kenny Davila's paper describing a state-of-the-art classifier for handwritten mathematical symbols has been accepted for oral presentation at ICFHR 2014 being held in Greece this coming September.
• (Apr.) Keita Del Valle's paper describing a user study of the min math entry and search interface has been accepted for publication at ACM SIGIR 2014. SIGIR is being held in the Gold Coast, Australia in July.
• (Apr.) Matthias Reichenbach's paper on the effect of rendering math expression in search hits has been accepted for publication at ACM SIGIR 2014.
• (Mar.) The lab has released a number of systems for math recognition and retrieval as open source on GitHub (DPRL GitHub Page). We hope to add some more information to these pages soon.
• (Mar.) Kenny Davila has released his collection of isolated handwritten math symbol classifiers under GNU GPL v.3 (available here). In benchmarking experiments, the best classifiers (using an SVM with RBF kernel) have recognition rates that match or exceed the best published rates for the MathBrush handwritten symbol data set.
• (Mar.) As part of his upcoming sabbatical, Prof. Zanibbi will be a Visiting Professor at IRCCyN/IVC (Nantes, France) during June and July, and at the University of Waterloo Computer Science Department (Canada) in October.

  2013

• (Aug 2013) David Stalnaker has successfully defended his MSc thesis, Math Expression Retrieval Using Symbols Pairs in Layout Trees.
• (July 2013) Christopher Sasarak will be giving a presentation on his work with recognizing ASL in video using the Recognition Strategy Language at the Undergraduate Research Symposium on Friday August 2nd at 2:15pm (at the RIT Inn).
• (June 2013) Lei Hu and Siyu Zhu will be participating in the Doctoral Consortium at the International Conference on Document Analysis and Recognition in August.
• (June 2013) The lab welcomes Justin Kelly to the lab. Justin is a BSc (Computer Science) NSF REU student who is working over the summer on recognizing math in images.
• (June 2013) The lab welcomes Dr. Harold Mouchere from Univ. Nantes, France, who is visiting the lab and collaborating with Prof. Zanibbi during the month of June.
• (May 2013) Congratulations to Matthias Reichenbach, who has successfully defended his MSc project on the effect of rendering math expressions on relevance assessment for math search.
• (May 2013) Congratulations to Kenny Davila Castellanos on successfully defending his MSc project on image-based search of whiteboard notes in math lecture videos. Kenny will start his PhD in the DPRL for the NSF-funded AccessMath project this summer, co-supervised by Prof. Zanibbi and Prof. Ludi.
• (May 2013) Prof. Zanibbi has been awarded an NSF REU grant, which will be used to fund undergraduate researchers in the DPRL lab during summer 2013.
• (May 2013) ImagineRIT will be held on Saturday, May 4. The lab will be demonstrating the $m_{\mathrm{in}}$ math search interface in the atrium of the Golisano college.
• (April 2013) Congratulations to Matthias Reichenbach, who has secured a position with Square in San Francisco.
• (Mar. 2013) Prof. Zanibbi gave a talk on the dprl's work in math recognition and retrieval Creating User-Friendly Systems for Math Search as part of the RIT Imaging Science Seminar Series.
• (Feb. 2013) CROHMELib v. 0.1 and LgEval v. 0.2 have been released.
• (Jan. 2013) The $m_{\mathrm{in}}$ math search interface will be demonstrated at the 20th Document Recognition and Retrieval Conference being held in San Francisco Feb. 5-7th.
• (Jan. 2013) The 3rd international Competition on Recognition of Handwritten Mathematical Expressions (CROHME) has begun! Data and tools are available. The registration deadline is Feb. 15th.
• (Jan. 2013) The lab welcomes Francisco Alvaro, a PhD student from the University of Valencia, Spain, who will visit the DPRL between January and May of this year, as he carries out his research in mathematical notation recognition.
• (Jan. 2013) The LgEval library (a python library for evaluating structural pattern recognition algorithms by Profs. Zanibbi and Mouchere) has been posted on the Software page.

  2012

• (Dec. 2012) The lab welcomes Alex Canter to the lab, who will be looking at applying OCR to text extracted from natural scenes.
• (Nov. 2012) Siyu Zhu and Lei Hu have received RIT Graduate Research and Creativity Awards ($500 each), to be used in support of their travel to DRR in San Francisco, and ICPR in Japan, respectively.
• (Sept 2012) The lab welcome Matthias Reichenbach to the lab. Matthias is an MSc in HCI student, who will be completing a project on human relevance assessments for math search.
• (Sept 2012) Christopher Sasarak has received a student travel award in support of his poster presentation at HCIR 2012.
• (Sept 2012) The lab welcomes Awelemdy Orakwue (NSF REU Student) and Keita Del Valle (MSc Student, Human-Computer Interaction), both of whom will be working on HCI aspects of math recognition and retrieval.
• (Sept 2012) David Stalnaker has been offered a position with Google.
• (June 2012) Prof. Zanibbi will be giving a talk entitled Some Recent Developments in Math Retrieval at the Université de Nantes, France on June 14.
• (June 2012) Prof. Zanibbi is spending the month of June as a Visiting Professor at the IRCCyN/IVC research center located within Polytech Nantes, France.
• (May 2012) The DPRL lab will be demonstrating the min math search interface at ImagineRIT on Saturday, May 5.
• (April 2012) The lab welcomes Christopher Sasarak. Christopher will be working on the min interface and math recognition algorithms over the summer as an NSF REU student.
• (March 2012) Prof. Zanibbi gave a seminar on current math recognition and retrieval research in the DPRL at Lehigh University.
• (Jan 2012) The lab welcomes Meridangela Gutierrez Jhong. Meridangela is an undergraduate research assistant, who will be carrying out work on math retrieval in LaTeX documents.

  2011

• (Dec 2011) The lab welcomes David Stalnaker, a BSc/MSc student in Computer Science who will be working as a research assistant on the project.
• (Dec 2011) The lab welcomes Robert LiVolsi, a BSc/MSc student in Computer Engineering, who will be working on machine learning algorithms for handwritten math recognition.
• (Nov 2011) Thomas Schellenberg has successfully defended his MSc thesis, Layout-Based Substitution Tree Indexing and Retrieval for Mathematical Expressions.
• (Oct/Nov 2011) Prof. Zanibbi gave talks on math recognition and retrieval research in the DPRL at the Rochester chapter meetings for the SPIE and IEEE Computer Society (slides are available from the IEEE web page).
• (Sept 2011) The lab welcomes Siyu Zhu to the lab. Siyu is a PhD student in Imaging Science, who will be working on math retrieval using images.
• (Sept 2011) Thomas Schellenberg, Prof. Yuan and Prof. Zanibbi's paper Layout-based substitution tree indexing and retrieval for mathematical expresions has been accepted for presentation at Document Recognition and Retrieval 2012.
• (Sept 2011) Lei Hu and Prof. Zanibbi presented research papers on HMM-based math symbol recognition and image-based math retrieval, along with a poster describing a new metric for evaluating math recognition systems at ICDAR 2011 in Beijing. The metric was developed in collaboration with researchers from Université de Nantes (France) and Queen's University (Canada).
• (Aug 2011) Ben Holm has successfully defended his MSc thesis, Evaluation of RSL History as a Tool for Assistance in the Development and Evaluation of Computer Vision Algorithms.
• (July 2011) Prof. Zanibbi and Prof. Dorothea Blostein's survey on recognition and retrieval of mathematical notation has been accepted for publication in the International Journal on Document Analysis and Recognition.
• (May 2011) A first prototype of a web-based system for pen/finger-based math entry will be presented at ImagineRIT on May 7th.
• (April 2011) We welcome Kevin Hart, who will be working on the project as part of an NSF Research Experience for Undergraduates (NSF REU) project.