Recognition methods are traditionally evaluated on a black box basis, looking only at the differences between final interpretations and ground truth. Black box evaluations hide the process by which interpretations are arrived at, making it difficult to compare, understand, and integrate recognition methods, slowing overall advance in the area. We propose to "open the box" by representing recognition methods as explicit sequences of decision functions that accept, reject, or reconsider interpretation elements. We present a new formal notation for this purpose, RSL, and an interpreter for RSL that records a complete history of decisions made during recognition.
From decision histories one may evaluate individual decisions, recover intermediate interpretations, and observe new evaluation metrics that characterize recognition processes. We present two such metrics, historical recall and historical precision. The benefits of a decision-based, white box evaluation are demonstrated through a detailed comparison of table cell detection by two algorithms. We will also outline how a decision-based approach may be used to support the intelligent combination and optimization of recognition algorithms.
This is joint work with Dorothea Blostein and James R. Cordy, Queen's University.
Colloquia Series page.