DATA FOR DRR 2013 PAPER
Primitive Label Graph Evaluation
R. Zanibbi (rlaz@cs.rit.edu)


NOTE: data and results for the VisionObjects system has been removed
at their request.

RESULTS AND DATA

The final results reported in the DRR 2013 paper ("Evaluating Structural
Pattern Recognition for Handwritten Math via Primitive Label Graphs") for
each system (other than for VisionObjects) are available in files with the
"Results" prefix in the current directory.

System outputs may be found in 00_systemOutputs, and the ground truth in
00_testDataGT in the current directory, as inkml (.inkml) and label graph
(.inkml.bg) files.

The Analysis/ directories contains the following sub-directories.

	00_InkMLConvertErrors/
		Contains logs for inkml to label graph (.bg) conversion.
	Correct/
		Contains lists of files recognized correctly by each system.
	Errors/
		Contains generated error messages when running the evaluation script over
		each system.
	Diffs/
		Contains .csv 'diff' files, used to record recognition errors made
		in each system for each input file.
	Metrics/
		Contains .csv 'metric' files, which record metrics for each system,
		and for each input file.

The misc/ directory contains some files compiled during testing, and the
old file-to-file lists used for evaluation using the CROHME 2012 perl
evaluation script. misc/data_mods - a 'diff' of the two files present in 
this directory gives the names of files removed for comparison due to stokes
belonging to multiple symbols for the DRR analysis.

CORRECTION:

The metrics being produced were being affected by additional nodes in the VO
and Sabanci results, being counted as target nodes (after target graphs had
these extra nodes added). This was changed in bg.py so that the *true* number
of target nodes, edges, segments, and segment relationships is computed for
metrics. 


STATISTICS FOR GROUND TRUTH:

ResultstestDataGT has been generated in order to 1) sanity check the evaluation
script on 'perfect data,' and 2) provide correct counts for numbers of nodes
(strokes) and edges (stroke relationships) in the ground truth set.
