Having two scheduling strategies, I'd like to test which one performs better. The input data (several problem test cases) for the strategies are randomly generated. The strategies are used to compute an optimal feasible schedule. During the scheduling, several things are measured: scheduling time, number of backtracks performed, number of nodes in the search tree, and other criteria. I'd like to compare the strategies at each of the criteria.
I can't determine any distribution for the measured data. Moreover, the data might strongly differ for each test case. For example, the scheduling times might be:
| test case | strategy #1 | strategy #2 | |------------|-------------|-------------| | #1 | 300 | 500 | | #2 | 1200 | 3300 | | #3 | 150 | 140 | | #4 | 2340 | 6872 | | #5 | 4354 | 9335 | | #6 | 972 | 869 |
Is there a test statistic I can use to perform a hypothesis test such as: "strategy #1 runs faster than strategy #2"? What is the best way to measure which of the two strategies performs better at a criterion?