For context, a friend hosts a pub trivia night and would like to know a good way to score ranking questions. For example, put these five movies from the 70's in order of release: Jaws, Star Wars, Alien, Blazing Saddles, and Superman. There is a correct ordering, but how should partial credit be allotted to incorrect orderings?
Somewhat more formally, suppose that you put out a question asking participants to order $N$ things. What are good ways to grade their responses with partial credit?
For example, suppose $N=5$ and participants are asked to put B, C, E, A, and D in order. Full credit (whatever that means) is given for ABCDE. But what's a good way to give partial credit for other permutations?
For example, I think EDCBA should get no credit at all, since it's completely backwards (despite having one letter in the right position). I think ACBDE should get the highest possible partial credit (whatever that would be), since it is only one transposition away from correct. (Personally I'd also factor in that the transposition is in the middle, so at least the participant got the extremal letters correct. However, that turns this into a non-symmetric problem, and I'm happy to throw out that consideration.) I think BCDEA should get a fair amount of partial credit, since the relative ordering of 4 out of 5 items is correct, despite having no letters in the correct position.
I'm aware of a several reasonable-sounding methods to assign partial credit. Each method measures how "far away" the submitted answer is to the correct answer, and then uses some function to turn that distance into a score in a negative manner (the more the distance the less the score.)
Possibilities for measuring distance:
$\bullet$ Count the minimal number of adjacent transpositions that turn the submitted answer into the correct answer.
$\bullet$ Create an $N$-vector measuring the distance from each letter's submitted position to its correct position, and put a notion of size on that vector. For example DCABE would have vector $(3,1,2,2,0)$. Size could be the standard Euclidean distance, the square of Euclidean distance, the taxicab metric...
$\bullet$ Other ideas?
Then there are lots of reasonable functions that turn distance into a score.
$\bullet$ $\mbox{score(distance)}=100\%\cdot r^{\mbox{distance}}$, for some $r<1$.
$\bullet$ $\mbox{score(distance)}=100\%-k\%\cdot\mbox{distance}$, where $k\%\cdot\mbox{(maximal distance)}=100\%$.
$\bullet$ $\mbox{score(distance)}=\displaystyle\frac{100\%}{1+\mbox{distance}}$
$\bullet$ Other ideas?
What I am looking for is more than just new and different proposals for scoring (although I'd love to hear more of those too). I am more interested in reading a convincing argument for preferring a particular method of scoring.