For our lab I implemented simple ELO rating-based image contest. Everyone submitted images and then had 10 chances to compare two random pictures.
I used following function to determine ratings:
$$R_0(a)=R_0(b)=1400\\P(a,b)=1/(10^{\frac{(R(a) - R(b)}{400}} + 1)\\R(a)=R(a)+K(1-P(a,b))$$
Image $a$ here is the one "picked" from pair $\{a,b\}$, $K=20$. $R(a)$ is rating of image $a$ at any given moment.
I was wondering whether it is important what is the order of votes. Say, if you have triple of $a,b,c$ images, will results change if votes are cast in order $\{a,b\},\{b,c\}$ or $\{b,c\},\{a,b\}$
From non-linearity of $P(a,b)$ it seems that order of votes will matter, at least when number of votes is relatively small.
To answer that question, I ran simple MATLAB simulation. For 1000 iterations, ELO scheme was applied to votes, but votes where shuffled each time. I calculated distribution of results for two values of parameter $K$ and here is what I got:
What conclusion can I draw from this result, except that there is probability of different result with same votes? Should we use smaller K parameter for future contests, given that we have limited amount of voters (20+ people) who can handle limited amount of votes (10).
Should we share first place prize or split it according to some formula that takes into account standard deviations of these distributions? From what I understand, smaller K parameter will decrease contribution of each vote, so it should be more fair in regard to order of votes.