3
$\begingroup$

Say you take a test and are told your individual score, as a percentage. Eg: 95%.

You are also told the aggregate score (average) of all students together, as well as how many students in total took the test. Eg: 92% average of 14 students. You are not allowed to know individual scores of any other students.

What algorithm could you apply to estimate your percentile ranking among students? Eg. You scored in the upper 10% of students. How can you use assumptions about distribution and deviation to calculate a realistic percentile ranking?

  • 3
    Without doing any assumption on the score distribution, is almost impossible to answer this question. The worst case is that 3/95 of the student has scored 0, and 92/95 of them $95+\epsilon$, in that case you are ranked in the worst 3/95. The best case is that everyone has scored 92 point, so you are the best one. Every percentile between 3/95 and 1 can be reached with the right distribution. You need to do some hypotesis on score distribution (eg. each student have the same probability p of know an answer, and there are 10 question. find p such the mean is 92, and find your percentile)2012-04-23

2 Answers 2

5

I assume that when you say "The aggregate average of everyone else's scores is, say, 92%" you mean "The aggregate average of everyone's score including you is 92%", but that does not really change anything.

What you cannot say.

Consider two situations: one is that someone else got 89% on the test, you -- 95%, and all others -- 92%. Then yours is the best. On the other hand, suppose there are 10 people taking test. Someone else got 25%, you -- 95%, and everyone else -- 100%. Then the average is still 92%, but now you are the second to the last. So, you cannot really deduce your rank.

What you can say.

One thing you know is that, according to the Pigeonhole principle, there is someone who got less than 92%, because there is someone (namely, you) who got more than 92%, and 92% is the average score. So, your score is not the worst. :) But, as I have shown you, it can easily be the second to the last.

You can say more, if you know some other statistics (even without knowing the actual distribution). For example, suppose you know that the minimum score was 90%. Then you know that for every 2 scores above yours there must be at least 3 scores below you. This means that you are in the first 40% roughly.

  • 0
    From what I've understand, the problem is something like: 1. calculate score for the current user 2. recalculate some global variable (like the average score till now) 3. repeat from 1. In this architecture, you can store the number of score in each bucket when you calculate the average score.2012-04-23
1

@Vadim's answer and @carlop's comment address the issue that there's very little you can formally say without further information. But it sounds like what is wanted are some ad hoc informal but concrete suggestions for distribution assumptions, so here are a few.

Normally your percentile should not depend on the number of test-takers, since, if you assume that scores arise independently from some fixed underlying probability distribution, then your percentile as a function of your score $S$ is best thought of as $\mathrm{Pr}[x. This is also the context of grading on a curve, the percentiles are defined independent of the number of scores to be assigned.

Idea 1: Assume the scores are uniform on an interval containing 100%. Then the mean $M$ is also the median, and your percentile is $(S-2M+100)/(200-2M)$ if your score is above the minimum $2M-100$ (and zero otherwise). This has the advantage that the calculation is very easy, but the disadvantage that it only works if M>50 and has no support below a minimum score.

Idea 2: Assume the scores are more-or-less normally distributed and that 100% is a specific kind of "extraordinary," say 3 standard deviations from the mean. I.e. the standard deviation is $(100-M)/3$. Again the calculation is easy but the choice of 3 std.dev. is entirely arbitrary.

Idea 3: Assume the scores are more-or-less normally distributed, the top score is 100, and that the test-takers' scores are uniform in the percentiles, i.e. the top score is at the $N/(N+1)$ percentile for $N$ takers. This looks more sophisticated because it uses the additional piece of information $N$, but as discussed above that's not necessarily appropriate. It also now requires the quantile function for the normal distribution to compute the standard deviation.

Here's a picture for the example with given mean 92%. Example distributions

Using Idea 1 we assume the scores are uniform from 84-100 (shown in blue) and 95 is at the 69th percentile.

Using Idea 2 we assume the std.dev. is 8/3 (shown in red) and 95 is at the 87th percentile.

Using Idea 3 with $N=14$ (shown in yellow) we assume 100 is at about the 93rd percentile, about 1.5 std.dev. from the mean, then 95 is at the 71st percentile. With $N=10^5$ (shown in green) we assume 100 is about 4.3 std.dev. from the mean, then 95 is at about the 95th percentile.

Finally, here's an approximate reconstruction of the shape of the distribution of scores from the 2006 SAT scores from the published percentiles (1600 scale). For what it's worth Idea 2 does a decent job. The percentile of the perfect score on the 1600 scale was 99.93 which would be about 3.2 std.dev. away for a normal distribution, on the 2400 scale was 99.98 which would correspond to about 3.5 std.dev. enter image description here

  • 0
    Thank you Zander, these suggestions are very helpful, exactly what I needed.2012-05-16