2
$\begingroup$

I would like to find out the formula for CandidateAbility used in the European PISA-test, which tests 9th grade pupil's abilities. Unfortunately the agency which publishes the results does not provide many mathematical facts. They say they use a logit-function to determine pupils' abilities in terms of percentage of correctly solved problems from a fixed problem set and the average problem difficulty for that set (never mind the definition of that). Googling for "logit" revealed the following formula:

$\mathrm{CandidateAbility} = \log \left( \frac{x}{1-x} \right) + \mathrm{AverageDifficulty}$

where $x$ denotes the fraction of correctly solved problems.

Assuming $\mathrm{AverageDifficulty}=0$ for now, this is centered around 0.5, i.e. a pupil solving half of the problems gets assigned ability zero. However, the PISA-agency says that they center the scale around 0.625, i.e. a pupil solving 62.5 percent of the problems gets assigned 0.

Now I can imagine many ways of modifying the above formula to achieve this. The first that come to my mind are:

$\mathrm{CandidateAbility} = \log \left( \frac{x}{1-x} \right) + \mathrm{AverageDifficulty} - \log \left( \frac{0.625}{1-0.625} \right),$ just shifting the outcome of the formula, and

$\mathrm{CandidateAbility} = \log \left( \frac{x-c}{1-(x-c)} \right) + \mathrm{AverageDifficulty}$

where c=0.625-0.5 (the difference between the new and the old center), modifying the input into the log-term.

My question is: Is there any modification of the formula, either one of the above or something entirely different, which is most natural from a statistician's point of view? Any suggestion would be welcome and could be used to counter-check against the data that is provided by the PISA-agency. Thanks!

  • 0
    Sorry for not $m$entioning it: They do indeed use a sort of Rasch $m$odel.2011-03-28

3 Answers 3

1

Logistic regressions in testing usually come from Item Response Theory:

http://en.wikipedia.org/wiki/Item_response_theory

For the logit scale, shifts (say from $a$ to $b$) are always of the form

$L(b) = L(a) + C$

where $L(x) = \log (x/(1-x))$ is the logit function.

If ability is centered at 0, has an additive effect on performance measured in the logit scale, and ability 0 corresponds to 62.5 percent correct solutions, then:

$L( \text{Percentage Solved}) = L(62.5) + \text{Ability}$

To get percentage or ability, use the inverse function of $L$, which is $e^L / (1+e^L)$.

0

Since the logit function needs inputs from the $(0,1)$ interval, your second formula is out of question. On the other hand, the first formula looks like a reasonable guess.

0

Is this the item you found on the web? I couldn't write as vaguely or as badly as that if I tried. The title says "What is a logit?". Any respectable answer to that question would have to say that the logit of a number $p$ between $0$ and $1$ is $\log(p/(1-p))$.

http://en.wikipedia.org/wiki/Logit

It looks to me as if they may be trying to say that "CandidateAbility" is defined as the logit of the probability that a candidate will get a correct answer plus an amount added to that so that CandidateAbility will be $0$ when that probability is $1/2$. But you have to read between the lines to guess that that may be what is meant. I am offended by their lack of clarity.

If they take that probability to be the actual proportion of right answers that a candidate got, then the CandidateAbility is $\infty$ if the candidate gets ever answer right.

They seem to have set up the definition so that "average ability" means getting 50% of the answers right. That seems horribly misleading.