2
$\begingroup$

Let's say that for each trial, there are two possible outcomes, A and B. Over time, A shows that it occurs 54% of the time, B is 46%. But there is another variable that affects the outcome of each trial. So A is not always .54 and B is not always .46.

I believe I can predict more accurately than 54% the outcome of each trial, but how do I know how successful I am?

Example:

Trial 1. I say it is 65% A. It comes up A. Trial 2. I say it is 50% A. It comes up B. Trial 3. I say it is 61% A. It comes up A.

Obviously I have done better than the standard 54%, but by how much? How is it measurable?

What if instead of 65, 50, 61 I had said 52, 40, 55. Would I have been more accurate then? (just an example to illustrate my question)

What method could be applied to show my accuracy rating over several hundred trials, each with a different 'guessed' % for A, and either an A or B outcome.

Ideally I want to compare two or more guessing methods, to see which is more accurate. But I can't wrap my brain around how to assign accuracy points for each trial, since the guess varies with each trial.

1 Answers 1

1

Two common approaches are to calculate the relative likelihood of the observed data under your predictions and under the constant prediction such as 54 percent rate of A; and to quantify the "information" (relative entropy = Kullback Liebler divergence) of the data distribution given the predictions.

  • 0
    Suppose that the results in three trials are A,A,B. If predicted probabilities are p1, p2, p3 for outcomes of A in the trials, the chance of seeing those results (assuming prediction is correct) is $p_1 p_2 (1 - p_3)$. The ratio of this predicted probability using (p1, p2, p3) compared to the probability using another triple, such as (0.54,0.54,0.54), is called a "likelihood ratio" and is discussed in books on statistics.2011-08-20