4
$\begingroup$

Lets imagine that we have some experiments. Each experiment may result in one of the outcomes: A, B, C. So we have probability distributions for each experiment $P_A, P_B, P_C$ which is context-dependent, e.g.:

  1. $Context_1 \Rightarrow \{P_A^1, P_B^1, P_C^1\},$ experimental outcome is A
  2. $Context_2 \Rightarrow \{P_A^2, P_B^2, P_C^2\},$ experimental outcome is B
  3. $Context_3 \Rightarrow \{P_A^3, P_B^3, P_C^3\},$ experimental outcome is A
  4. $Context_4 \Rightarrow \{P_A^4, P_B^4, P_C^4\},$ experimental outcome is C

These probabilities are calculated by some function $F:Context\rightarrow \{P_A, P_B, P_C\}$

I want to estimate an absolute trust rate of this function. In other words, I want to be able to say "we can trust this function on 86%" like we do when we deal with Pearson's chi-square test.

Any suggestions?

  • 0
    Seems to me you need a Bayesian reformulation of your problem: $\mathbb{P}(X|Context)$.2010-11-28

2 Answers 2

1

If your predictions are repetitive I think you can do this. For example, if you always predict 50% chance of one choice, 30% for a second and 20% for the third (even if they are permuted on a particular play) you can do a standard chi-square test. Did you in fact get 50% right on the one you claim?

If your predictions vary continuously, you can still pick the highest probability and compute the observed number of times it is right compared to the prediction and the variance using the variance on each prediction is p(1-p). Then you can compare with the null hypothesis of 1/3, 1/3, 1/3

I'm not sure how to understand the statement "we can trust this function on 86%". I think the chi-square test can say "it is 86% probably that my predictions are better than the null hypothesis", but that doesn't tell you that the predictions are right.

2

Your function is really

$F: Context \mapsto \mathbb{P}(X|Context)$

or in other words you just want to determine the probability of a particular outcome, conditional on a certain context. The question is, what are your data? Do you know about the different contexts? Or do you only dispose of the outcomes?

Anyway, since $\mathbb{P}(X|Context)=\frac{\mathbb{P}(X \cap Context)}{\mathbb{P}(Context)} \; ,$ I think any test you'd want to devise for this function would boil down to the Chi-squared test anyway.