1
$\begingroup$

This question spills from another, although the topic of this question is signifcanly different. Old question for reference Goodness-of-fit test for nominal data?

Now, lets say I am doing this study for real. I have 4 teams and 4 corresponding colours. I ask 30 people to choose a colour and a team. The chance of "agreement" between team and colour is therefore $1/4.$

My null is "there is no difference between someones colour choice, and the team they support"

My observed values are; 10 people choose a colour that "agrees" with there team 20 people do not. With expected values 7.5 22.5

If I run a chi-squared test, I will find $H_0$ is accepted, despite the fact the vast majority do have a differing favourite colour and favourite team.

Why does the chi-squared test result seem so counter-intuitive in this case?

  • 1
    It is true that in your experiment, the majority of the people "disagree" , but the question is if the ratio 1:2 is statistically "significant". Because that is what matters for Ho to be rejected. Apparently it is not... – 2017-01-22
  • 0
    Thanks, it just seems so counterintuative. Am i missing something though, is it okay to get the expected values this crudley, i.e multiplying 1/4 and 3/4 by 30, or should I be using the observed values as a way to get the expected values? – 2017-01-22
  • 1
    The thing is, we human beings are much quicker convinced that we should reject Ho, just because one group is "larger" than the other. But in terms of statistical significance, it takes more effort on behalf of the H1 group to get the choice on their side. This is not only true for the Chi Square test, but for other tests too... – 2017-01-22
  • 0
    Thanks, imran, are my expected values correct though? Should I not be using the fact I know the distrubution of people's favourite colour for example more than I am? I am just multipling by 1/4 which seems wrong. – 2017-01-22

1 Answers 1

1

First, you have done the chi-squared test correctly: With Observed frequencies $X = (10, 20)$ for (Agree, Disagree), and Expected frequencies $E = (7.5, 22.5),$ I get the chi-squared GOF statistic to be $Q = 1.11.$ This is far below the 5% critical value 3.84 (with 1 degree of freedom). So you cannot reject the null hypothesis of choosing colors at random.

Second, with goodness-of-fit (GOF) tests, one must be very careful in interpreting 'failure to reject' to mean that $H_0$ is the truth. It may merely mean that you have insufficient data to put $H_0$ in doubt.

Third, especially at relatively small sample sizes, results of GOF tests are extremely sensitive to the number of subjects. If you had 60 agreements and 120 disagreements among 180 subjects (the same ratio as 10 agreements among 30), then you would have expected counts $E =(45, 135)$ and $Q = 6.67 > 3.84,$ so you would easily reject $H_0.$

This is why bar charts, often used to display categorical data, should always show counts (not relative frequencies); relative frequency bar charts for your $X = (10, 20)$ and for the significantly nonrandom $X = (60, 120)$ would be completely indistinguishable.

Last, perhaps 95% confidence intervals for the proportion of agreements in these two cases would put the situation in perspective.

For $Y = 10$ agreements in $n = 30,$ you'd have $\hat p = 10/30 = 1/4.$ An Agresti-style 95% CI for the true proportion $p$ of agreements is of the form $\tilde p \pm 1.96\sqrt{\tilde p(1-\tilde p)/(n+4)},$ where $\tilde p = (Y+2)/(n+4) = (10+2)/(30 + 4) = 0.3529412.$ The result is $(0.192, 0.514),$ which includes your hypothetical $p = 1/4.$

By contrast, For $Y = 60$ agreements in $n = 180$ the 95% Agresti CI would be $(0.269, 0.405),$ which does not include the hypothetical $p = 1/4.$

Notes: (1) If you are not familiar with Agresti-Coull CIs you can read about them in on the Wikipedia or (with more technical detail) NIST Internet sites. They have been shown to have more accurate coverage probabilities than CIs based just on $\hat p = Y/n.$ (2) I have never been great at proofreading, but this post is brought to you courtesy of the entire flu-remedy aisle at the local pharmacy. Please check all computations and leave a Comment if something doesn't seem right.

  • 0
    Thank you BruceET, you have really helped. – 2017-01-23