4 ms·

1. What sample size is necessary for a result of this magnitude to be significant? Because boys did 34% better when it was framed as a game. Did they have to do

by iopq 11y ago

1. What sample size is necessary for a result of this magnitude to be significant? Because boys did 34% better when it was framed as a game. Did they have to double their scores for it to be significant at this sample size?

mabbo 11y ago
> boys did 34% better No, two particular classes of students had their boys do 34%. In reality, the sample size was 4, not 80. They choose 4 classrooms, and saw these results. If the researchers switched the classroom treatment selections, are you certain we would have seen the same result? Or did we really just find out that boys in those two classrooms happen to be 34% better at reading? Those classrooms where boys did worse might have done worse for a large number of other reasons: those schools might be on the 'bad' side of town where less affluent students tend to go (happens all the time in small towns); that classroom might have been the one where the school has put all the trouble-maker kids because they know this particular teacher has the ability to teach those kinds of kids (or, conversely, that class might be the one that saw the huge boost when it was a game); the classes who did worse might have had a classmate dying in hospital that week, and the boys just didn't do very well under those conditions. There are too many other completely reasonable answers that aren't related to the test itself, given the methods used in this study. How I'd repeat the experiment: pool all the children before randomly selecting them for treatment groups to avoid bias from their origin classroom. Edit: And to be clear, I'm not saying the results are incorrect. They might be onto something amazing here. They just happen to be bad at statistics. Maybe they should frame it as a game.
- iopq 11y ago
  That's not how you count sample size. If there were four "classes" of one million boys and girls, does it still make the sample size four?
  - mabbo 11y ago
    I think I'm not stating my point very well. Apologies for that. The problem with saying that n=80 is that it presumes each 'test' (student evaluated) was chosen for their treatment independently of each other. This is clearly not true, as entire classes were chosen together. So any variable like 'actually this happens to be the gifted class' can throw the whole study up or down because that variable was not independent between the students tested. For example, let's say we did this same test in America, but two classes happened to be from the wealthiest part of Boston, and two classes were from the poorest part of Detroit. If, as we randomly selected which class was assigned each treatment, we happened to hit the 1 in 4 chance that both wealthy classes were given the 'this is a game' treatment, would we be ready to say 'the game was the cause of the better test results'? I don't believe so. That is why I find this study flawed, and why I disagree with the idea that n=80. They may be onto something, but they do not have strong statistical evidence supporting their hypothesis.
    - iopq 11y ago
      That's not a sample size, that's called bias. The sample size is still 80, but it could be potentially a biased sample.

by iopq 11y ago

mabbo 11y ago

> boys did 34% better No, two particular classes of students had their boys do 34%. In reality, the sample size was 4, not 80. They choose 4 classrooms, and saw these results. If the researchers switched the classroom treatment selections, are you certain we would have seen the same result? Or did we really just find out that boys in those two classrooms happen to be 34% better at reading? Those classrooms where boys did worse might have done worse for a large number of other reasons: those schools might be on the 'bad' side of town where less affluent students tend to go (happens all the time in small towns); that classroom might have been the one where the school has put all the trouble-maker kids because they know this particular teacher has the ability to teach those kinds of kids (or, conversely, that class might be the one that saw the huge boost when it was a game); the classes who did worse might have had a classmate dying in hospital that week, and the boys just didn't do very well under those conditions. There are too many other completely reasonable answers that aren't related to the test itself, given the methods used in this study. How I'd repeat the experiment: pool all the children before randomly selecting them for treatment groups to avoid bias from their origin classroom. Edit: And to be clear, I'm not saying the results are incorrect. They might be onto something amazing here. They just happen to be bad at statistics. Maybe they should frame it as a game.

iopq 11y ago
That's not how you count sample size. If there were four "classes" of one million boys and girls, does it still make the sample size four?
- mabbo 11y ago
  I think I'm not stating my point very well. Apologies for that. The problem with saying that n=80 is that it presumes each 'test' (student evaluated) was chosen for their treatment independently of each other. This is clearly not true, as entire classes were chosen together. So any variable like 'actually this happens to be the gifted class' can throw the whole study up or down because that variable was not independent between the students tested. For example, let's say we did this same test in America, but two classes happened to be from the wealthiest part of Boston, and two classes were from the poorest part of Detroit. If, as we randomly selected which class was assigned each treatment, we happened to hit the 1 in 4 chance that both wealthy classes were given the 'this is a game' treatment, would we be ready to say 'the game was the cause of the better test results'? I don't believe so. That is why I find this study flawed, and why I disagree with the idea that n=80. They may be onto something, but they do not have strong statistical evidence supporting their hypothesis.
  - iopq 11y ago
    That's not a sample size, that's called bias. The sample size is still 80, but it could be potentially a biased sample.