4 ms·

I've been hearing loads of reports of kids getting marks way off what was predicted, and clearly the algorithm is... wrong. This essay is a good explanation of

I've been hearing loads of reports of kids getting marks way off what was predicted, and clearly the algorithm is... wrong. This essay is a good explanation of why. One thing that annoys me a bit is that we as a society have discovered all sorts of interesting thing about statistics and decision theory, causality, etc, but then we don't apply them. Why didn't Ofqual find some people who understood how stats work, and and ask them for advice? It seems clear that if you just take a distribution from previous years and lay them on top of a ranking, you are creating a lot of problems. And you are hiding your incentives on the penalty function, which is something he mentioned. People have correctly figured out that a good kid at a bad school will not be found, and that the marginally second best kid in a year is going to get marked down. Whatever you think of exams, and I'm no fan, they do give people that mix of noise and signal that society needs to put to rest the issue of who deserves what. They're not entirely a lottery, not entirely skill or hard work, but if we need a way to distribute X number of future doctors, that's the long established way to do it.

twic 6y ago
> Why didn't Ofqual find some people who understood how stats work, and and ask them for advice? They wouldn't sign a five-year NDA: https://news.sky.com/story/a-levels-exam-regulator-ignored-expert-help-after-statisticians-wouldnt-sign-non-disclosure-agreements-12049289 https://news.sky.com/story/a-levels-exam-regulator-ignored-e...
- disgruntledphd2 6y ago
  That is completely mental, a five year NDA for something that would be completed by now? The nicest interpretation of this is that Ofqual had no idea what the hell they were doing.
  - twic 6y ago
    I suppose what you don't want is to publish everyone's exam results, and then have one of your helpful academics say "oh, but of course these results are rubbish", and then have a massive public scandal about how the results are rubbish. However, what they opted for was the same thing, but without the second step.
whywhywhywhy 6y ago
> I've been hearing loads of reports of kids getting marks way off what was predicted Why do we need the second set of testing at all if the predictions exist and if the marking is going to be judged against them to the extent we are trusting the prediction over the marking.
teamonkey 6y ago
> Why didn't Ofqual find some people who understood how stats work, and and ask them for advice? Because, fundamentally, the current UK government is not interested in scientific evidence or taking advice from experts in an area (c.f. Michael Gove's famous line "people of this country have had enough of experts")
- throwaway45349 6y ago
  At least include the full quote, which is far more reasonable in-context: "I think the people of this country have had enough of experts with organisations with acronyms saying that they know what is best and getting it consistently wrong."
  - halostatue 6y ago
    Wouldn’t that apply to the Tories, though? They _consistently_ get it wrong after saying they know best.
    - Smaug123 6y ago
      But they don't profess to be experts.
      - mhh__ 6y ago
        Hence Dominic Cummings?
        Smaug123 6y ago
        Cummings explicitly claims not to be an expert, and decries the fact that he can't seem to find anyone competent at the higher levels of the civil service anywhere.
  - NLips 6y ago
    Not really - the second half is just rhetoric to add flavour. He wasn't saying "People have had enough of the subset of experts who are always wrong", not least because obviously an "expert" who is always wrong is unwanted.
Mvandenbergh 6y ago
>Why didn't Ofqual find some people who understood how stats work, and and ask them for advice? A very substantial number of the people who work at Ofqual are statisticians of course, it just seems that they have made some serious mistakes. I don't understand why they couldn't have published at least the algorithm for peer review and discussion months ago. These issues could have been resolved with plenty of time left before results had to be out. Another problem with this algorithm is that every year, a certain number of exam takers just fuck it up on an epic scale. They didn't get any sleep, they get a question that completely flummoxes them, whatever. So the distribution of grades, even at a good school with a pretty good intake, will have some very low grades on them. A big school will have a U or so. Now come to this year - noone had the opportunity to mess up their actual exam so some poor probably-C, B-on-a-good-day kid has to be stuffed into the "U" to make the distribution fit. Purely on the basis that the big school they went to usually has at least one. The difference is, if you fuck the exam up, you come out knowing that! So when you get your U, you know how it happened. This year someone has had it assigned to them without actually screwing up the exam which seems rather harsh.
alasdair_ 6y ago
>I've been hearing loads of reports of kids getting marks way off what was predicted, and clearly the algorithm is... wrong. It's worth noting that the predictions that were initially planned to be used as part of the algorithm would have resulted in 38% higher grades than any other year, ever. It's clear that teacher predictions are ALSO "wrong".

twic 6y ago

> Why didn't Ofqual find some people who understood how stats work, and and ask them for advice? They wouldn't sign a five-year NDA: https://news.sky.com/story/a-levels-exam-regulator-ignored-expert-help-after-statisticians-wouldnt-sign-non-disclosure-agreements-12049289 https://news.sky.com/story/a-levels-exam-regulator-ignored-e...

disgruntledphd2 6y ago
That is completely mental, a five year NDA for something that would be completed by now? The nicest interpretation of this is that Ofqual had no idea what the hell they were doing.
- twic 6y ago
  I suppose what you don't want is to publish everyone's exam results, and then have one of your helpful academics say "oh, but of course these results are rubbish", and then have a massive public scandal about how the results are rubbish. However, what they opted for was the same thing, but without the second step.

whywhywhywhy 6y ago

> I've been hearing loads of reports of kids getting marks way off what was predicted Why do we need the second set of testing at all if the predictions exist and if the marking is going to be judged against them to the extent we are trusting the prediction over the marking.

teamonkey 6y ago

> Why didn't Ofqual find some people who understood how stats work, and and ask them for advice? Because, fundamentally, the current UK government is not interested in scientific evidence or taking advice from experts in an area (c.f. Michael Gove's famous line "people of this country have had enough of experts")

throwaway45349 6y ago
At least include the full quote, which is far more reasonable in-context: "I think the people of this country have had enough of experts with organisations with acronyms saying that they know what is best and getting it consistently wrong."
- halostatue 6y ago
  Wouldn’t that apply to the Tories, though? They _consistently_ get it wrong after saying they know best.
  - Smaug123 6y ago
    But they don't profess to be experts.
    - mhh__ 6y ago
      Hence Dominic Cummings?
      - Smaug123 6y ago
        Cummings explicitly claims not to be an expert, and decries the fact that he can't seem to find anyone competent at the higher levels of the civil service anywhere.
- NLips 6y ago
  Not really - the second half is just rhetoric to add flavour. He wasn't saying "People have had enough of the subset of experts who are always wrong", not least because obviously an "expert" who is always wrong is unwanted.

Mvandenbergh 6y ago

>Why didn't Ofqual find some people who understood how stats work, and and ask them for advice? A very substantial number of the people who work at Ofqual are statisticians of course, it just seems that they have made some serious mistakes. I don't understand why they couldn't have published at least the algorithm for peer review and discussion months ago. These issues could have been resolved with plenty of time left before results had to be out. Another problem with this algorithm is that every year, a certain number of exam takers just fuck it up on an epic scale. They didn't get any sleep, they get a question that completely flummoxes them, whatever. So the distribution of grades, even at a good school with a pretty good intake, will have some very low grades on them. A big school will have a U or so. Now come to this year - noone had the opportunity to mess up their actual exam so some poor probably-C, B-on-a-good-day kid has to be stuffed into the "U" to make the distribution fit. Purely on the basis that the big school they went to usually has at least one. The difference is, if you fuck the exam up, you come out knowing that! So when you get your U, you know how it happened. This year someone has had it assigned to them without actually screwing up the exam which seems rather harsh.

alasdair_ 6y ago

>I've been hearing loads of reports of kids getting marks way off what was predicted, and clearly the algorithm is... wrong. It's worth noting that the predictions that were initially planned to be used as part of the algorithm would have resulted in 38% higher grades than any other year, ever. It's clear that teacher predictions are ALSO "wrong".