7 ms·

What the world can learn from the UK's A-level grading fiasco

39 points by ransith 6y ago

jonplackett 6y ago
The problem was not using an algorithm, it was that they used a crap algorithm that could award students grades that were higher than those even available on the paper that they sat, and made it impossible to get a high grade if no-one fromm your college got one before. I'd love to see the details of the algorithm itself. Surely it can't have been too difficult to put some guard rails in there to prevent it moving any grade more than say, two places (maybe even one) from the predicted grade. Instead it seemed to have the freedom to do whatever it liked. Moving Bs to fails. Students predicted to fail getting As. That's just incompetence on the part of whoever made that algorithm.
- londons_explore 6y ago
  The algorithm was the same one used for decades, and previously published. It pretty much says: > You get your exam grade. If that isn't available, you get a grade set by averaging your peers from the same school, with the set of peers decided by a ranking set by your teacher. If that isn't available, you get a grade chosen by your raking in class selected from the distribution of your schools past performance. There isn't really any fairer way that doesn't lead to grade inflation when there is an element of dishonesty/optimism on the part of teachers. We already suffer an element of grade inflation, which causes employers to say things like "You must achieve grade A* in Maths to apply for this job", and applicants from years ago before that grade was even introduced are automatically excluded.
  - nicoburns 6y ago
    > There isn't really any fairer way that doesn't lead to grade inflation when there is an element of dishonesty/optimism on the part of teachers. IMO it's much better just to swallow the grade inflation for a year. It will naturally reset itself next year (when exams will be sat again), and the negative effect from it are pretty minimal. The algorithm is based on the assumption that performance of successive school years at a given school is roughly consistent, not only as an average but as a distribution. And that assumption simply doesn't hold.
    - MaxBarraclough 6y ago
      > IMO it's much better just to swallow the grade inflation for a year You're not alone, this is the thinking they're using. [0] > the negative effect from it are pretty minimal That doesn't sound right. Top-flight students will be disadvantaged. [0] https://www.bbc.co.uk/news/uk-53810655 https://www.bbc.co.uk/news/uk-53810655
    - londons_explore 6y ago
      > IMO it's much better just to swallow the grade inflation for a year. When hiring, I'll be crossing off any qualifications earned in 2020. As well as the results being 'fake', the actual learning was in most cases not being done because the students knew they could chill at home and not learn. Sure, some students were studying, but as an employer I don't want to take that risk. Would you want your doctor to be the one who messed around on TikTok for the whole year of his final exams?
      - tonyedgecombe 6y ago
        >Would you want your doctor to be the one who messed around on TikTok for the whole year of his final exams? Well they still have six years of medical school to get through. Perhaps the dropout rate for that will be higher.
      - hahno 6y ago
        Learn how to be human, you poor inadequate and damaged failure
    - pbhjpbhj 6y ago
      >The algorithm is based on the assumption that performance of successive school years at a given school is roughly consistent, not only as an average but as a distribution. And that assumption simply doesn't hold. // It seems likely to hold pretty consistently. What's the variation?
  - chippy 6y ago
    When did the algorithm get used in the past? I imagine it being applied to individuals when they had an accident and missed the exams, but did it get applied to entire schools?
    - londons_explore 6y ago
      It got applied to entire schools if for example the school had to close due to a fire/flood. My school had it applied to one of my exam subjects for the entire year. I suspect the exam papers were lost/stole/destroyed after we took the exam, but before they were marked. Not sure how, but everyone was assigned 100%, and nobody complained...
  - MaxBarraclough 6y ago
    There's more to it. The system being used will lead to grade-inflation, [0] as there are two grade-estimation systems in use and the student gets to keep the better of the two. The two systems used are teachers' grade predictions, and the grade-prediction algorithm. We discussed this two weeks ago at https://news.ycombinator.com/item?id=24191882 https://news.ycombinator.com/item?id=24191882 [0] https://www.bbc.co.uk/news/uk-53810655 https://www.bbc.co.uk/news/uk-53810655
  - jonplackett 6y ago
    The algorithm you describe seems to have nothing in common with what has happened. There are students getting many grade levels above or below what they were predicted. That makes zero sense. Grade inflation is a completely different topic and has happened even more now that they've decided you can have whichever grade is higher out of predicted VS algorithm, grades are up something like 20% on last year.
- m4r35n357 6y ago
  A description of the algorithm and its shortcomings: http://thaines.com/post/alevels2020 http://thaines.com/post/alevels2020
  - m4r35n357 6y ago
    Unfortunate that the HN "algorithm" favours the views of various clueless gobshites over the useful and relevant link I posted above . . .
- iso947 6y ago
  Like all algorithms it had bias built in - one of which was to inflate private school grades, making grades even worse of a measure of quality than normal. What was particularly inexcusable though was the government response - the implications of the algorithm were available to the government weeks before the results, and the public backlash from Scotland having similar problems happened a day before. Now you could argue that to avoid grade inflation as a while the government should have proceeded regardless, however wen the inevitable backlash occurred (predictable since July and known since the Scottish exam results earlier in the month) they turned round. This u-turn came too late for many students to benefit yet wiped out all benefits of grade inflation prevention - literally the worst of both worlds.
- tonyedgecombe 6y ago
  and made it impossible to get a high grade if no-one fromm your college got one before Well that is unlikely. If you are going to handle it algorithmically then what else would you expect. Clearly the teachers can't be relied on because now everybody is getting an A*, the grades are effectively useless as an indicator. I was disappointed although not surprised that the government caved on this.
  - meheleventyone 6y ago
    Personally I’d expect assessment of an individual to be about their performance not their schools past performance. Any system not predicated on that is doomed to fail in assessing students performance as this one obviously did.
    - tonyedgecombe 6y ago
      Well that's what exams are for but they went out the window as soon as the virus arrived. Whatever system the govenment put in place would be worse but it didn't need to be this bad.
      - meheleventyone 6y ago
        They could and probably should have done exactly what ended up happening and relying on prior data per student. Imperfect though it was at least it's about their attainment rather than an aggregate attainment.
  - chippy 6y ago
    Two reasons: The results indicate that the algorithm was flawed: "Ofqual figures show 39.1% of 700,000 teacher assessments were lowered by at least one grade" Secondly, many students and teachers complained. This second reason is the thing that "the world can learn" that the article focuses on.
    - tonyedgecombe 6y ago
      Which can be explained by teachers assessments being too optimistic. Now we have reverted to that you can see the results, grade inflation. Of course students complained, I would if I didn't get the grades I was hoping for. That doesn't mean the government has to cave in and pretend everybody is a winner.
- feintruled 6y ago
  And don't forget the totally illogical situation of awarding someone a higher grade for advanced maths then they got for basic maths. I had a passing thought of a different system - using basically the same calculation of a school's predicted grades, allocate each school a 'budget' of marks they can distribute as they see fit, based on whatever criteria they pick. Just to be clear, I don't think that's a particularly good system either, but a canny government could have deployed it to avoid much of the direct criticism.
  - deleted 6y ago
    [deleted]
  - simonh 6y ago
    So then you'd get parents complaining about teachers picking their favourites and downgrading pupils for personal reasons, which still would have been the government's fault for implementing a system open to such abuse.
tokai 6y ago
That anyone ever thought it was a good idea boggles the mind. Random lot would have been better.
- anitil 6y ago
  Oh think of the natural experiment if you randomly assigned Universities to students. Awful, unethical, but fascinating
agent008t 6y ago
How did they deal with this problem in other countries, say Ireland or Japan?
- makomk 6y ago
  It looks to me like Ireland was planning on using literally the exact same process, but after it caused such a political problem for the UK government they announced that data on how schools had previously performed wouldn't be used in the standardisation process: https://www.citizensinformation.ie/en/education/state_examinations/leaving_certificate_2020_calculated_grades.html https://www.citizensinformation.ie/en/education/state_examin... (Notice how the linked explaination document from May 2020 says this would be the basis for the calculated results.) How that's meant to work I'm not sure. The results aren't out yet.
  - smcl 6y ago
    Additionally Scotland had an similar issue - algorithmically adjusted grade estimates were discovered to have punished kids from poorer schools. This happened a few weeks earlier than the English A-Level results, which is interesting because the ruling party in the rest of the UK: 1. took a GREAT deal of pleasure in loudly trashing the Scottish government in the press ... 2. knew they faced the EXACT same issue themselves ... 3. did absolutely nothing about it ... 4. then did a frantic cleanup of the attacks (deleting tweets and such) after the English debacle unfolded At this point I shouldn’t be surprised. It’s to be expected - they’re politicians playing their little game. But this debacle was avoidable - they could have faced the music earlier and started coming up with an alternative, but they chose instead to gloat, score a cheap win and kick the metaphorical can down the road.
- Barrin92 6y ago
  Japan for the most part just continued with their exams as usual. I think they delayed bar exams for a bit, but for the most part school activity has continued throughout the pandemic, disease burden in the country hasn't been that high.
lordnacho 6y ago
It's hard to see how the algo could do anything but disappoint. If you don't have the exam information, what information have you got left? Just results from previous years, and a guess as to where you might be in the class, from a teacher who doesn't want to disappoint anyone. But people have since forever seen exams as a way to stand out from the crowd, individually. If you went to a bad school, exams were a chance to show some extra effort. If you were a slacker, you'd be found out. Without the specific exam results, all you've got is the reputation of the school, and chances are variation within a school is greater than between.
makomk 6y ago
In reality, I think this fiasco has more to do with politics than algorithms. One of the big problems is that the alternatives had downsides which were largely glossed over for political reasons - for example the university growth cap isn't just an arbitrary creation, it exists to stop more prestigious universities siphoning off a disproportionate share of students and leaving less prestigious ones in deep trouble and this seems likely to happen now. Another big problem is that private schools seem to have made more realistic predictions of their students' results than state schools, as demonstrated by the fact that just taking many of their predictions as-is only caused slightly more grade inflation than in the state school results where 40% were downgraded - and this was portrayed in the press as proof that it was an attack on state schools because their results were disproportionately affected by the algorithmic downgrading. A third problem is that what it was OK to be concerned about varied depending on which political side it benefitted. For example, there was a completely false and made up claim on social media and in publications that should know better that the Education Secretary had said taking the results would cause students to be promoted into jobs that they weren't competent to do: https://fullfact.org/education/gavin-williamson-fake-quote/ https://fullfact.org/education/gavin-williamson-fake-quote/ I can't imagine that it would've gone down any better if he claimed students would get onto courses they weren't good enough for, yet a week later after GCSEs went the other way and used predicted results that exact claim was uncritically regurgitated by the media.
- throwaway287391 6y ago
  > the university growth cap isn't just an arbitrary creation, it exists to stop more prestigious universities siphoning off a disproportionate share of students and leaving less prestigious ones in deep trouble and this seems likely to happen now. I've never understood this UK policy personally. In the US there are no such caps and yet less-than-top-tier universities have had no trouble attracting plenty of students. Top tier US universities (e.g. Harvard), with gigantic endowments of tens of billions of dollars in some cases, could easily afford to expand their student bodies two-fold or probably even ten-fold, and yet they don't, despite having no regulations stopping them from doing so. These prestigious universities apparently feel more than enough market pressure not to expand very much, as they feel (rightly or not) that their prestige would take a hit by doing so. I don't understand what the point of creating additional regulatory pressure on top of this would be when e.g. Harvard expanding their student body could easily be a very positive life-changing event for so many students.
supernova87a 6y ago
This virus situation has (or will) produce a lost year of productivity. But people don't want to admit that and are still trying to operate under old constraints or processes. It will break somewhere. They want to keep schools running, and years / classes of students progressing through. But they also want to have some semblance of standards, without being able to teach properly or assess progress properly. All these factors cannot be simultaneously satisfied. As others have said, you might just have to drop standards and have a lottery, or open enrollment. Putting in these ridiculously bad algorithms is worse than falling back on random chance. High school/university is where it becomes easiest to hide behind bad logic and less obvious consequences. I hope we're not going to do the same for doctors, pilots, etc. and just let them pass the test because classes were canceled.
- simonh 6y ago
  It's not about passing the test though, it's about sorting out who gets to go to university and who doesn't. If the training of pilots one year was compromised so none of them could certified, we'd go a year without certifying any new pilots. We'd cope somehow. Universities can't go a year without any new students though, that's just not a reasonable outcome. On the face of it basing grades on predicted outcomes and adjusting for the historical record of specific schools to overestimate those predictions seems to be a pragmatic approach.
  - heavenlyblue 6y ago
    You can’t learn to fly without really flying (although that is also probably arguable - there should some skills which are transferable from simulators). However a lot of western higher education is based on giving enough info on how to study yourself.
- BlackVanilla 6y ago
  It will be interesting to see if this lost year materialises in students. It seems true that as students have had less contact time with a teacher, done less work, or set less-rigorous work, then average attainment will be lower. If students who missed half a year of good teaching are just as smart by the end of compulsory education than previous cohorts who were not affected by COVID, then this illustrates a massive failing of the education system. I have a feeling that this won't affect the attainment of students in the long-run, as the education system is flawed. Perhaps, in place of usual teaching, students are taking control of their own education and focus on their weak areas, rather than being told to copy down 10 pages from a textbook verbatim in a lesson standardised for the whole class. Perhaps this can be a much needed demonstration of the education system's weaknesses and an opportunity for change.
076ae80a-3c97-4 6y ago
The only true fair solution would be to put everyone back at least a year and wait until the exams can actually be done. In the mean time they could've invested the energy in getting an online exam system working and improve/innovate online learning mechanisms so that if we go back into lockdown again they are ready for it.
- simonh 6y ago
  I don't think doubling the size of the classes in the year below, and forcing schools to retain an extra year's worth of students is at all practical or would actually solve any problems. Let alone all the new problems it would create. What do Universities do if they have a whole year's worth of students just not show up, then suddenly gets two years worth come in at the same time? I suspect you've not actually thought this through.
_0o6v 6y ago
Having worked with engineers and software managers in the UK Government, it doesn't surprise me that this was such a fuckup. The solution would have been not to completely close schools in the first place - there was no evidence that children were particularly susceptible to COVID or at risk then, and there's lots of evidence now that they aren't. Exams could well have taken place as normal - exam halls are, by definition, socially distanced.
- jackpeterfletch 6y ago
  Yeah, Germany just ran their exams. Too important.
- Tarq0n 6y ago
  The children might not be at risk themselves but their parents/grandparents are, and kids do spread the virus.
- makomk 6y ago
  Not closing schools was the government's preferred option for exactly the reason you point out. Unfortunately, the teachers' unions had different views on the matter and it's rather difficult to keep schools open without teachers, especially when they can easily just carry out a mass sick-out. Exam halls are probably a bit of a transmission risk though since they involve a bunch of people who wouldn't normally mix sitting in the same room sharing the same air for a few hours.
  - nicky0 6y ago
    Also once the schools have been closed it becomes a lot harder to properly prepare the pupils for the exams.
    - smcl 6y ago
      You’re right and this is often ignored - the few weeks or so prior to exams where the curriculum is largely over can significantly change your grades
- 6y ago
rob74 6y ago
Well, what can we learn from this? That it's not a good idea to simply assign a grade to someone for an exam they didn't take, and it's an even worse idea to base this grade only partly on their own performance? And, if you are going to do it, you'd better make sure that the grades exceed the students' expectations, otherwise of course there will be protests...
LatteLazy 6y ago
We know that university admissions is a very poor, deeply flawed process. That people object this year but didn't object last year is very telling. It shows that people will happily accept a bad system with poor results AS LONG AS they are given some small illusion of personal control over that system. As long as people can say "people like me are very unlikely to do well BUT I will be fine because I will work harder than people in my group do" they're happy, even though they are by definition wrong most of the time.
- NovemberWhiskey 6y ago
  The students just wanted to be measured against the objective standard that they were promised. This doesn't require assumptions of naïveté on their part about equality of opportunity w.r.t teaching quality, schools resources and so on.
scast 6y ago
Serious question: Why weren't the exams taken? Everything I have read about this glosses over this critical bit which I am very curious about. I understand there is/was a worldwide pandemic and all of that, but it feels to me that there wasn't necessarily a reason exams couldn't be taken in socially-distanced class rooms (with students wearing masks, as well as the teachers, of course) or if needed be in repurposed venues, like football stadiums (leaving plenty of space between students). Was there a concern for fairness should the exams proceed? Or was it safety? Or did the government put their foot down on no exams? Or was it the teachers/unions? What happened?!
- Traster 6y ago
  Competence. In the next couple of weeks Schools are going to come back in the UK. There will be little to no real routines to limit the virus, and certainly no funding. That's 7 months into the pandemic. To put it simply, it was easier to just disregard the personal achievement of an entire generation than to spend money on actually sorting out a reasonable testing measure. Figuring out a real way of making sure exams were taken would've required competence.
- DaiPlusPlus 6y ago
  > it feels to me that there wasn't necessarily a reason exams couldn't be taken in socially-distanced class rooms (with students wearing masks, as well as the teachers, of course) or if needed be in repurposed venues, like football stadiums (leaving plenty of space between students). Exactly what you’re proposing just isn’t feasible, because of the sheer numbers. I see American posters on reddit complain about having to do a handful of finals at the end of high-school, and the SAT/etc as well, which I understand is optional. “In my day...” (wow I’m old... this was only about 12 years ago) my sixth-form had about 1,300 students (650 in yr12/L6, 650 in yr13/U6). Most L6 students did 5 AS-level subjects and 4 A2 subjects. Most subjects are then comprised of modules (e.g. In mathematics: Pure P1/P2/etc, Stats S1/S2/S3/etc, Mechanics M1/M2/etc, Decision/Discrete D1/D2/etc), and each module has its exam at the end of the term/trimester. In my case, I took 5 subjects at AS-level (L6) and had no less than 8 exams in the May-June of my first year. Multiply that by 1,300 - with probably over 100 different exams. That’s almost 11,000 Covid-safe exam-sittings that need to be arranged. Social-distancing regs mean that you need four times the floor-space for an exam room than previously (doubling distance from 3 feet to 6 feet in both directions). Doing exams outside on a field wouldn’t work: inclement weather and even a gentle breeze and writing on paper difficult and distracting. Doing it indoors means you’ll not only fill your sports-halls and cafeterias (which we did every year anyway) and need to spill-over into smaller classrooms - which means you need many more exam invigilators. ...and invigilators, in my experience, tended to be older people (60s-70s) - often recent retiree teachers. The exact same people who are quite legitimately fearing for their life over Covid so we can’t blame them for choosing to stay home. ...so we have a 2-month long period where schools and colleges need 4x the space and with far fewer authorised staff to oversee it. Some schools will be able to handle it, others won’t. If it’s a combination secondary-school + sixth-form then they’ll also need to handle GCSE exams for the yr11 kids and SATs (unrelated to the US SAT exam) for the kids in yr9. Additionally schools also have “mocks” for yr10 (mock-GCSEs, but they still count towards your score in yr11). There may also be additional testing done for other years at the county-level. So that’s another few thousand exam sittings to add to that. My secondary-school sent the Yr7 and yr8 kids home for a week if they were overloaded with handling exams for so many. So if the LEA/exam-boards were to press-on with the exams then that’s unfair to the kids at schools that don’t have the capacity. ——- Grading “by algorithm” - especially when that algorithm isn’t public - nor probably even we’ll-understood by the MPs in-charge - is a bad idea, yes. But I can’t think of a workable alternative: people’s lives should not depend on the outcome of exams - but we can’t trust teachers own subjective grading of their own students to be necessarily and sufficiently objective enough. There’s no economically-viable solution to this problem - even without a pandemic going on. —— Now that I think about it - I suppose one option would be still do in-person exams, but only do the core/essential exams for the most important course modules and use that as the basis for university admissions - so if this pandemic happened 12 years ago I’d only sit the P1/P2/P3/etc exams for mathematics and disregard Mechanics/Statistics/Discrete - ditto for physics, and so on. So the exam load would drop from my estimate of 11,000 to maybe 7,000-ish - I don’t think that would be small enough to manage still)
dang 6y ago
Recent and related: https://news.ycombinator.com/item?id=24185621 https://news.ycombinator.com/item?id=24185621. I feel like there may have been others?
renewiltord 6y ago
Why not just hold the tests, except outside and far apart. They have stadiums and stuff. Honestly, this whole thing seems like a manufactured problem. You could hold them outside, and stagger them so there are ten papers, and then just normal-curve-adjust to the first paper. Other countries already do multi-day tests easily. And if you don't want to do curving, just treat it the way you treat GREs/IELTS/TOEFL etc. Don't curve and just use overall percentile.
- oh_sigh 6y ago
  Maybe cheating? It would be easier to sneak a peek at your smartphone if one proctor has to cover an acre of students.
  - swyx 6y ago
    if you can cheat a test off a smartphone its probably not a good test
    - oh_sigh 6y ago
      Cheating via smartphone can encompass anything from directly looking up an answer, to sending someone the essay prompt and having them send you a completed essay which you just copy down. Can you name a good test by your standards?
    - deleted 6y ago
      [deleted]
  - renewiltord 6y ago
    That's fair. The ten tests thing still works. That's 1/10 the people occupying the same volume. Transmission is going to be minimal (or absent) if masked.
- meigwilym 6y ago
  > You could hold them outside Given British weather, this would be a desperate gample at best. > They have stadiums and stuff. I don't think even Eton have their own stadium. It might be worth you considering a different country's culture before generalising sweepingly.

39 points by ransith 6y ago

The problem was not using an algorithm, it was that they used a crap algorithm that could award students grades that were higher than those even available on the paper that they sat, and made it impossible to get a high grade if no-one fromm your college got one before. I'd love to see the details of the algorithm itself. Surely it can't have been too difficult to put some guard rails in there to prevent it moving any grade more than say, two places (maybe even one) from the predicted grade. Instead it seemed to have the freedom to do whatever it liked. Moving Bs to fails. Students predicted to fail getting As. That's just incompetence on the part of whoever made that algorithm.

londons_explore 6y ago
The algorithm was the same one used for decades, and previously published. It pretty much says: > You get your exam grade. If that isn't available, you get a grade set by averaging your peers from the same school, with the set of peers decided by a ranking set by your teacher. If that isn't available, you get a grade chosen by your raking in class selected from the distribution of your schools past performance. There isn't really any fairer way that doesn't lead to grade inflation when there is an element of dishonesty/optimism on the part of teachers. We already suffer an element of grade inflation, which causes employers to say things like "You must achieve grade A* in Maths to apply for this job", and applicants from years ago before that grade was even introduced are automatically excluded.
- nicoburns 6y ago
  > There isn't really any fairer way that doesn't lead to grade inflation when there is an element of dishonesty/optimism on the part of teachers. IMO it's much better just to swallow the grade inflation for a year. It will naturally reset itself next year (when exams will be sat again), and the negative effect from it are pretty minimal. The algorithm is based on the assumption that performance of successive school years at a given school is roughly consistent, not only as an average but as a distribution. And that assumption simply doesn't hold.
  - MaxBarraclough 6y ago
    > IMO it's much better just to swallow the grade inflation for a year You're not alone, this is the thinking they're using. [0] > the negative effect from it are pretty minimal That doesn't sound right. Top-flight students will be disadvantaged. [0] https://www.bbc.co.uk/news/uk-53810655 https://www.bbc.co.uk/news/uk-53810655
  - londons_explore 6y ago
    > IMO it's much better just to swallow the grade inflation for a year. When hiring, I'll be crossing off any qualifications earned in 2020. As well as the results being 'fake', the actual learning was in most cases not being done because the students knew they could chill at home and not learn. Sure, some students were studying, but as an employer I don't want to take that risk. Would you want your doctor to be the one who messed around on TikTok for the whole year of his final exams?
    - tonyedgecombe 6y ago
      >Would you want your doctor to be the one who messed around on TikTok for the whole year of his final exams? Well they still have six years of medical school to get through. Perhaps the dropout rate for that will be higher.
    - hahno 6y ago
      Learn how to be human, you poor inadequate and damaged failure
  - pbhjpbhj 6y ago
    >The algorithm is based on the assumption that performance of successive school years at a given school is roughly consistent, not only as an average but as a distribution. And that assumption simply doesn't hold. // It seems likely to hold pretty consistently. What's the variation?
- chippy 6y ago
  When did the algorithm get used in the past? I imagine it being applied to individuals when they had an accident and missed the exams, but did it get applied to entire schools?
  - londons_explore 6y ago
    It got applied to entire schools if for example the school had to close due to a fire/flood. My school had it applied to one of my exam subjects for the entire year. I suspect the exam papers were lost/stole/destroyed after we took the exam, but before they were marked. Not sure how, but everyone was assigned 100%, and nobody complained...
- MaxBarraclough 6y ago
  There's more to it. The system being used will lead to grade-inflation, [0] as there are two grade-estimation systems in use and the student gets to keep the better of the two. The two systems used are teachers' grade predictions, and the grade-prediction algorithm. We discussed this two weeks ago at https://news.ycombinator.com/item?id=24191882 https://news.ycombinator.com/item?id=24191882 [0] https://www.bbc.co.uk/news/uk-53810655 https://www.bbc.co.uk/news/uk-53810655
- jonplackett 6y ago
  The algorithm you describe seems to have nothing in common with what has happened. There are students getting many grade levels above or below what they were predicted. That makes zero sense. Grade inflation is a completely different topic and has happened even more now that they've decided you can have whichever grade is higher out of predicted VS algorithm, grades are up something like 20% on last year.
m4r35n357 6y ago
A description of the algorithm and its shortcomings: http://thaines.com/post/alevels2020 http://thaines.com/post/alevels2020
- m4r35n357 6y ago
  Unfortunate that the HN "algorithm" favours the views of various clueless gobshites over the useful and relevant link I posted above . . .
iso947 6y ago
Like all algorithms it had bias built in - one of which was to inflate private school grades, making grades even worse of a measure of quality than normal. What was particularly inexcusable though was the government response - the implications of the algorithm were available to the government weeks before the results, and the public backlash from Scotland having similar problems happened a day before. Now you could argue that to avoid grade inflation as a while the government should have proceeded regardless, however wen the inevitable backlash occurred (predictable since July and known since the Scottish exam results earlier in the month) they turned round. This u-turn came too late for many students to benefit yet wiped out all benefits of grade inflation prevention - literally the worst of both worlds.
tonyedgecombe 6y ago
and made it impossible to get a high grade if no-one fromm your college got one before Well that is unlikely. If you are going to handle it algorithmically then what else would you expect. Clearly the teachers can't be relied on because now everybody is getting an A*, the grades are effectively useless as an indicator. I was disappointed although not surprised that the government caved on this.
- meheleventyone 6y ago
  Personally I’d expect assessment of an individual to be about their performance not their schools past performance. Any system not predicated on that is doomed to fail in assessing students performance as this one obviously did.
  - tonyedgecombe 6y ago
    Well that's what exams are for but they went out the window as soon as the virus arrived. Whatever system the govenment put in place would be worse but it didn't need to be this bad.
    - meheleventyone 6y ago
      They could and probably should have done exactly what ended up happening and relying on prior data per student. Imperfect though it was at least it's about their attainment rather than an aggregate attainment.
- chippy 6y ago
  Two reasons: The results indicate that the algorithm was flawed: "Ofqual figures show 39.1% of 700,000 teacher assessments were lowered by at least one grade" Secondly, many students and teachers complained. This second reason is the thing that "the world can learn" that the article focuses on.
  - tonyedgecombe 6y ago
    Which can be explained by teachers assessments being too optimistic. Now we have reverted to that you can see the results, grade inflation. Of course students complained, I would if I didn't get the grades I was hoping for. That doesn't mean the government has to cave in and pretend everybody is a winner.
feintruled 6y ago
And don't forget the totally illogical situation of awarding someone a higher grade for advanced maths then they got for basic maths. I had a passing thought of a different system - using basically the same calculation of a school's predicted grades, allocate each school a 'budget' of marks they can distribute as they see fit, based on whatever criteria they pick. Just to be clear, I don't think that's a particularly good system either, but a canny government could have deployed it to avoid much of the direct criticism.
- deleted 6y ago
  [deleted]
- simonh 6y ago
  So then you'd get parents complaining about teachers picking their favourites and downgrading pupils for personal reasons, which still would have been the government's fault for implementing a system open to such abuse.

tokai 6y ago

That anyone ever thought it was a good idea boggles the mind. Random lot would have been better.

anitil 6y ago
Oh think of the natural experiment if you randomly assigned Universities to students. Awful, unethical, but fascinating

agent008t 6y ago

How did they deal with this problem in other countries, say Ireland or Japan?

makomk 6y ago
It looks to me like Ireland was planning on using literally the exact same process, but after it caused such a political problem for the UK government they announced that data on how schools had previously performed wouldn't be used in the standardisation process: https://www.citizensinformation.ie/en/education/state_examinations/leaving_certificate_2020_calculated_grades.html https://www.citizensinformation.ie/en/education/state_examin... (Notice how the linked explaination document from May 2020 says this would be the basis for the calculated results.) How that's meant to work I'm not sure. The results aren't out yet.
- smcl 6y ago
  Additionally Scotland had an similar issue - algorithmically adjusted grade estimates were discovered to have punished kids from poorer schools. This happened a few weeks earlier than the English A-Level results, which is interesting because the ruling party in the rest of the UK: 1. took a GREAT deal of pleasure in loudly trashing the Scottish government in the press ... 2. knew they faced the EXACT same issue themselves ... 3. did absolutely nothing about it ... 4. then did a frantic cleanup of the attacks (deleting tweets and such) after the English debacle unfolded At this point I shouldn’t be surprised. It’s to be expected - they’re politicians playing their little game. But this debacle was avoidable - they could have faced the music earlier and started coming up with an alternative, but they chose instead to gloat, score a cheap win and kick the metaphorical can down the road.
Barrin92 6y ago
Japan for the most part just continued with their exams as usual. I think they delayed bar exams for a bit, but for the most part school activity has continued throughout the pandemic, disease burden in the country hasn't been that high.

lordnacho 6y ago

It's hard to see how the algo could do anything but disappoint. If you don't have the exam information, what information have you got left? Just results from previous years, and a guess as to where you might be in the class, from a teacher who doesn't want to disappoint anyone. But people have since forever seen exams as a way to stand out from the crowd, individually. If you went to a bad school, exams were a chance to show some extra effort. If you were a slacker, you'd be found out. Without the specific exam results, all you've got is the reputation of the school, and chances are variation within a school is greater than between.

makomk 6y ago

In reality, I think this fiasco has more to do with politics than algorithms. One of the big problems is that the alternatives had downsides which were largely glossed over for political reasons - for example the university growth cap isn't just an arbitrary creation, it exists to stop more prestigious universities siphoning off a disproportionate share of students and leaving less prestigious ones in deep trouble and this seems likely to happen now. Another big problem is that private schools seem to have made more realistic predictions of their students' results than state schools, as demonstrated by the fact that just taking many of their predictions as-is only caused slightly more grade inflation than in the state school results where 40% were downgraded - and this was portrayed in the press as proof that it was an attack on state schools because their results were disproportionately affected by the algorithmic downgrading. A third problem is that what it was OK to be concerned about varied depending on which political side it benefitted. For example, there was a completely false and made up claim on social media and in publications that should know better that the Education Secretary had said taking the results would cause students to be promoted into jobs that they weren't competent to do: https://fullfact.org/education/gavin-williamson-fake-quote/ https://fullfact.org/education/gavin-williamson-fake-quote/ I can't imagine that it would've gone down any better if he claimed students would get onto courses they weren't good enough for, yet a week later after GCSEs went the other way and used predicted results that exact claim was uncritically regurgitated by the media.

throwaway287391 6y ago
> the university growth cap isn't just an arbitrary creation, it exists to stop more prestigious universities siphoning off a disproportionate share of students and leaving less prestigious ones in deep trouble and this seems likely to happen now. I've never understood this UK policy personally. In the US there are no such caps and yet less-than-top-tier universities have had no trouble attracting plenty of students. Top tier US universities (e.g. Harvard), with gigantic endowments of tens of billions of dollars in some cases, could easily afford to expand their student bodies two-fold or probably even ten-fold, and yet they don't, despite having no regulations stopping them from doing so. These prestigious universities apparently feel more than enough market pressure not to expand very much, as they feel (rightly or not) that their prestige would take a hit by doing so. I don't understand what the point of creating additional regulatory pressure on top of this would be when e.g. Harvard expanding their student body could easily be a very positive life-changing event for so many students.

supernova87a 6y ago

This virus situation has (or will) produce a lost year of productivity. But people don't want to admit that and are still trying to operate under old constraints or processes. It will break somewhere. They want to keep schools running, and years / classes of students progressing through. But they also want to have some semblance of standards, without being able to teach properly or assess progress properly. All these factors cannot be simultaneously satisfied. As others have said, you might just have to drop standards and have a lottery, or open enrollment. Putting in these ridiculously bad algorithms is worse than falling back on random chance. High school/university is where it becomes easiest to hide behind bad logic and less obvious consequences. I hope we're not going to do the same for doctors, pilots, etc. and just let them pass the test because classes were canceled.

simonh 6y ago
It's not about passing the test though, it's about sorting out who gets to go to university and who doesn't. If the training of pilots one year was compromised so none of them could certified, we'd go a year without certifying any new pilots. We'd cope somehow. Universities can't go a year without any new students though, that's just not a reasonable outcome. On the face of it basing grades on predicted outcomes and adjusting for the historical record of specific schools to overestimate those predictions seems to be a pragmatic approach.
- heavenlyblue 6y ago
  You can’t learn to fly without really flying (although that is also probably arguable - there should some skills which are transferable from simulators). However a lot of western higher education is based on giving enough info on how to study yourself.
BlackVanilla 6y ago
It will be interesting to see if this lost year materialises in students. It seems true that as students have had less contact time with a teacher, done less work, or set less-rigorous work, then average attainment will be lower. If students who missed half a year of good teaching are just as smart by the end of compulsory education than previous cohorts who were not affected by COVID, then this illustrates a massive failing of the education system. I have a feeling that this won't affect the attainment of students in the long-run, as the education system is flawed. Perhaps, in place of usual teaching, students are taking control of their own education and focus on their weak areas, rather than being told to copy down 10 pages from a textbook verbatim in a lesson standardised for the whole class. Perhaps this can be a much needed demonstration of the education system's weaknesses and an opportunity for change.

076ae80a-3c97-4 6y ago

The only true fair solution would be to put everyone back at least a year and wait until the exams can actually be done. In the mean time they could've invested the energy in getting an online exam system working and improve/innovate online learning mechanisms so that if we go back into lockdown again they are ready for it.

simonh 6y ago
I don't think doubling the size of the classes in the year below, and forcing schools to retain an extra year's worth of students is at all practical or would actually solve any problems. Let alone all the new problems it would create. What do Universities do if they have a whole year's worth of students just not show up, then suddenly gets two years worth come in at the same time? I suspect you've not actually thought this through.

_0o6v 6y ago

Having worked with engineers and software managers in the UK Government, it doesn't surprise me that this was such a fuckup. The solution would have been not to completely close schools in the first place - there was no evidence that children were particularly susceptible to COVID or at risk then, and there's lots of evidence now that they aren't. Exams could well have taken place as normal - exam halls are, by definition, socially distanced.

jackpeterfletch 6y ago
Yeah, Germany just ran their exams. Too important.
Tarq0n 6y ago
The children might not be at risk themselves but their parents/grandparents are, and kids do spread the virus.
makomk 6y ago
Not closing schools was the government's preferred option for exactly the reason you point out. Unfortunately, the teachers' unions had different views on the matter and it's rather difficult to keep schools open without teachers, especially when they can easily just carry out a mass sick-out. Exam halls are probably a bit of a transmission risk though since they involve a bunch of people who wouldn't normally mix sitting in the same room sharing the same air for a few hours.
- nicky0 6y ago
  Also once the schools have been closed it becomes a lot harder to properly prepare the pupils for the exams.
  - smcl 6y ago
    You’re right and this is often ignored - the few weeks or so prior to exams where the curriculum is largely over can significantly change your grades
6y ago

rob74 6y ago

Well, what can we learn from this? That it's not a good idea to simply assign a grade to someone for an exam they didn't take, and it's an even worse idea to base this grade only partly on their own performance? And, if you are going to do it, you'd better make sure that the grades exceed the students' expectations, otherwise of course there will be protests...

LatteLazy 6y ago

We know that university admissions is a very poor, deeply flawed process. That people object this year but didn't object last year is very telling. It shows that people will happily accept a bad system with poor results AS LONG AS they are given some small illusion of personal control over that system. As long as people can say "people like me are very unlikely to do well BUT I will be fine because I will work harder than people in my group do" they're happy, even though they are by definition wrong most of the time.

NovemberWhiskey 6y ago
The students just wanted to be measured against the objective standard that they were promised. This doesn't require assumptions of naïveté on their part about equality of opportunity w.r.t teaching quality, schools resources and so on.

scast 6y ago

Serious question: Why weren't the exams taken? Everything I have read about this glosses over this critical bit which I am very curious about. I understand there is/was a worldwide pandemic and all of that, but it feels to me that there wasn't necessarily a reason exams couldn't be taken in socially-distanced class rooms (with students wearing masks, as well as the teachers, of course) or if needed be in repurposed venues, like football stadiums (leaving plenty of space between students). Was there a concern for fairness should the exams proceed? Or was it safety? Or did the government put their foot down on no exams? Or was it the teachers/unions? What happened?!

Traster 6y ago
Competence. In the next couple of weeks Schools are going to come back in the UK. There will be little to no real routines to limit the virus, and certainly no funding. That's 7 months into the pandemic. To put it simply, it was easier to just disregard the personal achievement of an entire generation than to spend money on actually sorting out a reasonable testing measure. Figuring out a real way of making sure exams were taken would've required competence.
DaiPlusPlus 6y ago
> it feels to me that there wasn't necessarily a reason exams couldn't be taken in socially-distanced class rooms (with students wearing masks, as well as the teachers, of course) or if needed be in repurposed venues, like football stadiums (leaving plenty of space between students). Exactly what you’re proposing just isn’t feasible, because of the sheer numbers. I see American posters on reddit complain about having to do a handful of finals at the end of high-school, and the SAT/etc as well, which I understand is optional. “In my day...” (wow I’m old... this was only about 12 years ago) my sixth-form had about 1,300 students (650 in yr12/L6, 650 in yr13/U6). Most L6 students did 5 AS-level subjects and 4 A2 subjects. Most subjects are then comprised of modules (e.g. In mathematics: Pure P1/P2/etc, Stats S1/S2/S3/etc, Mechanics M1/M2/etc, Decision/Discrete D1/D2/etc), and each module has its exam at the end of the term/trimester. In my case, I took 5 subjects at AS-level (L6) and had no less than 8 exams in the May-June of my first year. Multiply that by 1,300 - with probably over 100 different exams. That’s almost 11,000 Covid-safe exam-sittings that need to be arranged. Social-distancing regs mean that you need four times the floor-space for an exam room than previously (doubling distance from 3 feet to 6 feet in both directions). Doing exams outside on a field wouldn’t work: inclement weather and even a gentle breeze and writing on paper difficult and distracting. Doing it indoors means you’ll not only fill your sports-halls and cafeterias (which we did every year anyway) and need to spill-over into smaller classrooms - which means you need many more exam invigilators. ...and invigilators, in my experience, tended to be older people (60s-70s) - often recent retiree teachers. The exact same people who are quite legitimately fearing for their life over Covid so we can’t blame them for choosing to stay home. ...so we have a 2-month long period where schools and colleges need 4x the space and with far fewer authorised staff to oversee it. Some schools will be able to handle it, others won’t. If it’s a combination secondary-school + sixth-form then they’ll also need to handle GCSE exams for the yr11 kids and SATs (unrelated to the US SAT exam) for the kids in yr9. Additionally schools also have “mocks” for yr10 (mock-GCSEs, but they still count towards your score in yr11). There may also be additional testing done for other years at the county-level. So that’s another few thousand exam sittings to add to that. My secondary-school sent the Yr7 and yr8 kids home for a week if they were overloaded with handling exams for so many. So if the LEA/exam-boards were to press-on with the exams then that’s unfair to the kids at schools that don’t have the capacity. ——- Grading “by algorithm” - especially when that algorithm isn’t public - nor probably even we’ll-understood by the MPs in-charge - is a bad idea, yes. But I can’t think of a workable alternative: people’s lives should not depend on the outcome of exams - but we can’t trust teachers own subjective grading of their own students to be necessarily and sufficiently objective enough. There’s no economically-viable solution to this problem - even without a pandemic going on. —— Now that I think about it - I suppose one option would be still do in-person exams, but only do the core/essential exams for the most important course modules and use that as the basis for university admissions - so if this pandemic happened 12 years ago I’d only sit the P1/P2/P3/etc exams for mathematics and disregard Mechanics/Statistics/Discrete - ditto for physics, and so on. So the exam load would drop from my estimate of 11,000 to maybe 7,000-ish - I don’t think that would be small enough to manage still)

dang 6y ago

Recent and related: https://news.ycombinator.com/item?id=24185621 https://news.ycombinator.com/item?id=24185621. I feel like there may have been others?

renewiltord 6y ago

Why not just hold the tests, except outside and far apart. They have stadiums and stuff. Honestly, this whole thing seems like a manufactured problem. You could hold them outside, and stagger them so there are ten papers, and then just normal-curve-adjust to the first paper. Other countries already do multi-day tests easily. And if you don't want to do curving, just treat it the way you treat GREs/IELTS/TOEFL etc. Don't curve and just use overall percentile.

oh_sigh 6y ago
Maybe cheating? It would be easier to sneak a peek at your smartphone if one proctor has to cover an acre of students.
- swyx 6y ago
  if you can cheat a test off a smartphone its probably not a good test
  - oh_sigh 6y ago
    Cheating via smartphone can encompass anything from directly looking up an answer, to sending someone the essay prompt and having them send you a completed essay which you just copy down. Can you name a good test by your standards?
  - deleted 6y ago
    [deleted]
- renewiltord 6y ago
  That's fair. The ten tests thing still works. That's 1/10 the people occupying the same volume. Transmission is going to be minimal (or absent) if masked.
meigwilym 6y ago
> You could hold them outside Given British weather, this would be a desperate gample at best. > They have stadiums and stuff. I don't think even Eton have their own stadium. It might be worth you considering a different country's culture before generalising sweepingly.