12 ms·

Why did the A-level algorithm say no?

jansan 6y ago
So is "controversial" the same as unfair or bad? Journalists often use "controversial" to discredit something that is not in accordance to their ideology. Controversial politicians (which politician isn't controversial?), controversial university professors, controversial speeches, controversial thoughts...
- chongli 6y ago
  It's controversial in the sense that a bunch of people are protesting about it. Whether it's actually unfair or bad is unknown. Fairness is a complicated thing. A lot of it is based on feeling and relative status, rather than some objective measure of well-being. One person might find it deeply unfair that another person has a lot of money, relatively speaking. But take that money away and redistribute it equally between the two and now the latter person may find the situation unfair.
  - jonp888 6y ago
    If what is written here is true then it's way beyond 'unfair' and on towards mathemtically nonsensicle: https://twitter.com/alexhern/status/1294200553661227008 https://twitter.com/alexhern/status/1294200553661227008 It is claimed that in any class where the chance of someone failing is judged to be > 0%, even if it is a tiny fraction of a percentage point, then the algorithm requires that someone in the class MUST be given a failing grade, even if it the weakest student in a very high performing class.
    - macspoofing 6y ago
      >If what is written here is true ... So is it? The guy is mining theoretical examples, but I wonder if in practice anyone has been hurt by this. And if they have (let's say a non-failing student was given a failing grade), why not handle those cases within a manual appeals process? After all, the exams weren't written, so no matter what mechanism you choose their will be students that get higher markers than they otherwise would have received and lower marker than they otherwise deserved - but I suspect, even in those cases, the marks will be in the same ballpark.
      - weego 6y ago
        Of course children have been hurt by this is practice, there are numerous cases of children getting down graded from their predicted results so badly they're having University offer places withdrawn, and it's not by a point or two, we're talking straight A students getting demotd to Ds across the board because their school has a history of bad results, which are irrelevant to any given student right now. Oxford have decided to ignore results and honor previously given offers because the process is so untrustworthy.
        macspoofing 6y ago
        >Of course children have been hurt by this is practice, there are numerous cases of children getting down graded from their predicted results so badly they're having University offer places withdrawn Yes. I understand there will be cases like this, but what numbers are we talking about? And you were never going to prevent circumstances like those. >we're talking straight A students getting demotd to Ds across the board because their school has a history of bad results, which are irrelevant to any given student right now. That's not irrelevant. Schools don't flip their standards from year or another. A bad school this year, will be a bad school next year. And many students are hurt by mark inflation. It isn't fair that one school will give out B, while another A+ for the same academic performance - and that certainly happens. There was a case in Canada some time ago where it came out that Universities were scoring high-schools based on performance of their alumni and normalizing the marks of new applicants based on those metrics. It was a small controversy, but what else are they going to do? The alternative is to assume that 'A' from one school is the same as 'A' from another - which is grossly unfair. >Oxford have decided to ignore results and honor previously given offers because the process is so untrustworthy. Good. Everyone understands that this is a weird time and everyone is trying their best. This is why this does not worry me one bit. I fully expect all schools that rely on these marks to be a little bit more understanding.
        pjc50 6y ago
        Marks aren't awarded by the school per se, they're awarded by one of the agencies such as AQA. This results in a lot of angry and upset teachers as the marks just received are a surprise to them too. The Scottish government SQA has already backed down on its "adjustment" and agreed to use estimated marks from teachers and mock exams.
  - weego 6y ago
    It's very clearly unfair, I'm not sure what your example has to do with anything. No students performance should be a sum based on how previous students have done at their school. That's literally reinforcing the postcode divide in education quality that were supposed to turn a blind eye to. My home town school has historically refused to even put children in for exams if they didn't get at least a C grade in their mocks because they valued league table position so highly (helps with investment and alumni donations etc). That cynical and should-be-illegal approach clearly benefits their students in this situation more than schools in the area that do right by their children. And more than any of that, this is the real world effects of the bullshit algorithm peddlers have been pushing for the last few years. Algorithms can solve any real world problem given the data. Well they can't. They can give the proposition of an answer to the problem, but at costs to some that are too high to bear.
    - deleted 6y ago
      [deleted]
    - oh_sigh 6y ago
      My high school had one person 3 years prior to my senior year get into an Ivy, and then before that it was no one in recent memory. In my senior class, 6 of us got into Ivies. I can only imagine what my life would be like if we only got the "fair" allotment of zero Ivy spots in a system like this.
      - stordoff 6y ago
        Similar happened to me (UK) - my school went from performing under the national average for the preceding three years to being notably above for my year group, and no one could remember anyone going to Oxbridge, but there were two of us in that year (the school barely even knew about the extra exams we had to take - required for one, optional for the other - so the prep. was more or less entirely self-taught).
- handelaar 6y ago
  In this case, yes it is. Low-performing students at private schools are having their estimated Cs and Ds bumped up to As. Stellar students at schools in poor areas who got As in their mock exams and have teacher-predicted As are being awarded Ds instead.
  - JoeSmithson 6y ago
    > "Low-performing students at private schools are having their estimated Cs and Ds bumped up to As" Do you have an example of this happening? It seems basically impossible from my understanding of the process. ----- Edit: I think I understand how it can happen. A school with a cohort that is small, but large enough to get the algorithm, and a previously exemplary results, has an unusually bad pupil. I can see this happening in a subject like Further Maths where only good students take it
- dynamite-ready 6y ago
  I wanted to use the word 'contentious', but I felt that would bend Hacker News rules. 'controversial' certainly feels like the right word in this case. The scheme is flawed, as it clearly penalises some students, and benefits others. And many are rightfully outraged by that. So yes. 'Controversial', I think.
Jabbles 6y ago
Do teachers recommend a grade, or a score? If they only recommend grades, how did the algorithm pick which students to change the grades for?
- Gaelan 6y ago
  I believe they recommend a grade, and rank the students within each recommended grade.
  - handelaar 6y ago
    And then, after that, this algorithm nonsensically applies an enormous upward or downward adjustment depending on what someone decided was the quality (read: prevalent socioeconomic status) of the school they attend.
- chongli 6y ago
  The algorithm took the teachers' recommended grades and used them only to rank the students at that school. The actual grades were calculated based on past years' performance for each school. So if you were the top performer at a low-ranking school, your grade would be in line with top performers at that school in previous years.
  - jarvist 6y ago
    No, teachers submitted both predicted grades, and in large cohort classes (such as English and Maths), a rank ordering of the entire cohort.
    - glomph 6y ago
      They had to submit ranking even if the class size was very small.
  - NLips 6y ago
    No. The system aimed for no nationwide grade inflation. When it then took a number of private school’s predictions at face value (no alteration if you paid for a small class size), meaning fewer high grades available for other schools. So lots of state schools had across the board drops from previous years.
pfortuny 6y ago
Well, there were no exams proper. The algorithm gives a mark for the A-levels.
bananaface 6y ago
[deleted]
- macspoofing 6y ago
  Who is to say your method is any more fair? There's nothing they can do to please everyone. Someone is going to benefit. Someone is going to lose. In the big picture, good students will end up with good marks, and bad students will end up with bad markets.
- pjc50 6y ago
  The A level is a two year course with assessment at the end. There are no previously examined grades, only estimated and mock ones. https://www.aqa.org.uk/subjects/science/as-and-a-level/chemistry-7404-7405/scheme-of-assessment https://www.aqa.org.uk/subjects/science/as-and-a-level/chemi... (Modular and coursework based A levels were popular under labour but reverted by the Tories)
  - dkarp 6y ago
    Wow, I didn’t realise it changed. What a horrible situation where a whole year group are reliant on their teachers’ opinion of them and the success of other kids at the same school the previous year. Neither of which these children have any direct control over.
    - mcguire 6y ago
      Eh, well, if they're the right sort of people, they'll catch it on the bounce.
  - bananaface 6y ago
    Huh! I didn't realise it had changed. We took exams in both years when I took it.
macspoofing 6y ago
A big pile of 'meh'. They are trying to make a statistical guess based on priors because the pandemic prevented the exams from being written. They are going to get things wrong. If they have an appeals process for things that are completely out line (e.g. a passing student getting a failing grade), then that's the best you can hope for given the circumstances. In the big picture, the good students are going to get good markers, average students are going to get average markets, and bad students will get bad markets. Again, it is what it is.
- alexanderskates 6y ago
  The problem is that the prior appears to be placed over the school, rather than the individual -- ie if your school had a low proportion of high achievers in previous years, students are finding their grades marked down, almost regardless of their performance. This results in particularly high grade disparity between independent and state schools. So it is not so much the case that good students get good marks, bad students get bad marks, but rather good schools get good marks, "bad" schools get bad marks.
  - macspoofing 6y ago
    Because mark inflation is a thing, and schools are different. There are schools that will give out A+ and others a B, for the same academic performance. School reputations don't change from year to year. So a school that inflates marks one year, will more than likely inflate them the next year. Are you suggesting that this should not be taken into consideration? That you should simply trust the relative weight of grade (relative to all the other schools) at face-value? >"bad" schools get bad marks. Why the quotes? There are schools that are at the bottom of academic rankings. That's a fact of reality. >students are finding their grades marked down, almost regardless of their performance. What numbers are we talking about here? No matter what algorithm or heuristic they chose, some unfairness was going to happen and my argument is that these cases are overstated. I'm sure they exist because we're talking about hundreds of thousands of students all in different circumstances. Having said that, in cases of egregious outcome, an appeals process would make sense. Also I'm sure many universities will take the pandemic into consideration and the fact that this was a best attempt at replicating standardizing test result without a test actually being taken.
corin_ 6y ago
Two things worth noting that the article doesn't go into... 1) "The year-on-year rise in proportion of students achieving A or A+ grades was much higher at independent schools than state comprehensives" https://www.theguardian.com/education/2020/aug/13/england-a-level-downgrades-hit-pupils-from-disadvantaged-areas-hardest https://www.theguardian.com/education/2020/aug/13/england-a-... Essentially, students lucky enough to be in private education were more likely to see their grades go up / stay the same, compared to students receiving free education seeing grades going down. Has been explained away as a byproduct of a sensible algorithm, but considering the government's track record and private school backgrounds (85% of the UK's Prime Ministers have been educated at Eton, arguably the country's most exclusive private school) it's hard not to be sceptical that this wasn't intentional and class-motivated. 2) "Appears that Ofqual's algorithm caused today's A-level chaos. Ofqual chair Roger Taylor, also chairs the Centre for Data Ethics & Innovation (CDEI). Cummings' fave AI consultants - Faculty, have some juicy contracts with CDEI. And Faculty's COO Richard Sargeant is on CDEI board" https://twitter.com/milesking10/status/1293886007771893762?s=21 https://twitter.com/milesking10/status/1293886007771893762?s... (Ofqual is the department that oversees exams and responsible for this situation, they report to Gavin Williams the government's Education Secretary. Cummings is a controversial special adverse to the Prime Minister, and Faculty is a firm he used when he was leading one of the two primary pro-Brexit campaigns before the referendum. Subjective statement: Cummings is a vile, evil man.)
- mjburgess 6y ago
  > it's hard not to be sceptical that this wasn't intentional and class-motivated. It's incredibly easy "not to be sceptical". The teacher's predicted grades are, on average, unprecedented. And represent many times greater than any previous yoy increase. This is far less true at top-performing schools where teachers are predicting far more consistent grades (by the very nature of being a top performing school). Teachers shouldn't have been asked to predict grades. That's the heart of the problem. Not some harebrained in-public conspiracy. This whole thing is dumb. And the media are deliberately failing to explain basic facts about the education system to provoke this kind of nonesense outrage.
  - 6y ago
naruciakk 6y ago
Couldn't the UK just have a regular exam, just with the social distancing measures implemented? The exams were held in many European countries with these measures and nothing bad happened and the children has received a fair examination for the University
- remus 6y ago
  Presumably it'd be very difficult to set such an exam because pupils will have missed so much of the normal curriculum.
  - sukilot 6y ago
    Why? High schoolers can read books and video chat.
    - onion2k 6y ago
      Children from poor backgrounds can't video chat because they don't have access to computers. So that's nonsense.
      - toyg 6y ago
        So instead their grades got nuked for a lifetime. Hurrah!
- willtim 6y ago
  Because opening pubs was the top priority of the government, not holding exams.
- polytely 6y ago
  My brother here in the Netherlands just got his average grade (of the test throughout the year that count toward the final grade) for each subject as final grade, which seems like it's fair enough.
  - 6y ago
wiredfool 6y ago
You can curve the scores on a meaningless exam and give the impression that some have learned and some have not, or I guess, you can skip the exam entirely and go to the end product. Pirsig would have something to say here.
gwern 6y ago
"In Scotland the accusations of unfairness prompted a switch to using teachers' predicted grades. These predictions were collected in England too - but were discounted as being the deciding factor, because they were so generous that it would have meant a huge increase in top grades, up to 38%. There were also doubts about the consistency and fairness of predictions and whether the cautious and realistic could have lost out to the ambitiously optimistic." As always, the question when using an algorithm or test is, 'compared to what?'
krisoft 6y ago
This was so foreseeable when they anounced the plans. One possible solution could have been if they do the same statistical estimation game and then offer a choice to each student: you can take your estimated grades or go for an exam. If you take the exam you get what you het there. It might go lower or higher than the estimate. This would have incentivised that only those who are certain of themselves and their estimated grades are insuficient towards their goals will go for the tests. And since these are outliers already there will be a lot less of them than the general population, thus it is easier to organise socially distanced exams for them. Of course nothing is easy at scale and many details need to be ironed out but thats what we keep the Miniszry of Education for.
- thom 6y ago
  The whole point is that it hasn’t been possible to sit exams.
  - sukilot 6y ago
    The world is still upside down. Run exams later and start college later.
  - Mirioron 6y ago
    Why not? Is it truly so impossible to spread students out a little more? Maybe don't do oral exams, but anything written shouldn't be an issue.
  - krisoft 6y ago
    If you take that as an axiom then sure. My point is that there are solutions other than “everyone sit exams as usual” and “everyone gets an estimated grade”.
  - toyg 6y ago
    Not possible to sit exams for everyone. A limited staged / “bubbled” approach in July/August could have worked fine. But it would require actual coordination and precision, and that’s not the sort of thing a Tory government of “big thinkers” can deliver.
Magnets 6y ago
https://www.ucas.com/file/292726/download?token=wswAnzge https://www.ucas.com/file/292726/download?token=wswAnzge "In 2019, 21% (31,220) of accepted 18 year old applicants met or exceeded their predicted grades, a decrease of 3 percentage points. In addition, 43.2% of accepted applicants had a difference of three or more A level grades – an increase of 5 percentage points (7,190 applicants more) since 2018." Teacher predicted grades are almost useless. Now Scotland has caved to political pressure and wants to inflate grades https://www.bbc.co.uk/news/uk-scotland-scotland-politics-53723734 https://www.bbc.co.uk/news/uk-scotland-scotland-politics-537... "However, these grades, taken overall, would represent a significant improvement on previous years - including a jump of 20 percentage points in the pass rate for pupils from the most deprived areas." "If the results had purely been based on the estimates from teachers, pass rates at grades A-C would have increased by 10.4 percentage points for National 5, by 14 percentage points for Higher and by 13.4 percentage points for Advanced Higher. These estimated results would have led to a higher annual change than had ever been seen before in Scottish exam results if they had not been moderated by the SQA." https://www.bbc.co.uk/news/uk-scotland-53636296 https://www.bbc.co.uk/news/uk-scotland-53636296 A really difficult situation when what you have to work from is already inaccurate.
tqi 6y ago
Are we to believe that teacher's predictions are less fraught with bias and inequity? (https://www.bbc.com/news/education-53776938 https://www.bbc.com/news/education-53776938)
Diggsey 6y ago
There's no good answer here, the unfairness is an inevitable side effect of lockdown, along with people dying due to stopped cancer treatments and a host of other issues. Using teacher's predicted grades is stupid: teachers use predictions in whatever way they think will best motivate their students. If the student is lazy, they'll under-predict to shock them into revising more. If the student is not very confident, they'll over-predict to encourage them. Unless the students are actually allowed to take an exam, all methods are unfair. Using mock results is at least tangentially related to the students' abilities.
- nelaboras 6y ago
  You are under the mistaken assumption that somehow exams are more fair or get a better view of what students can do. It's well established that exams measure a few limited congnitive constructs and its not the smartest or most hard working students that perform best, but rather those who are prepared by teachers for those exact constructs and the precise way of answering them. A brilliant student at a public school might study hard and still not get as good results as a mediocre and lazy student at a private school which tells them and drills them on the exam topics (often having their teachers in the exam boards and anyway paying more so that those teachers that care and know how yo play the game ate likely to be recruited there). It's absurd to think a central exam gives a more accurate picture of... What? Aptitude? Effort? ...? Than the teachers can have from working with students often for several years, seeing their work ethics, involvement in class, personal situations, etc. In short: the system was broken and biased to begin with (at least in England, Scotland is doing much better). Now this bias and favour of 'better' schools is explicitly enforced through this weighting algorithm.
  - harry8 6y ago
    Exams, being far, far from perfect /ARE/ more fair. Why? because it is the same exam for everyone. The person marking the exam has no information about the person who wrote the paper, not their name, not what they look like, not how rich they are, not how pretty they are, not what school they attend. Whether or not you can achieve better outcomes or get a better indication of capability with other means, that total anonymity and an absolutely common hurdle is a kind of fair that is tough to match in assessment. It shouldn't be under-vauled. We should be sure that any alternative really is better. Let's not pretend that teachers' aren't subject to unconscious bias. I've also met a few who were just horrible human beings who had no business being in teaching.
nelaboras 6y ago
An obvious point not discussed is that the exams system was already biased to begin with. Previous years were already unfair as great kids at bad schools have much fewer chances than less great kids for which the parents can pay a great school. England especially is awful at achieving a fair system (that gives everyone a fair chance at good grades) because schools are so stratified - by design of funding rules, etc.
- throwaway_pdp09 6y ago
  You're stating the obvious, a very well-known obvious. Perhaps you can suggest any system that is not intrinsically unfair (I'd be interested as I went both to schools in sink areas, and an expensive private school). (this was in the UK BTW) > ...is awful at achieving a fair system [...] because schools are so stratified - by design of funding rules etc. I'd say it was more down to the culture of parents in the poor areas mostly not giving a shit about their kid's education (and the roots of that aren't that the parents are intrinsically bad, it's a mindset of apathy inherited from their parents and the daily pressures that come with permanently worrying about money).
  - CaptainZapp 6y ago
    I'm not surprised that you're downvoted and may be into oblivion. But you actually make a very valid point. That said I also believe that the current UK government is far beyond any benefit of a doubt in terms of competence and maliciousnes.
    - throwaway_pdp09 6y ago
      I'd prefer constructive comebacks from people rather than downvotes. I can't learn from downvotes. But OMG the competence of the UK govt. It's like they literally can't do... anything without screwing it up. It is bizarre. Then they try to paper over it with words.
      - CaptainZapp 6y ago
HeavenFox 6y ago
Can someone explain to me the significance of A-level grades? If I understand Wikipedia correctly, students apply to universities using predicted grades anyways, and receive offers that are conditioned on the final grades. How about just make all offers unconditional? And if you are not going to universities, do anyone actually care about your A-level grades?
- corin_ 6y ago
  At least a couple of places, including one of Oxford Uni's colleges, have agreed to take anyone who was offered a conditional place, and there has been a hashtag trending on Twitter of #HonourTheOffer I expect many more to follow, so it's possible that the universities will clear up an awful lot of that mess - but while some academics are saying yes great bring them all in, others are strongly against the idea of expanding to let more people in this year. As to whether they're significant apart from for UCAS applications (university) - I'm not too sure. Personally I wouldn't care much about them on a CV of someone I was thinking of hiring in the best of years yet alone if I knew their results came from 2020. But with unemployment rising fast, and much worse soon once government gradually removes support offered over the summer to keep people furloughed (companies paid to have their staff not working but kept officially on the payroll until they could get back to work), we've already heard stories in some places or a shop or a pub advertising a job offer and getting 1000+ applications... and in a terrible job market, school exam grades could likely be a decider for who gets called to interview for many working class jobs. (They already are even outside a terrible job market, that just amplifies it.) edit: Oh, and while I don't know what proportion of people doing A Levels don't want to go to university (or want to go, but fail) - we're also a few days away from this same situation happening with the GCSE exams, which pretty much everyone legally has to do around age 16.
matthewheath 6y ago
While these events are terrible, I am pleased that the decisions made by Ofqual are open to judicial review—and that the Good Law Project are considering issuing a claim to force Ofqual to fix this mess.
lordnacho 6y ago
I've been listening to talk radio here in the UK, and it's a constant stream of disappointed kids/teachers/parents phoning in. Some of the stories are horrendous, people expecting AAA getting CCD and similar. It seems unlikely that you'd ever have three teachers predicting you to get top marks, but you end up with that on results day. But about the algorithm, it seems to be an impossible task. You cannot satisfy the requirement of the gross statistics being roughly the same, ie little grade inflation on average, while also identifying talented individuals. The outline of the algorithm seems to be that you basically take the gross stats for earlier years from a school, and then you take the teacher's guesses about who will do well, and make up a distribution of grades around that. This is probably what most people would do if forced into such an exercise. And you can add wrinkles to it like a bit of grade inflation, adjustment for overconfident predictions, and so on. But you can't get that piece of information that you really need, which is a mix of signal and noise about the true distribution of the kid's abilities. If you have a talented year in a low performing school, you'll never know. And vice versa. The year before I finished the IB, two kids got the top 45/45. That's normally achieved by something like 1/400 kids worldwide, and it was a class of maybe 30 kids. That would never have been awarded without examinations. In addition the other presumed goal, of having the grades as close to the predictions as possible, give the algo designer an incentive well known from machine learning: guess safely. Squish the distribution on the edges, that way you're not off much. This is bad for people who shoud be getting As, because many will get Bs. And it's good on the other end, though I don't know where the lump ends (C or D?).
Stierlitz 6y ago
“There have been two key pieces of information used to produce estimated grades: how students have been ranked in ability and how well their school or college has performed in exams in recent years.” “there was no direct connection between an individual's prior achievement and their predicted grade.” Who wrote this algorithm and can we see it?

So is "controversial" the same as unfair or bad? Journalists often use "controversial" to discredit something that is not in accordance to their ideology. Controversial politicians (which politician isn't controversial?), controversial university professors, controversial speeches, controversial thoughts...

chongli 6y ago
It's controversial in the sense that a bunch of people are protesting about it. Whether it's actually unfair or bad is unknown. Fairness is a complicated thing. A lot of it is based on feeling and relative status, rather than some objective measure of well-being. One person might find it deeply unfair that another person has a lot of money, relatively speaking. But take that money away and redistribute it equally between the two and now the latter person may find the situation unfair.
- jonp888 6y ago
  If what is written here is true then it's way beyond 'unfair' and on towards mathemtically nonsensicle: https://twitter.com/alexhern/status/1294200553661227008 https://twitter.com/alexhern/status/1294200553661227008 It is claimed that in any class where the chance of someone failing is judged to be > 0%, even if it is a tiny fraction of a percentage point, then the algorithm requires that someone in the class MUST be given a failing grade, even if it the weakest student in a very high performing class.
  - macspoofing 6y ago
    >If what is written here is true ... So is it? The guy is mining theoretical examples, but I wonder if in practice anyone has been hurt by this. And if they have (let's say a non-failing student was given a failing grade), why not handle those cases within a manual appeals process? After all, the exams weren't written, so no matter what mechanism you choose their will be students that get higher markers than they otherwise would have received and lower marker than they otherwise deserved - but I suspect, even in those cases, the marks will be in the same ballpark.
    - weego 6y ago
      Of course children have been hurt by this is practice, there are numerous cases of children getting down graded from their predicted results so badly they're having University offer places withdrawn, and it's not by a point or two, we're talking straight A students getting demotd to Ds across the board because their school has a history of bad results, which are irrelevant to any given student right now. Oxford have decided to ignore results and honor previously given offers because the process is so untrustworthy.
      - macspoofing 6y ago
        >Of course children have been hurt by this is practice, there are numerous cases of children getting down graded from their predicted results so badly they're having University offer places withdrawn Yes. I understand there will be cases like this, but what numbers are we talking about? And you were never going to prevent circumstances like those. >we're talking straight A students getting demotd to Ds across the board because their school has a history of bad results, which are irrelevant to any given student right now. That's not irrelevant. Schools don't flip their standards from year or another. A bad school this year, will be a bad school next year. And many students are hurt by mark inflation. It isn't fair that one school will give out B, while another A+ for the same academic performance - and that certainly happens. There was a case in Canada some time ago where it came out that Universities were scoring high-schools based on performance of their alumni and normalizing the marks of new applicants based on those metrics. It was a small controversy, but what else are they going to do? The alternative is to assume that 'A' from one school is the same as 'A' from another - which is grossly unfair. >Oxford have decided to ignore results and honor previously given offers because the process is so untrustworthy. Good. Everyone understands that this is a weird time and everyone is trying their best. This is why this does not worry me one bit. I fully expect all schools that rely on these marks to be a little bit more understanding.
        pjc50 6y ago
        Marks aren't awarded by the school per se, they're awarded by one of the agencies such as AQA. This results in a lot of angry and upset teachers as the marks just received are a surprise to them too. The Scottish government SQA has already backed down on its "adjustment" and agreed to use estimated marks from teachers and mock exams.
- weego 6y ago
  It's very clearly unfair, I'm not sure what your example has to do with anything. No students performance should be a sum based on how previous students have done at their school. That's literally reinforcing the postcode divide in education quality that were supposed to turn a blind eye to. My home town school has historically refused to even put children in for exams if they didn't get at least a C grade in their mocks because they valued league table position so highly (helps with investment and alumni donations etc). That cynical and should-be-illegal approach clearly benefits their students in this situation more than schools in the area that do right by their children. And more than any of that, this is the real world effects of the bullshit algorithm peddlers have been pushing for the last few years. Algorithms can solve any real world problem given the data. Well they can't. They can give the proposition of an answer to the problem, but at costs to some that are too high to bear.
  - deleted 6y ago
    [deleted]
  - oh_sigh 6y ago
    My high school had one person 3 years prior to my senior year get into an Ivy, and then before that it was no one in recent memory. In my senior class, 6 of us got into Ivies. I can only imagine what my life would be like if we only got the "fair" allotment of zero Ivy spots in a system like this.
    - stordoff 6y ago
      Similar happened to me (UK) - my school went from performing under the national average for the preceding three years to being notably above for my year group, and no one could remember anyone going to Oxbridge, but there were two of us in that year (the school barely even knew about the extra exams we had to take - required for one, optional for the other - so the prep. was more or less entirely self-taught).
handelaar 6y ago
In this case, yes it is. Low-performing students at private schools are having their estimated Cs and Ds bumped up to As. Stellar students at schools in poor areas who got As in their mock exams and have teacher-predicted As are being awarded Ds instead.
- JoeSmithson 6y ago
  > "Low-performing students at private schools are having their estimated Cs and Ds bumped up to As" Do you have an example of this happening? It seems basically impossible from my understanding of the process. ----- Edit: I think I understand how it can happen. A school with a cohort that is small, but large enough to get the algorithm, and a previously exemplary results, has an unusually bad pupil. I can see this happening in a subject like Further Maths where only good students take it
dynamite-ready 6y ago
I wanted to use the word 'contentious', but I felt that would bend Hacker News rules. 'controversial' certainly feels like the right word in this case. The scheme is flawed, as it clearly penalises some students, and benefits others. And many are rightfully outraged by that. So yes. 'Controversial', I think.

Jabbles 6y ago

Do teachers recommend a grade, or a score? If they only recommend grades, how did the algorithm pick which students to change the grades for?

Gaelan 6y ago
I believe they recommend a grade, and rank the students within each recommended grade.
- handelaar 6y ago
  And then, after that, this algorithm nonsensically applies an enormous upward or downward adjustment depending on what someone decided was the quality (read: prevalent socioeconomic status) of the school they attend.
chongli 6y ago
The algorithm took the teachers' recommended grades and used them only to rank the students at that school. The actual grades were calculated based on past years' performance for each school. So if you were the top performer at a low-ranking school, your grade would be in line with top performers at that school in previous years.
- jarvist 6y ago
  No, teachers submitted both predicted grades, and in large cohort classes (such as English and Maths), a rank ordering of the entire cohort.
  - glomph 6y ago
    They had to submit ranking even if the class size was very small.
- NLips 6y ago
  No. The system aimed for no nationwide grade inflation. When it then took a number of private school’s predictions at face value (no alteration if you paid for a small class size), meaning fewer high grades available for other schools. So lots of state schools had across the board drops from previous years.

pfortuny 6y ago

Well, there were no exams proper. The algorithm gives a mark for the A-levels.

bananaface 6y ago

[deleted]

macspoofing 6y ago
Who is to say your method is any more fair? There's nothing they can do to please everyone. Someone is going to benefit. Someone is going to lose. In the big picture, good students will end up with good marks, and bad students will end up with bad markets.
pjc50 6y ago
The A level is a two year course with assessment at the end. There are no previously examined grades, only estimated and mock ones. https://www.aqa.org.uk/subjects/science/as-and-a-level/chemistry-7404-7405/scheme-of-assessment https://www.aqa.org.uk/subjects/science/as-and-a-level/chemi... (Modular and coursework based A levels were popular under labour but reverted by the Tories)
- dkarp 6y ago
  Wow, I didn’t realise it changed. What a horrible situation where a whole year group are reliant on their teachers’ opinion of them and the success of other kids at the same school the previous year. Neither of which these children have any direct control over.
  - mcguire 6y ago
    Eh, well, if they're the right sort of people, they'll catch it on the bounce.
- bananaface 6y ago
  Huh! I didn't realise it had changed. We took exams in both years when I took it.

macspoofing 6y ago

A big pile of 'meh'. They are trying to make a statistical guess based on priors because the pandemic prevented the exams from being written. They are going to get things wrong. If they have an appeals process for things that are completely out line (e.g. a passing student getting a failing grade), then that's the best you can hope for given the circumstances. In the big picture, the good students are going to get good markers, average students are going to get average markets, and bad students will get bad markets. Again, it is what it is.

alexanderskates 6y ago
The problem is that the prior appears to be placed over the school, rather than the individual -- ie if your school had a low proportion of high achievers in previous years, students are finding their grades marked down, almost regardless of their performance. This results in particularly high grade disparity between independent and state schools. So it is not so much the case that good students get good marks, bad students get bad marks, but rather good schools get good marks, "bad" schools get bad marks.
- macspoofing 6y ago
  Because mark inflation is a thing, and schools are different. There are schools that will give out A+ and others a B, for the same academic performance. School reputations don't change from year to year. So a school that inflates marks one year, will more than likely inflate them the next year. Are you suggesting that this should not be taken into consideration? That you should simply trust the relative weight of grade (relative to all the other schools) at face-value? >"bad" schools get bad marks. Why the quotes? There are schools that are at the bottom of academic rankings. That's a fact of reality. >students are finding their grades marked down, almost regardless of their performance. What numbers are we talking about here? No matter what algorithm or heuristic they chose, some unfairness was going to happen and my argument is that these cases are overstated. I'm sure they exist because we're talking about hundreds of thousands of students all in different circumstances. Having said that, in cases of egregious outcome, an appeals process would make sense. Also I'm sure many universities will take the pandemic into consideration and the fact that this was a best attempt at replicating standardizing test result without a test actually being taken.

corin_ 6y ago

Two things worth noting that the article doesn't go into... 1) "The year-on-year rise in proportion of students achieving A or A+ grades was much higher at independent schools than state comprehensives" https://www.theguardian.com/education/2020/aug/13/england-a-level-downgrades-hit-pupils-from-disadvantaged-areas-hardest https://www.theguardian.com/education/2020/aug/13/england-a-... Essentially, students lucky enough to be in private education were more likely to see their grades go up / stay the same, compared to students receiving free education seeing grades going down. Has been explained away as a byproduct of a sensible algorithm, but considering the government's track record and private school backgrounds (85% of the UK's Prime Ministers have been educated at Eton, arguably the country's most exclusive private school) it's hard not to be sceptical that this wasn't intentional and class-motivated. 2) "Appears that Ofqual's algorithm caused today's A-level chaos. Ofqual chair Roger Taylor, also chairs the Centre for Data Ethics & Innovation (CDEI). Cummings' fave AI consultants - Faculty, have some juicy contracts with CDEI. And Faculty's COO Richard Sargeant is on CDEI board" https://twitter.com/milesking10/status/1293886007771893762?s=21 https://twitter.com/milesking10/status/1293886007771893762?s... (Ofqual is the department that oversees exams and responsible for this situation, they report to Gavin Williams the government's Education Secretary. Cummings is a controversial special adverse to the Prime Minister, and Faculty is a firm he used when he was leading one of the two primary pro-Brexit campaigns before the referendum. Subjective statement: Cummings is a vile, evil man.)

mjburgess 6y ago
> it's hard not to be sceptical that this wasn't intentional and class-motivated. It's incredibly easy "not to be sceptical". The teacher's predicted grades are, on average, unprecedented. And represent many times greater than any previous yoy increase. This is far less true at top-performing schools where teachers are predicting far more consistent grades (by the very nature of being a top performing school). Teachers shouldn't have been asked to predict grades. That's the heart of the problem. Not some harebrained in-public conspiracy. This whole thing is dumb. And the media are deliberately failing to explain basic facts about the education system to provoke this kind of nonesense outrage.
- 6y ago

naruciakk 6y ago

Couldn't the UK just have a regular exam, just with the social distancing measures implemented? The exams were held in many European countries with these measures and nothing bad happened and the children has received a fair examination for the University

remus 6y ago
Presumably it'd be very difficult to set such an exam because pupils will have missed so much of the normal curriculum.
- sukilot 6y ago
  Why? High schoolers can read books and video chat.
  - onion2k 6y ago
    Children from poor backgrounds can't video chat because they don't have access to computers. So that's nonsense.
    - toyg 6y ago
      So instead their grades got nuked for a lifetime. Hurrah!
willtim 6y ago
Because opening pubs was the top priority of the government, not holding exams.
polytely 6y ago
My brother here in the Netherlands just got his average grade (of the test throughout the year that count toward the final grade) for each subject as final grade, which seems like it's fair enough.
- 6y ago

wiredfool 6y ago

You can curve the scores on a meaningless exam and give the impression that some have learned and some have not, or I guess, you can skip the exam entirely and go to the end product. Pirsig would have something to say here.

gwern 6y ago

"In Scotland the accusations of unfairness prompted a switch to using teachers' predicted grades. These predictions were collected in England too - but were discounted as being the deciding factor, because they were so generous that it would have meant a huge increase in top grades, up to 38%. There were also doubts about the consistency and fairness of predictions and whether the cautious and realistic could have lost out to the ambitiously optimistic." As always, the question when using an algorithm or test is, 'compared to what?'

krisoft 6y ago

This was so foreseeable when they anounced the plans. One possible solution could have been if they do the same statistical estimation game and then offer a choice to each student: you can take your estimated grades or go for an exam. If you take the exam you get what you het there. It might go lower or higher than the estimate. This would have incentivised that only those who are certain of themselves and their estimated grades are insuficient towards their goals will go for the tests. And since these are outliers already there will be a lot less of them than the general population, thus it is easier to organise socially distanced exams for them. Of course nothing is easy at scale and many details need to be ironed out but thats what we keep the Miniszry of Education for.

thom 6y ago
The whole point is that it hasn’t been possible to sit exams.
- sukilot 6y ago
  The world is still upside down. Run exams later and start college later.
- Mirioron 6y ago
  Why not? Is it truly so impossible to spread students out a little more? Maybe don't do oral exams, but anything written shouldn't be an issue.
- krisoft 6y ago
  If you take that as an axiom then sure. My point is that there are solutions other than “everyone sit exams as usual” and “everyone gets an estimated grade”.
- toyg 6y ago
  Not possible to sit exams for everyone. A limited staged / “bubbled” approach in July/August could have worked fine. But it would require actual coordination and precision, and that’s not the sort of thing a Tory government of “big thinkers” can deliver.

Magnets 6y ago

https://www.ucas.com/file/292726/download?token=wswAnzge https://www.ucas.com/file/292726/download?token=wswAnzge "In 2019, 21% (31,220) of accepted 18 year old applicants met or exceeded their predicted grades, a decrease of 3 percentage points. In addition, 43.2% of accepted applicants had a difference of three or more A level grades – an increase of 5 percentage points (7,190 applicants more) since 2018." Teacher predicted grades are almost useless. Now Scotland has caved to political pressure and wants to inflate grades https://www.bbc.co.uk/news/uk-scotland-scotland-politics-53723734 https://www.bbc.co.uk/news/uk-scotland-scotland-politics-537... "However, these grades, taken overall, would represent a significant improvement on previous years - including a jump of 20 percentage points in the pass rate for pupils from the most deprived areas." "If the results had purely been based on the estimates from teachers, pass rates at grades A-C would have increased by 10.4 percentage points for National 5, by 14 percentage points for Higher and by 13.4 percentage points for Advanced Higher. These estimated results would have led to a higher annual change than had ever been seen before in Scottish exam results if they had not been moderated by the SQA." https://www.bbc.co.uk/news/uk-scotland-53636296 https://www.bbc.co.uk/news/uk-scotland-53636296 A really difficult situation when what you have to work from is already inaccurate.

tqi 6y ago

Are we to believe that teacher's predictions are less fraught with bias and inequity? (https://www.bbc.com/news/education-53776938 https://www.bbc.com/news/education-53776938)

Diggsey 6y ago

There's no good answer here, the unfairness is an inevitable side effect of lockdown, along with people dying due to stopped cancer treatments and a host of other issues. Using teacher's predicted grades is stupid: teachers use predictions in whatever way they think will best motivate their students. If the student is lazy, they'll under-predict to shock them into revising more. If the student is not very confident, they'll over-predict to encourage them. Unless the students are actually allowed to take an exam, all methods are unfair. Using mock results is at least tangentially related to the students' abilities.

nelaboras 6y ago
You are under the mistaken assumption that somehow exams are more fair or get a better view of what students can do. It's well established that exams measure a few limited congnitive constructs and its not the smartest or most hard working students that perform best, but rather those who are prepared by teachers for those exact constructs and the precise way of answering them. A brilliant student at a public school might study hard and still not get as good results as a mediocre and lazy student at a private school which tells them and drills them on the exam topics (often having their teachers in the exam boards and anyway paying more so that those teachers that care and know how yo play the game ate likely to be recruited there). It's absurd to think a central exam gives a more accurate picture of... What? Aptitude? Effort? ...? Than the teachers can have from working with students often for several years, seeing their work ethics, involvement in class, personal situations, etc. In short: the system was broken and biased to begin with (at least in England, Scotland is doing much better). Now this bias and favour of 'better' schools is explicitly enforced through this weighting algorithm.
- harry8 6y ago
  Exams, being far, far from perfect /ARE/ more fair. Why? because it is the same exam for everyone. The person marking the exam has no information about the person who wrote the paper, not their name, not what they look like, not how rich they are, not how pretty they are, not what school they attend. Whether or not you can achieve better outcomes or get a better indication of capability with other means, that total anonymity and an absolutely common hurdle is a kind of fair that is tough to match in assessment. It shouldn't be under-vauled. We should be sure that any alternative really is better. Let's not pretend that teachers' aren't subject to unconscious bias. I've also met a few who were just horrible human beings who had no business being in teaching.

nelaboras 6y ago

An obvious point not discussed is that the exams system was already biased to begin with. Previous years were already unfair as great kids at bad schools have much fewer chances than less great kids for which the parents can pay a great school. England especially is awful at achieving a fair system (that gives everyone a fair chance at good grades) because schools are so stratified - by design of funding rules, etc.

throwaway_pdp09 6y ago
You're stating the obvious, a very well-known obvious. Perhaps you can suggest any system that is not intrinsically unfair (I'd be interested as I went both to schools in sink areas, and an expensive private school). (this was in the UK BTW) > ...is awful at achieving a fair system [...] because schools are so stratified - by design of funding rules etc. I'd say it was more down to the culture of parents in the poor areas mostly not giving a shit about their kid's education (and the roots of that aren't that the parents are intrinsically bad, it's a mindset of apathy inherited from their parents and the daily pressures that come with permanently worrying about money).
- CaptainZapp 6y ago
  I'm not surprised that you're downvoted and may be into oblivion. But you actually make a very valid point. That said I also believe that the current UK government is far beyond any benefit of a doubt in terms of competence and maliciousnes.
  - throwaway_pdp09 6y ago
    I'd prefer constructive comebacks from people rather than downvotes. I can't learn from downvotes. But OMG the competence of the UK govt. It's like they literally can't do... anything without screwing it up. It is bizarre. Then they try to paper over it with words.
    - CaptainZapp 6y ago

HeavenFox 6y ago

Can someone explain to me the significance of A-level grades? If I understand Wikipedia correctly, students apply to universities using predicted grades anyways, and receive offers that are conditioned on the final grades. How about just make all offers unconditional? And if you are not going to universities, do anyone actually care about your A-level grades?

corin_ 6y ago
At least a couple of places, including one of Oxford Uni's colleges, have agreed to take anyone who was offered a conditional place, and there has been a hashtag trending on Twitter of #HonourTheOffer I expect many more to follow, so it's possible that the universities will clear up an awful lot of that mess - but while some academics are saying yes great bring them all in, others are strongly against the idea of expanding to let more people in this year. As to whether they're significant apart from for UCAS applications (university) - I'm not too sure. Personally I wouldn't care much about them on a CV of someone I was thinking of hiring in the best of years yet alone if I knew their results came from 2020. But with unemployment rising fast, and much worse soon once government gradually removes support offered over the summer to keep people furloughed (companies paid to have their staff not working but kept officially on the payroll until they could get back to work), we've already heard stories in some places or a shop or a pub advertising a job offer and getting 1000+ applications... and in a terrible job market, school exam grades could likely be a decider for who gets called to interview for many working class jobs. (They already are even outside a terrible job market, that just amplifies it.) edit: Oh, and while I don't know what proportion of people doing A Levels don't want to go to university (or want to go, but fail) - we're also a few days away from this same situation happening with the GCSE exams, which pretty much everyone legally has to do around age 16.

matthewheath 6y ago

While these events are terrible, I am pleased that the decisions made by Ofqual are open to judicial review—and that the Good Law Project are considering issuing a claim to force Ofqual to fix this mess.

lordnacho 6y ago

I've been listening to talk radio here in the UK, and it's a constant stream of disappointed kids/teachers/parents phoning in. Some of the stories are horrendous, people expecting AAA getting CCD and similar. It seems unlikely that you'd ever have three teachers predicting you to get top marks, but you end up with that on results day. But about the algorithm, it seems to be an impossible task. You cannot satisfy the requirement of the gross statistics being roughly the same, ie little grade inflation on average, while also identifying talented individuals. The outline of the algorithm seems to be that you basically take the gross stats for earlier years from a school, and then you take the teacher's guesses about who will do well, and make up a distribution of grades around that. This is probably what most people would do if forced into such an exercise. And you can add wrinkles to it like a bit of grade inflation, adjustment for overconfident predictions, and so on. But you can't get that piece of information that you really need, which is a mix of signal and noise about the true distribution of the kid's abilities. If you have a talented year in a low performing school, you'll never know. And vice versa. The year before I finished the IB, two kids got the top 45/45. That's normally achieved by something like 1/400 kids worldwide, and it was a class of maybe 30 kids. That would never have been awarded without examinations. In addition the other presumed goal, of having the grades as close to the predictions as possible, give the algo designer an incentive well known from machine learning: guess safely. Squish the distribution on the edges, that way you're not off much. This is bad for people who shoud be getting As, because many will get Bs. And it's good on the other end, though I don't know where the lump ends (C or D?).

Stierlitz 6y ago

“There have been two key pieces of information used to produce estimated grades: how students have been ranked in ability and how well their school or college has performed in exams in recent years.” “there was no direct connection between an individual's prior achievement and their predicted grade.” Who wrote this algorithm and can we see it?