5 ms·

Do teachers recommend a grade, or a score? If they only recommend grades, how did the algorithm pick which students to change the grades for?

Gaelan 6y ago
I believe they recommend a grade, and rank the students within each recommended grade.
- handelaar 6y ago
  And then, after that, this algorithm nonsensically applies an enormous upward or downward adjustment depending on what someone decided was the quality (read: prevalent socioeconomic status) of the school they attend.
chongli 6y ago
The algorithm took the teachers' recommended grades and used them only to rank the students at that school. The actual grades were calculated based on past years' performance for each school. So if you were the top performer at a low-ranking school, your grade would be in line with top performers at that school in previous years.
- jarvist 6y ago
  No, teachers submitted both predicted grades, and in large cohort classes (such as English and Maths), a rank ordering of the entire cohort.
  - glomph 6y ago
    They had to submit ranking even if the class size was very small.
- NLips 6y ago
  No. The system aimed for no nationwide grade inflation. When it then took a number of private school’s predictions at face value (no alteration if you paid for a small class size), meaning fewer high grades available for other schools. So lots of state schools had across the board drops from previous years.
marksomnian 6y ago
As I understood it: 1. Teachers submit both an estimated grade and a relative rank ordering for each student 2. Ofqual calculate a distribution based on the predicted grades 3. They compare it to grade distributions at that school from previous years 4. If need be, they shift the distribution up or down to be roughly in line with that school's prior results 5. They then use the rank ordering to assign students grades from that distribution (e.g. in a class with 20 students, each student is worth 5% of the distribution - so if it predicted 10% A* (highest grade), 15% A, etc., the top 2 students would get an A*, the next 3 an A and so on)
- jarvist 6y ago
  It's more complicated than this. They haven't just taken the rank ordering by the teachers and projected onto a curve. Here is the 319 page document published on results day. This is the totality of the current description of the 'algorithm'. https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/909368/6656-1_Awarding_GCSE__AS__A_level__advanced_extension_awards_and_extended_project_qualifications_in_summer_2020_-_interim_report.pdf https://assets.publishing.service.gov.uk/government/uploads/... There's some kind of predictive linear model (page 40) which is based on a student's prior attainment (i.e. GCSE and SATs 'data'), but then modelled at the level of the exam centre (i.e. school). These models are separately fitted for different bins (I think ten are shown at one point) of achievement. The whole document is pretty incoherent. I suspect the 'algorithm' is a set of excel spreadsheets.
Twirrim 6y ago
Here's a write-up of the situation, shared on Facebook publicly, from a friend involved in careers at a UK Sixth Form college (where students take their A-Levels): ################### I've spent the last two days dealing with very upset young people who have had their plans ripped up. Some will still be able to go to university, but not the one they planned on and have been forced into clearing, some won't be able to go at all. So the story behind this is as follows. Teachers at colleges and schools submitted grades to exam boards earlier in the summer based on their predictions as to how students would do in exams they would not take due to C19. The exam boards then mauled these with an algorithm. Our current understanding is that subject cohorts of less than 10 have not had the algorithm applied and 15-20 only slightly. Most private and many small Sixth forms in schools fall in to this category, (note this is students in a subject at a provider not total numbers at that provider) Some centre assessed grades (CAGs) will have been assigned optomistically (inflated) by teachers in an attempt to advantage them and their students but the vast majority will have been genuine assessments based on submitted work and teacher experience. They had no idea just how much adjustment would be applied, I'm sure if they had them they might not have been quite so honest! [Edit: a colleague has pointed out that sometimes predicted grades are higher than achieved because there are unanticipated issues very close to exams such as illness, bereavement a stress which are not evident in year. This cannot be predicted by teachers and compensated for in advance, although many schools will have tried to adjust for students who experienced issues in year] To compensate for this small amount of ingrained optimisation and unpredictability the exam boards have moderated down larger schools CAGs (but not the smaller cohorts). This includes many sixth form colleges with thousands of exam entries. Sixth form colleges and larger sixth forms are very good at estimating grades and have been extremely thorough in their efforts to assign realistic grades. Our own teachers spent weeks compiling grades from previous work and tests to get the most accurate assessment they could. No one wanted their students to miss out, no one underestimated their CAGs. The exam boards were not going to stand for a big increase in grades this year so they applied an aggressive algorithm to the larger subject cohorts. Our college for example has hundreds of A level maths students which makes us a target for harsh moderation despite having increadable value added and a history of very accurate (not over inflated) predictions; vast numbers of our students have seen their grades dropped by one or two grade boundaries, I have only seen one moderated up. Many of our students have missed out on university places. Despite positive news stories about one Oxbridge University College most universities have not moderated their offers in light of this crisis and many students at private schools will have been able to swoop in on those places and adjust upwards to higher tarrif courses. Now, whether you think this is a government fix up or not; it is not fair on poorer students at bigger schools and coincidentally some more 'privileged' students have been given another leg up. Talk to your MP talk about it on social media and lobby to get this fixed because it's going to mess up a lot of young people's futures. These young people are now voters and I hope they will be registering their concern with their local MPs which may motivate them to put pressure on the minister and in turn the exam boards If you think this was a fiasco then hold on tight because next weeks GCSE results will be based on less information and are likely to be even more skewed! We will hit a second crisis of students not being able to access A levels and colleges having to juggle the same situation as universities are this week. Many GCSE students will be awarded grades which either over inflate or under play their ability. Many will lose an A level place and others will end up on courses which they are 'qualified' for but are overwhelmingly too hard for them, causing distress and self esteem issues. This is on top of the emerging increase in youth mental health issues due to C19 and the lockdown. If you feel the government's last minute addition of the 'tripple lock' and free appeals makes everything OK then I have disappointing news for you. Mock exams are internally administered in a variety of different ways. Allowing the use of mocks rather than CAGs will lead to even more disparity. September exams for students, out of education and in lock down for 6 months with no access to resources and who thought they had finished their A levels? I'll leave you to draw your own conclusions about whether that will be helpful. We in education will continue, we will pick up the pieces, we will do everything we can to make sure these young people have the best outcomes they can. ###################

Gaelan 6y ago

I believe they recommend a grade, and rank the students within each recommended grade.

handelaar 6y ago
And then, after that, this algorithm nonsensically applies an enormous upward or downward adjustment depending on what someone decided was the quality (read: prevalent socioeconomic status) of the school they attend.

chongli 6y ago

The algorithm took the teachers' recommended grades and used them only to rank the students at that school. The actual grades were calculated based on past years' performance for each school. So if you were the top performer at a low-ranking school, your grade would be in line with top performers at that school in previous years.

jarvist 6y ago
No, teachers submitted both predicted grades, and in large cohort classes (such as English and Maths), a rank ordering of the entire cohort.
- glomph 6y ago
  They had to submit ranking even if the class size was very small.
NLips 6y ago
No. The system aimed for no nationwide grade inflation. When it then took a number of private school’s predictions at face value (no alteration if you paid for a small class size), meaning fewer high grades available for other schools. So lots of state schools had across the board drops from previous years.

marksomnian 6y ago

As I understood it: 1. Teachers submit both an estimated grade and a relative rank ordering for each student 2. Ofqual calculate a distribution based on the predicted grades 3. They compare it to grade distributions at that school from previous years 4. If need be, they shift the distribution up or down to be roughly in line with that school's prior results 5. They then use the rank ordering to assign students grades from that distribution (e.g. in a class with 20 students, each student is worth 5% of the distribution - so if it predicted 10% A* (highest grade), 15% A, etc., the top 2 students would get an A*, the next 3 an A and so on)

jarvist 6y ago
It's more complicated than this. They haven't just taken the rank ordering by the teachers and projected onto a curve. Here is the 319 page document published on results day. This is the totality of the current description of the 'algorithm'. https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/909368/6656-1_Awarding_GCSE__AS__A_level__advanced_extension_awards_and_extended_project_qualifications_in_summer_2020_-_interim_report.pdf https://assets.publishing.service.gov.uk/government/uploads/... There's some kind of predictive linear model (page 40) which is based on a student's prior attainment (i.e. GCSE and SATs 'data'), but then modelled at the level of the exam centre (i.e. school). These models are separately fitted for different bins (I think ten are shown at one point) of achievement. The whole document is pretty incoherent. I suspect the 'algorithm' is a set of excel spreadsheets.

Twirrim 6y ago

Here's a write-up of the situation, shared on Facebook publicly, from a friend involved in careers at a UK Sixth Form college (where students take their A-Levels): ################### I've spent the last two days dealing with very upset young people who have had their plans ripped up. Some will still be able to go to university, but not the one they planned on and have been forced into clearing, some won't be able to go at all. So the story behind this is as follows. Teachers at colleges and schools submitted grades to exam boards earlier in the summer based on their predictions as to how students would do in exams they would not take due to C19. The exam boards then mauled these with an algorithm. Our current understanding is that subject cohorts of less than 10 have not had the algorithm applied and 15-20 only slightly. Most private and many small Sixth forms in schools fall in to this category, (note this is students in a subject at a provider not total numbers at that provider) Some centre assessed grades (CAGs) will have been assigned optomistically (inflated) by teachers in an attempt to advantage them and their students but the vast majority will have been genuine assessments based on submitted work and teacher experience. They had no idea just how much adjustment would be applied, I'm sure if they had them they might not have been quite so honest! [Edit: a colleague has pointed out that sometimes predicted grades are higher than achieved because there are unanticipated issues very close to exams such as illness, bereavement a stress which are not evident in year. This cannot be predicted by teachers and compensated for in advance, although many schools will have tried to adjust for students who experienced issues in year] To compensate for this small amount of ingrained optimisation and unpredictability the exam boards have moderated down larger schools CAGs (but not the smaller cohorts). This includes many sixth form colleges with thousands of exam entries. Sixth form colleges and larger sixth forms are very good at estimating grades and have been extremely thorough in their efforts to assign realistic grades. Our own teachers spent weeks compiling grades from previous work and tests to get the most accurate assessment they could. No one wanted their students to miss out, no one underestimated their CAGs. The exam boards were not going to stand for a big increase in grades this year so they applied an aggressive algorithm to the larger subject cohorts. Our college for example has hundreds of A level maths students which makes us a target for harsh moderation despite having increadable value added and a history of very accurate (not over inflated) predictions; vast numbers of our students have seen their grades dropped by one or two grade boundaries, I have only seen one moderated up. Many of our students have missed out on university places. Despite positive news stories about one Oxbridge University College most universities have not moderated their offers in light of this crisis and many students at private schools will have been able to swoop in on those places and adjust upwards to higher tarrif courses. Now, whether you think this is a government fix up or not; it is not fair on poorer students at bigger schools and coincidentally some more 'privileged' students have been given another leg up. Talk to your MP talk about it on social media and lobby to get this fixed because it's going to mess up a lot of young people's futures. These young people are now voters and I hope they will be registering their concern with their local MPs which may motivate them to put pressure on the minister and in turn the exam boards If you think this was a fiasco then hold on tight because next weeks GCSE results will be based on less information and are likely to be even more skewed! We will hit a second crisis of students not being able to access A levels and colleges having to juggle the same situation as universities are this week. Many GCSE students will be awarded grades which either over inflate or under play their ability. Many will lose an A level place and others will end up on courses which they are 'qualified' for but are overwhelmingly too hard for them, causing distress and self esteem issues. This is on top of the emerging increase in youth mental health issues due to C19 and the lockdown. If you feel the government's last minute addition of the 'tripple lock' and free appeals makes everything OK then I have disappointing news for you. Mock exams are internally administered in a variety of different ways. Allowing the use of mocks rather than CAGs will lead to even more disparity. September exams for students, out of education and in lock down for 6 months with no access to resources and who thought they had finished their A levels? I'll leave you to draw your own conclusions about whether that will be helpful. We in education will continue, we will pick up the pieces, we will do everything we can to make sure these young people have the best outcomes they can. ###################