3 ms·

A recurring theme here seems to be the bottom 10%, or bottom 20%, or what have you, scores in the US are dragging down the other scores. If those were ignored,

A recurring theme here seems to be the bottom 10%, or bottom 20%, or what have you, scores in the US are dragging down the other scores. If those were ignored, the US would presumably be way up top. But that is not the case. If you compare the top 10% of PISA scores in Japan, Singapore, South Korea etc. - or Finland, Switzerland, Holland - to the top 10% in the US, the US top 10% is below their top 10% as well. http://educationbythenumbers.org/content/top-us-students-fare-poorly-international-pisa-test-scores-shanghai-tops-world-finland-slips_693/ http://educationbythenumbers.org/content/top-us-students-far...

acdha 12y ago
That kind of study is really hard for non-experts to draw conclusions from because the process is full of confounds. Even before you look at the test content you'd want to compare across demographic profiles (i.e. parental education levels or income) to see whether that trend holds up. In the case of the PISA scores, there's 2013 paper claiming that some percentage of the differences are explained by a sampling bias over-representing low-income students: http://www.epi.org/publication/us-student-performance-testing/ http://www.epi.org/publication/us-student-performance-testin... That could have an effect on the top-end scores as well if it reduced the percentage of top students in the pool. The other big problem you get is that this is a very narrow test of a single subject and Goodhart's law reigns supreme in this kind of comparison. The total difference being discussed is a 2% delta between the top US decile and Shanghai, which was the world leader. Even if that's not just random fluctuation, it could be explained entirely by something as simple as Singapore making that test a priority whereas it's largely off the radar in the US. That's not an accusation of cheating – it could be as simple as whether kids have drilled as heavily on a specific skill which is used more in one national curriculum than the other. EDIT: To clarify: this shouldn't be read as criticism of PISA – I'm sure they've made considerable effort to address this challenges – but simply that this kind of comparison is a HARD challenge.

acdha 12y ago

That kind of study is really hard for non-experts to draw conclusions from because the process is full of confounds. Even before you look at the test content you'd want to compare across demographic profiles (i.e. parental education levels or income) to see whether that trend holds up. In the case of the PISA scores, there's 2013 paper claiming that some percentage of the differences are explained by a sampling bias over-representing low-income students: http://www.epi.org/publication/us-student-performance-testing/ http://www.epi.org/publication/us-student-performance-testin... That could have an effect on the top-end scores as well if it reduced the percentage of top students in the pool. The other big problem you get is that this is a very narrow test of a single subject and Goodhart's law reigns supreme in this kind of comparison. The total difference being discussed is a 2% delta between the top US decile and Shanghai, which was the world leader. Even if that's not just random fluctuation, it could be explained entirely by something as simple as Singapore making that test a priority whereas it's largely off the radar in the US. That's not an accusation of cheating – it could be as simple as whether kids have drilled as heavily on a specific skill which is used more in one national curriculum than the other. EDIT: To clarify: this shouldn't be read as criticism of PISA – I'm sure they've made considerable effort to address this challenges – but simply that this kind of comparison is a HARD challenge.