5 ms·

If the SAT were a joke, you wouldn't expect scores to follow a bell curve with only ~0.3% of people getting a perfect score, but they do.

disgruntledphd2 7y ago
This isn't a good argument, because the SAT (and other tests of the same kind) are normalised to have this property. The SAT may not be good tests, but that fact has nothing to do with the presence or absence of a normal distribution.
- ebrenes 7y ago
  The SATs have a top score, if it were easy to game you'd find many people would clump at the top. That would make it impossible to normalize that distribution into a bell-shape because they're all clustered into the same bucket with no way of spreading them. OP's point was that the perfect scores remain consistently outside of people's grasp despite the variety of resources available to prep for the exam. I only once managed to hit perfect score and my other best scores were one or two questions off. I had been taking the test since I was 11 (for various extracurricular camps/activities) and prepped multiple times for them. The biggest scores jumps were more closely related with my age and academic achievements than anything else.
  - chongli 7y ago
    A lot of the reason the SATs are able to maintain a spread that lets them normalize the distribution is that they fill the tests with stupid tricks that fool people into wasting time. If you do a ton of prep then you learn to spot these tricks and then a lot of the questions become really easy and you can knock them off quickly. A poor smart student may have mastery of the material but their score will suffer if they don't know the tricks.
    - barry-cotter 7y ago
      And you can find out about all these tricks for free on Khan Academy or for $20 in a SAT prep book.
      - chongli 7y ago
        There are plenty of kids in the US who can't afford to bring lunch to school, let alone pay $20 for a SAT prep book. Many of these kids don't have a computer or Internet access at home for that matter, so Khan Academy isn't much good to them either. When we talk about rich vs poor kids, there's a HUGE gap that many people don't realize.
        pishpash 7y ago
        They have public libraries with both computers and books. Access has never been the issue if there is the will.
        convivialdingo 7y ago
        I don’t think that gap is huge in terms of internet access - there’s certainly a gap but it’s mostly a function of parents limiting access in my experience. 84% of teens have smartphones in the US. A large percentage of that remaining 16% have access to internet at school, as 98% of schools have broadband internet.
        banachtarski 7y ago
        Anecdote: When I was younger, I used to take a public bus to a local bookstore and work through the SAT practice books (on separate scrap paper of course).
        chongli 7y ago
        It sounds like you didn't have to pick up younger siblings at school when you were younger. That's the reality for a lot of poor kids. Raised by a single parent, the older siblings are in charge of looking after the younger ones. The parent often does not get home from work until late (and may have multiple jobs).
  - disgruntledphd2 7y ago
    That is correct. So the SATs are (almost certainly) 3-pl (actually 2) IRT models. Essentially, it's a multivariate generalised linear mixed model to estimate both question difficulty and participant ability. Normally, they'll estimate the abilities on the logistic scale, and use the percentile to back transform to a standard normal. Most people don't cluster at the top because they are a good proxy for g, which is an imaginary statistical construct that we use to explain differences in school outcomes. So I had a long digression here about the usefulness of penalties for guessing, but it turns out the SATs don't do that anymore, so wth? (ETS invented IRT, that's why I'm pretty sure).
  - austincheney 7y ago
    > The SATs have a top score, if it were easy to game you'd find many people would clump at the top. I understand and appreciate my school was statistically abnormal but that is exactly what happened. I graduated in a class 334 students and several achieved perfect scores. Perhaps more than a third of the students achieved greater than 1300 and greater than 20% achieved greater than 1400. This is when 1600 was the perfect score. All these scores indicate is the degree of conditioning imposed upon a student. I know people want these tests to mean something more for personal reasons, but according to all available data this is biased wishful thinking. The research on standardized convergent testing indicates it is not a measure of academic success or potential, but rather an indicator/discriminator of class distinctions due to availability of preparation. Because I did not come from well groomed pedigree, did not value the subculture of excess vanity, and came from a family that was lower positioned financially I deliberately inverted the goals for a personal social experiment. I wanted to see how close to the bottom I could get and still graduate on time. This was exciting because the risks were greater. If you fail to estimate the conditions correctly you don’t graduate whereas other people get a slightly lower test score or grade point average. Because the goals were different you had plan and weigh the conditions in unexpected ways. Even with all the effort I put in there were still 5 people who graduated with a lower class rank than me. What impact did that have in later life? I did not get a free ride to an Ivy League school like many of my classmates, enter corporate management immediately out of college, or become a corporate executive within 10 years. I did enter and graduate college. I became a self taught software developer without much challenge and have found very low resistance attaining employment as a senior developer in my full time job. I also became a managing principal in my part time job without as much effort. The greatest tragedy in all this isn’t lost status or earning potential but how boring life has turned out.
salty_biscuits 7y ago
If you had a multiple choice quiz with the actual questions hidden you would get a normal distribution in results. Anything where you sum lots of small random effects.
- im3w1l 7y ago
  If the test was not measuring anything at all, then you would expect repeat takers to have completely unrelated scores on first and last test.
  - salty_biscuits 7y ago
    That's true, I'm just pointing out that the normality of the distribution of the results alone proves nothing.

disgruntledphd2 7y ago

This isn't a good argument, because the SAT (and other tests of the same kind) are normalised to have this property. The SAT may not be good tests, but that fact has nothing to do with the presence or absence of a normal distribution.

ebrenes 7y ago
The SATs have a top score, if it were easy to game you'd find many people would clump at the top. That would make it impossible to normalize that distribution into a bell-shape because they're all clustered into the same bucket with no way of spreading them. OP's point was that the perfect scores remain consistently outside of people's grasp despite the variety of resources available to prep for the exam. I only once managed to hit perfect score and my other best scores were one or two questions off. I had been taking the test since I was 11 (for various extracurricular camps/activities) and prepped multiple times for them. The biggest scores jumps were more closely related with my age and academic achievements than anything else.
- chongli 7y ago
  A lot of the reason the SATs are able to maintain a spread that lets them normalize the distribution is that they fill the tests with stupid tricks that fool people into wasting time. If you do a ton of prep then you learn to spot these tricks and then a lot of the questions become really easy and you can knock them off quickly. A poor smart student may have mastery of the material but their score will suffer if they don't know the tricks.
  - barry-cotter 7y ago
    And you can find out about all these tricks for free on Khan Academy or for $20 in a SAT prep book.
    - chongli 7y ago
      There are plenty of kids in the US who can't afford to bring lunch to school, let alone pay $20 for a SAT prep book. Many of these kids don't have a computer or Internet access at home for that matter, so Khan Academy isn't much good to them either. When we talk about rich vs poor kids, there's a HUGE gap that many people don't realize.
      - pishpash 7y ago
        They have public libraries with both computers and books. Access has never been the issue if there is the will.
      - convivialdingo 7y ago
        I don’t think that gap is huge in terms of internet access - there’s certainly a gap but it’s mostly a function of parents limiting access in my experience. 84% of teens have smartphones in the US. A large percentage of that remaining 16% have access to internet at school, as 98% of schools have broadband internet.
      - banachtarski 7y ago
        Anecdote: When I was younger, I used to take a public bus to a local bookstore and work through the SAT practice books (on separate scrap paper of course).
        chongli 7y ago
        It sounds like you didn't have to pick up younger siblings at school when you were younger. That's the reality for a lot of poor kids. Raised by a single parent, the older siblings are in charge of looking after the younger ones. The parent often does not get home from work until late (and may have multiple jobs).
- disgruntledphd2 7y ago
  That is correct. So the SATs are (almost certainly) 3-pl (actually 2) IRT models. Essentially, it's a multivariate generalised linear mixed model to estimate both question difficulty and participant ability. Normally, they'll estimate the abilities on the logistic scale, and use the percentile to back transform to a standard normal. Most people don't cluster at the top because they are a good proxy for g, which is an imaginary statistical construct that we use to explain differences in school outcomes. So I had a long digression here about the usefulness of penalties for guessing, but it turns out the SATs don't do that anymore, so wth? (ETS invented IRT, that's why I'm pretty sure).
- austincheney 7y ago
  > The SATs have a top score, if it were easy to game you'd find many people would clump at the top. I understand and appreciate my school was statistically abnormal but that is exactly what happened. I graduated in a class 334 students and several achieved perfect scores. Perhaps more than a third of the students achieved greater than 1300 and greater than 20% achieved greater than 1400. This is when 1600 was the perfect score. All these scores indicate is the degree of conditioning imposed upon a student. I know people want these tests to mean something more for personal reasons, but according to all available data this is biased wishful thinking. The research on standardized convergent testing indicates it is not a measure of academic success or potential, but rather an indicator/discriminator of class distinctions due to availability of preparation. Because I did not come from well groomed pedigree, did not value the subculture of excess vanity, and came from a family that was lower positioned financially I deliberately inverted the goals for a personal social experiment. I wanted to see how close to the bottom I could get and still graduate on time. This was exciting because the risks were greater. If you fail to estimate the conditions correctly you don’t graduate whereas other people get a slightly lower test score or grade point average. Because the goals were different you had plan and weigh the conditions in unexpected ways. Even with all the effort I put in there were still 5 people who graduated with a lower class rank than me. What impact did that have in later life? I did not get a free ride to an Ivy League school like many of my classmates, enter corporate management immediately out of college, or become a corporate executive within 10 years. I did enter and graduate college. I became a self taught software developer without much challenge and have found very low resistance attaining employment as a senior developer in my full time job. I also became a managing principal in my part time job without as much effort. The greatest tragedy in all this isn’t lost status or earning potential but how boring life has turned out.

salty_biscuits 7y ago

If you had a multiple choice quiz with the actual questions hidden you would get a normal distribution in results. Anything where you sum lots of small random effects.

im3w1l 7y ago
If the test was not measuring anything at all, then you would expect repeat takers to have completely unrelated scores on first and last test.
- salty_biscuits 7y ago
  That's true, I'm just pointing out that the normality of the distribution of the results alone proves nothing.