7 ms·

If this was a paper submitted by one of my students, they would fail. Why would these faculty members sign their names to this drivel? If you read carefully, yo

If this was a paper submitted by one of my students, they would fail. Why would these faculty members sign their names to this drivel? If you read carefully, you see how useless SAT and ACT scores really are. Here are just a few insane things. > Figure 1 shows a linear relationship between SAT scores and cumulative first-year GPA for Dartmouth students. SAT scores are grouped into 16 equal-sized bins, and we plot mean first-year GPA (y-axis) against mean SAT score (x-axis) in the bin.4 Within any smaller range of scores, first-year GPA and the SAT maintain the same linear relationship. Total nonsense. Figure 1 does not mean that SAT/ACT scores are predictive of academic achievement! Without seeing the variance you cannot make this statement. At all. The fact that there is a linear relationship means nothing statistically. > SAT by itself explains about 22% of the variation in first-year GPA So we're explaining less than 1/5th of the variance. This makes SAT scores basically worthless, this account for almost none of the real world performance of people, never mind how redundant they are with GPAs. The correct title for this would be: SAT and ACT are not highly predictive of academic achievement at Dartmouth. There's a massive disconnect between the title and the data. The data shows there is likely some statistical effect on performance by SAT. The title would have you think that if you know a student's SAT scores, you would be good at predicting their outcomes at Dartmouth. The data shows this is wrong: if you only used SAT scores, you would almost always be wrong in predicting how well a student would do. A lot of papers play this game, they imply that statistically significant means meaningful. Nothing could be further from the truth. Something can be statistically significant but inconsequential. Never mind that they define as success at Dartmouth as your first year GPA.

toomim 3y ago
Um, 22% > 1/5.
anon291 3y ago
I'm confused. The authors cite several instances of research done at other institutions as the main evidence that SAT/ACT scores are predictive of overall success. At that point, the a priori assumption is that Dartmouth would behave the same way, which they then go on to show. I agree that they could have fully analyzed graduation rates and full-term GPA, but other studies have already done that. I do believe their analysis is thorough enough (looking at first year GPA) to adequately show to me that there is absolutely no reason to believe that Dartmouth would be any different than the other institutions at which much larger / more rigorous studies have been done. I mean... do you have a full double-blind study done when you're sick to determine the right dosage for you, or does your doctor consult an existing dosing study, make sure nothing is substantially different between you and the study group, and then decide upon an appropriate dose. There's a certain point at which over-analyzing makes little sense. The authors here ARE NOT engaged in research. This is NOT peer-reviewed. It's a letter elucidating an opinion by researchers in the field based on their knowledge of previous literature in the field and then a surface-level analysis to make sure that those results ought to apply to Dartmouth as well. Moreover, you're hyperbolizing yourself. Namely, you criticize the title, which is not the author's title at all. Their letter is titled "Report From Working Group on the Role of Standardized Test Scores in Undergraduate Admissions" The authors main claim (on the first page after niceties) is that ".Our overall conclusion is that the use of SAT and ACT scores is an essential method by which Admissions can identify applicants who will succeed at Dartmouth." This is not "SAT / ACT scores are highly predictive...". It's an entirely different statement. Let's not confuse HN editorializing with the contents of the letter. > The title would have you think that if you know a student's SAT scores, you would be good at predicting their outcomes at Dartmouth. The data shows this is wrong: if you only used SAT scores, you would almost always be wrong in predicting how well a student would do. Except if you read the text, that's not at all what the authors claim. The authors do not claim the title (again... that's HN's title, not the paper's). The authors say: > In short, consideration of SAT scores allows Admissions to identify applicants who will thrive academically at Dartmouth better than does the use of high school GPA alone. which is true. Even if you don't think it's a useful predictor, it is a more useful predictor than GPA. This is a reasonable contrast because in the absence of test scores, this would be the only data available. As for the other critiques. I agree that Figure 1 is not useful and shouldn't have been included. However, the table at the end is pretty straightforwards and provides essentially the same conclusion as above, which is that SAT is better than GPA. Moreover, just because the figure they included is bad does not mean the data they studied was. We'd need to know more, not wholesale reject their conclusion. The authors then go on to argue that other papers agree with these conclusions (they do), and that alternates to both SAT and GPA (letters of rec, etc) have been shown to exacerbate inequality while having very little power to predict how well a student will do. > The fact that there is a linear relationship means nothing statistically. It does... in fact... mean that there is a linear relationship in the observed data with the given levels of significance. I'm not sure what exactly you're going at here. I agree a scatter plot of all the data might be more useful, but honestly, given the plethora of other work on the subject published by so many disparate institutions of higher ed, I really have no reason to believe dartmouth would be much different. So while the included figure is not great, I'm guessing that if we saw all the data the conclusion would probably still be the same.
- light_hue_1 3y ago
  When I talk about statistics on HN it feels like I'm taking crazy pills. The HN community pretends to be intellectual, but the level of mathematical education here is terrible. > Moreover, you're hyperbolizing yourself. Namely, you criticize the title, which is not the author's title at all. Their letter is titled "Report From Working Group on the Role of Standardized Test Scores in Undergraduate Admissions" The authors clearly say "SAT and ACT scores are highly predictive of academic performance at Dartmouth". That is what they want to show. And this is totally false. They cannot accurately predict high vs low academic performance based on SAT or ACT scores. Their data clearly shows this is impossible. > Except if you read the text, that's not at all what the authors claim. The authors do not claim the title (again... that's HN's title, not the paper's). The authors say: Huh?! That's literally their claim. They even say in the conclusion "They are significantly predictive of academic success at Dartmouth" This is just totally wrong. They are a statistically significant but practically insignificant predictor of academic success when you define it in the most absurd and irrelevant way (first year GPA). > It does... in fact... mean that there is a linear relationship in the observed data with the given levels of significance. Summarizing data with some line, without any statistics, and saying "look the line slopes up", does not mean there is a linear relationship. At all. You cannot conclude one from the other. The fact that this isn't obvious is truly frightening.
  - anon291 3y ago
    You're right those sentences are there. My bad, I should have kept a closer eye. I looked at the author's title, which is indeed not as sensational as the HN title though. > They are a statistically significant but practically insignificant predictor of academic success when you define it in the most absurd and irrelevant way (first year GPA). As I said in my post, the bulk of their argument references other studies at similar institutions > Summarizing data with some line, without any statistics, and saying "look the line slopes up", does not mean there is a linear relationship. At all. You cannot conclude one from the other. They run their own linear regression. In no way is the argument "look the lines look linear".
  - hirvi74 3y ago
1980phipsi 3y ago
> Total nonsense. Figure 1 does not mean that SAT/ACT scores are predictive of academic achievement! Without seeing the variance you cannot make this statement. At all. The fact that there is a linear relationship means nothing statistically. You are putting words in their mouth from that quote. They didn't use that Figure to claim the test scores are predictive of academic achievement. They specifically say it is evidence of a linear relationship. They argue in the next paragraph that they are and use regression analysis to make their case. > So we're explaining less than 1/5th of the variance. This makes SAT scores basically worthless, this account for almost none of the real world performance of people, never mind how redundant they are with GPAs. Compare that to high school GPA, which explains ~9%. The test optional movement implicitly puts more weight on high school grades by putting less weight on test scores. I'm not saying their analysis is perfect, but your arguments aren't that strong... At the end of the day, the analysis focuses on a subset of people who are going to Dartmouth. This is a highly filtered subsample of the total population. If you took a random sampling of 18 year olds, gave them the SAT, and then enrolled them at Dartmouth, then I hypothesize you would see a significantly stronger relationship between test scores and college grades than what is observed in their data.
- light_hue_1 3y ago
  The knowledge of statistics on HN is astoundingly bad. Figure 1 shows nothing. At all. Without knowing if the results are statistically meaningful, showing a linear relationship means nothing. You can draw no conclusions from that figure. The only thing it does is lie with data, to imply to an observer that doesn't understand statistics that this means there is a meaningful relationship between the two variables. > I'm not saying their analysis is perfect, but your arguments aren't that strong... They said "SAT and ACT scores are highly predictive of academic performance". This means that they can predict academic performance based on SAT. They absolutely cannot. Their own data shows this beyond any doubt.
deleted 3y ago
[deleted]
deleted 3y ago
[deleted]
gnicholas 3y ago
I'm not super great at statistics, despite having taken several classes in college (which is now decades in the rear-view mirror). I am interested in your critique here but can't quite follow what you're suggesting. Can you make it in plainer terms? I understand that HN has blindspots, but I'm a bit skeptical that HN has a blindspot and a gaggle of Dartmouth professors who were selected to do data analysis have the same one.
deleted 3y ago
[deleted]
deleted 3y ago
[deleted]
deleted 3y ago
[deleted]
deleted 3y ago
[deleted]
deleted 3y ago
[deleted]
deleted 3y ago
[deleted]
deleted 3y ago
[deleted]
deleted 3y ago
[deleted]

toomim 3y ago

Um, 22% > 1/5.

anon291 3y ago

I'm confused. The authors cite several instances of research done at other institutions as the main evidence that SAT/ACT scores are predictive of overall success. At that point, the a priori assumption is that Dartmouth would behave the same way, which they then go on to show. I agree that they could have fully analyzed graduation rates and full-term GPA, but other studies have already done that. I do believe their analysis is thorough enough (looking at first year GPA) to adequately show to me that there is absolutely no reason to believe that Dartmouth would be any different than the other institutions at which much larger / more rigorous studies have been done. I mean... do you have a full double-blind study done when you're sick to determine the right dosage for you, or does your doctor consult an existing dosing study, make sure nothing is substantially different between you and the study group, and then decide upon an appropriate dose. There's a certain point at which over-analyzing makes little sense. The authors here ARE NOT engaged in research. This is NOT peer-reviewed. It's a letter elucidating an opinion by researchers in the field based on their knowledge of previous literature in the field and then a surface-level analysis to make sure that those results ought to apply to Dartmouth as well. Moreover, you're hyperbolizing yourself. Namely, you criticize the title, which is not the author's title at all. Their letter is titled "Report From Working Group on the Role of Standardized Test Scores in Undergraduate Admissions" The authors main claim (on the first page after niceties) is that ".Our overall conclusion is that the use of SAT and ACT scores is an essential method by which Admissions can identify applicants who will succeed at Dartmouth." This is not "SAT / ACT scores are highly predictive...". It's an entirely different statement. Let's not confuse HN editorializing with the contents of the letter. > The title would have you think that if you know a student's SAT scores, you would be good at predicting their outcomes at Dartmouth. The data shows this is wrong: if you only used SAT scores, you would almost always be wrong in predicting how well a student would do. Except if you read the text, that's not at all what the authors claim. The authors do not claim the title (again... that's HN's title, not the paper's). The authors say: > In short, consideration of SAT scores allows Admissions to identify applicants who will thrive academically at Dartmouth better than does the use of high school GPA alone. which is true. Even if you don't think it's a useful predictor, it is a more useful predictor than GPA. This is a reasonable contrast because in the absence of test scores, this would be the only data available. As for the other critiques. I agree that Figure 1 is not useful and shouldn't have been included. However, the table at the end is pretty straightforwards and provides essentially the same conclusion as above, which is that SAT is better than GPA. Moreover, just because the figure they included is bad does not mean the data they studied was. We'd need to know more, not wholesale reject their conclusion. The authors then go on to argue that other papers agree with these conclusions (they do), and that alternates to both SAT and GPA (letters of rec, etc) have been shown to exacerbate inequality while having very little power to predict how well a student will do. > The fact that there is a linear relationship means nothing statistically. It does... in fact... mean that there is a linear relationship in the observed data with the given levels of significance. I'm not sure what exactly you're going at here. I agree a scatter plot of all the data might be more useful, but honestly, given the plethora of other work on the subject published by so many disparate institutions of higher ed, I really have no reason to believe dartmouth would be much different. So while the included figure is not great, I'm guessing that if we saw all the data the conclusion would probably still be the same.

light_hue_1 3y ago
When I talk about statistics on HN it feels like I'm taking crazy pills. The HN community pretends to be intellectual, but the level of mathematical education here is terrible. > Moreover, you're hyperbolizing yourself. Namely, you criticize the title, which is not the author's title at all. Their letter is titled "Report From Working Group on the Role of Standardized Test Scores in Undergraduate Admissions" The authors clearly say "SAT and ACT scores are highly predictive of academic performance at Dartmouth". That is what they want to show. And this is totally false. They cannot accurately predict high vs low academic performance based on SAT or ACT scores. Their data clearly shows this is impossible. > Except if you read the text, that's not at all what the authors claim. The authors do not claim the title (again... that's HN's title, not the paper's). The authors say: Huh?! That's literally their claim. They even say in the conclusion "They are significantly predictive of academic success at Dartmouth" This is just totally wrong. They are a statistically significant but practically insignificant predictor of academic success when you define it in the most absurd and irrelevant way (first year GPA). > It does... in fact... mean that there is a linear relationship in the observed data with the given levels of significance. Summarizing data with some line, without any statistics, and saying "look the line slopes up", does not mean there is a linear relationship. At all. You cannot conclude one from the other. The fact that this isn't obvious is truly frightening.
- anon291 3y ago
  You're right those sentences are there. My bad, I should have kept a closer eye. I looked at the author's title, which is indeed not as sensational as the HN title though. > They are a statistically significant but practically insignificant predictor of academic success when you define it in the most absurd and irrelevant way (first year GPA). As I said in my post, the bulk of their argument references other studies at similar institutions > Summarizing data with some line, without any statistics, and saying "look the line slopes up", does not mean there is a linear relationship. At all. You cannot conclude one from the other. They run their own linear regression. In no way is the argument "look the lines look linear".
- hirvi74 3y ago

1980phipsi 3y ago

> Total nonsense. Figure 1 does not mean that SAT/ACT scores are predictive of academic achievement! Without seeing the variance you cannot make this statement. At all. The fact that there is a linear relationship means nothing statistically. You are putting words in their mouth from that quote. They didn't use that Figure to claim the test scores are predictive of academic achievement. They specifically say it is evidence of a linear relationship. They argue in the next paragraph that they are and use regression analysis to make their case. > So we're explaining less than 1/5th of the variance. This makes SAT scores basically worthless, this account for almost none of the real world performance of people, never mind how redundant they are with GPAs. Compare that to high school GPA, which explains ~9%. The test optional movement implicitly puts more weight on high school grades by putting less weight on test scores. I'm not saying their analysis is perfect, but your arguments aren't that strong... At the end of the day, the analysis focuses on a subset of people who are going to Dartmouth. This is a highly filtered subsample of the total population. If you took a random sampling of 18 year olds, gave them the SAT, and then enrolled them at Dartmouth, then I hypothesize you would see a significantly stronger relationship between test scores and college grades than what is observed in their data.

light_hue_1 3y ago
The knowledge of statistics on HN is astoundingly bad. Figure 1 shows nothing. At all. Without knowing if the results are statistically meaningful, showing a linear relationship means nothing. You can draw no conclusions from that figure. The only thing it does is lie with data, to imply to an observer that doesn't understand statistics that this means there is a meaningful relationship between the two variables. > I'm not saying their analysis is perfect, but your arguments aren't that strong... They said "SAT and ACT scores are highly predictive of academic performance". This means that they can predict academic performance based on SAT. They absolutely cannot. Their own data shows this beyond any doubt.

deleted 3y ago

[deleted]

deleted 3y ago

[deleted]

gnicholas 3y ago

I'm not super great at statistics, despite having taken several classes in college (which is now decades in the rear-view mirror). I am interested in your critique here but can't quite follow what you're suggesting. Can you make it in plainer terms? I understand that HN has blindspots, but I'm a bit skeptical that HN has a blindspot and a gaggle of Dartmouth professors who were selected to do data analysis have the same one.

deleted 3y ago

[deleted]

deleted 3y ago

[deleted]

deleted 3y ago

[deleted]

deleted 3y ago

[deleted]

deleted 3y ago

[deleted]

deleted 3y ago

[deleted]

deleted 3y ago

[deleted]

deleted 3y ago

[deleted]