5 ms·

Standardized tests are a lousy way of deciding things. But at least they're better than non-standardized tests. At my university (SFU, in Canada) admission to

Standardized tests are a lousy way of deciding things. But at least they're better than non-standardized tests. At my university (SFU, in Canada) admission to most faculties is based on high school grades. That's grades awarded by high schools -- we used to have province-wide grade 12 exams, but for almost all subjects those were abolished a few years ago. It's widely recognized at the post-secondary level that a 90% grade from some schools means much more than a 90% grade from some other schools; but if anyone is actually adjusting high school grades to account for these differences, they're not admitting it. Entrance scholarships, and admission in one faculty (Business), is determined partly based on high school grades and partly based on extracurricular activities -- in most cases, volunteer activities. Great, we're bringing in well-rounded students who are dedicated to helping the community, right? No; it turns out that they admit that they're only volunteering in order to pad their resumes -- in some cases because high-priced consultants hired by their parents told them exactly what volunteering they should do and what to say about it when they're writing their scholarship applications. (I've been lobbying for years to get an analysis done of how the rating of student extracurricular activities correlates with the median income of their postal code. I'm still waiting.) The SAT become popular because it could identify smart kids who would otherwise have been overlooked due to their race, gender, religion, or class. Is it perfect? Hell no. But as we discard standardized tests, I worry that we're moving back to a world where university admission will depend less on aptitude and more on having parents who have the money and the connections to get you the "right" extracurricular experience.

kyleschiller 9y ago
Yeah, I don't think Sternberg is advocating for ending standardized testing: "My colleagues and I developed assessments for creativity, common sense and wisdom... we doubled prediction [accuracy] for academic performance, and with Kaleidoscope we could predict the quality of extracurricular performance, which the SAT doesn’t do." It sounds like he just wants to reform them, and has some strong initial results to suggest that we could do much better.
- nindalf 9y ago
  Any metric can be gamed. It only works until people figure out how.
  - yorwba 9y ago
    The trick is to change your metrics often enough, so people can never be sure they gamed the right one.
    - markdown 9y ago
      "You mean I spent a hundred hours volunteering at the soup kitchen for nothing‽"
      - arcticfox 9y ago
        Perfect!
    - andai 9y ago
      See also: once a metric becomes known, it ceases to be reliable.
  - danieltillett 9y ago
    No one has figured out how to game well designed IQ tests. That is the appeal of high g-loaded tests like the SAT.
cbhl 9y ago
> but if anyone is actually adjusting high school grades to account for these differences, they're not admitting it. I am pretty sure this was the standard line for campus tours at the University of Waterloo -- that you should take high school exams at your school and not at some diploma mill, because Waterloo did adjust your entrance average based on what high school you came from, and how those students did once they got to university. (That said, I also heard that there are so many "perfect" candidates that they're contemplating rejecting students at random in order to hit class size numbers, because scaling up requires more lecture halls and more dorms, and those things take years to build out.)
- cperciva 9y ago
  Sorry, I should have elaborated. My understanding is that Toronto, Waterloo, and a few other top universities in Ontario have had a "black book" of this nature for a long time. I'm not aware of any universities in BC admitting to doing the same thing.
- vonmoltke 9y ago
  > that you should take high school exams at your school and not at some diploma mill I have never heard this term applied to a high school before. Is this a thing in Canada?
  - majormajor 9y ago
    It's a thing in the US too, especially in high-end athletic circles with traveling teams, etc (the flip side of the argument is that elite athletic prospects can't fit traditional high school schedules into their routines easily).
cheetos 9y ago
I knew a classmate who openly admitted to putting made up volunteer activities on her college applications, arguing there was no chance the school would fact check her. She was right!
- pzh 9y ago
  The problem with these is that they couldn't have been significant enough to make a dent in the outcome. Pretty much all significant achievements are fact-checkable. E.g. you may falsely claim that you were doing karate or basketball, but if you did win any trophies, then most selective schools wouldn't care about you.
  - cperciva 9y ago
    You might be surprised. I was on the committee which adjudicated major entrance scholarships for six years, and I can count on my fingers the number of times someone said "can we get some verification of this?" We've never thought that fraud was a common problem, and when you're competing to recruit the top students you really don't want to go back to them and accuse them of dishonesty.. And (although we're revising the system and hopefully changing this part) the relative weighting of "significant achievements" of an easily fact-checked nature vs. "doing lots of stuff" has skewed heavily towards the latter; you could have a resume full of "volunteered at X" and "leader of student club Y" which would be worth as much as the difference between a 90% high school average and a 95% high school average, while containing nothing which would show up online or elicit commentary from a referee beyond "this student is heavily involved". (We're putting together a new system starting next year which is aimed to be more about identifying students who are in some way exceptional as opposed to merely being generally all-round good people. A large part of the motivation for this is to avoid easily "gamed" metrics; nobody spends 15 years playing violin or becomes a national chess master simply because it will look good on their resume. So with luck we'll end up having less of a fact-checking problem in the future; but we're still going to have a significant amount of trust built into the system.)
    - Bakary 9y ago
chrisseaton 9y ago
> At my university (SFU, in Canada) admission to most faculties is based on high school grades Do you not interview the applicants though? I think the best thing to do is to set a basic requirement for grades from their highschool, then get people in and ask them to take your own tests and give them an interview with one of the teaching staff to make your own judgement about whether they're suitable or not. Accepting someone to be your student for four years without ever having met them is as crazy as any other problems.
- cperciva 9y ago
  We're a public university admitting over 7,000 students per year. There simply aren't the resources to interview students... not to mention the fact that SFU competes with other institutions and would likely lose a large number of students if we required them to come in for interviews.
  - kwhitefoot 9y ago
    if you don't have the resources to interview them how can you have the resources to teach them?
- barrkel 9y ago
  As we all know, people never judge based on looks or accent or verbal word choice. We select for talent even if it's from someone with a different culture and social mileau to us, right?
  - chrisseaton 9y ago
    Well there's no need to be so sarcastic and snarky. I did say some universities administer their own tests, so that's as objective as a test normally can be (not perfect), and probably more objective than one administered across different schools as people have already said. I don't think it would be unreasonable to combine that with an interview. Teaching is a personal thing - academics will want to check that they can teach you and you're able to learn from them. The best university interviews I did were like a short teaching session and I came away with an opinion about who I wanted to learn from as much as I presume they had an opinion about who they wanted to spend their time teaching.
analog31 9y ago
One potential benefit of using high school grades is that it spreads out the opportunity to attend college, over a wider geographic area than just the affluent cities. In my state (Wisconsin), admission to the flagship state college is distributed in some similar way, though I don't know the details. Now of course we can ask if it's the best way to distribute this resource, and I think it depends to some extent on what you think is the mission of the college. At least in Wisconsin, there has been a traditional idea that the college exists to serve the development of the entire state. (The Wisconsin Idea). Politically, it gives people in the far corners of the state a stake in supporting public higher ed. Of course that's just one take on the mission of one school.
- caseysoftware 9y ago
  Texas is similar. If you're in the top 10% of your graduating class, you have a spot in the University of Texas system. Though UT-Austin (the flagship) uses the top 8%. Ref: http://catalog.utexas.edu/archive/2015-16/general-information/admission/undergraduate-admission/freshman-admission/ http://catalog.utexas.edu/archive/2015-16/general-informatio...
walshemj 9y ago
Seriously there is no national standard for high school tests in Canada?
- cperciva 9y ago
  Correct. Not even provincial standards in many cases.
  - apricot 9y ago
    Quebec uses something called the R score to rank applicants to universities. The R score of a student is the sum of the student's Z score (how many standard deviations above the mean), plus the group's Z score relative to all students. https://en.wikipedia.org/wiki/R_score https://en.wikipedia.org/wiki/R_score
Bakary 9y ago
>I worry that we're moving back to a world where university admission will depend less on aptitude and more on having parents who have the money and the connections to get you the "right" extracurricular experience. We already moved back to this system and have done so for a long time, in my opinion. There's also a second round of this phenomenon with unpaid internships.

kyleschiller 9y ago

Yeah, I don't think Sternberg is advocating for ending standardized testing: "My colleagues and I developed assessments for creativity, common sense and wisdom... we doubled prediction [accuracy] for academic performance, and with Kaleidoscope we could predict the quality of extracurricular performance, which the SAT doesn’t do." It sounds like he just wants to reform them, and has some strong initial results to suggest that we could do much better.

nindalf 9y ago
Any metric can be gamed. It only works until people figure out how.
- yorwba 9y ago
  The trick is to change your metrics often enough, so people can never be sure they gamed the right one.
  - markdown 9y ago
    "You mean I spent a hundred hours volunteering at the soup kitchen for nothing‽"
    - arcticfox 9y ago
      Perfect!
  - andai 9y ago
    See also: once a metric becomes known, it ceases to be reliable.
- danieltillett 9y ago
  No one has figured out how to game well designed IQ tests. That is the appeal of high g-loaded tests like the SAT.

cbhl 9y ago

> but if anyone is actually adjusting high school grades to account for these differences, they're not admitting it. I am pretty sure this was the standard line for campus tours at the University of Waterloo -- that you should take high school exams at your school and not at some diploma mill, because Waterloo did adjust your entrance average based on what high school you came from, and how those students did once they got to university. (That said, I also heard that there are so many "perfect" candidates that they're contemplating rejecting students at random in order to hit class size numbers, because scaling up requires more lecture halls and more dorms, and those things take years to build out.)

cperciva 9y ago
Sorry, I should have elaborated. My understanding is that Toronto, Waterloo, and a few other top universities in Ontario have had a "black book" of this nature for a long time. I'm not aware of any universities in BC admitting to doing the same thing.
vonmoltke 9y ago
> that you should take high school exams at your school and not at some diploma mill I have never heard this term applied to a high school before. Is this a thing in Canada?
- majormajor 9y ago
  It's a thing in the US too, especially in high-end athletic circles with traveling teams, etc (the flip side of the argument is that elite athletic prospects can't fit traditional high school schedules into their routines easily).

cheetos 9y ago

I knew a classmate who openly admitted to putting made up volunteer activities on her college applications, arguing there was no chance the school would fact check her. She was right!

pzh 9y ago
The problem with these is that they couldn't have been significant enough to make a dent in the outcome. Pretty much all significant achievements are fact-checkable. E.g. you may falsely claim that you were doing karate or basketball, but if you did win any trophies, then most selective schools wouldn't care about you.
- cperciva 9y ago
  You might be surprised. I was on the committee which adjudicated major entrance scholarships for six years, and I can count on my fingers the number of times someone said "can we get some verification of this?" We've never thought that fraud was a common problem, and when you're competing to recruit the top students you really don't want to go back to them and accuse them of dishonesty.. And (although we're revising the system and hopefully changing this part) the relative weighting of "significant achievements" of an easily fact-checked nature vs. "doing lots of stuff" has skewed heavily towards the latter; you could have a resume full of "volunteered at X" and "leader of student club Y" which would be worth as much as the difference between a 90% high school average and a 95% high school average, while containing nothing which would show up online or elicit commentary from a referee beyond "this student is heavily involved". (We're putting together a new system starting next year which is aimed to be more about identifying students who are in some way exceptional as opposed to merely being generally all-round good people. A large part of the motivation for this is to avoid easily "gamed" metrics; nobody spends 15 years playing violin or becomes a national chess master simply because it will look good on their resume. So with luck we'll end up having less of a fact-checking problem in the future; but we're still going to have a significant amount of trust built into the system.)
  - Bakary 9y ago

chrisseaton 9y ago

> At my university (SFU, in Canada) admission to most faculties is based on high school grades Do you not interview the applicants though? I think the best thing to do is to set a basic requirement for grades from their highschool, then get people in and ask them to take your own tests and give them an interview with one of the teaching staff to make your own judgement about whether they're suitable or not. Accepting someone to be your student for four years without ever having met them is as crazy as any other problems.

cperciva 9y ago
We're a public university admitting over 7,000 students per year. There simply aren't the resources to interview students... not to mention the fact that SFU competes with other institutions and would likely lose a large number of students if we required them to come in for interviews.
- kwhitefoot 9y ago
  if you don't have the resources to interview them how can you have the resources to teach them?
barrkel 9y ago
As we all know, people never judge based on looks or accent or verbal word choice. We select for talent even if it's from someone with a different culture and social mileau to us, right?
- chrisseaton 9y ago
  Well there's no need to be so sarcastic and snarky. I did say some universities administer their own tests, so that's as objective as a test normally can be (not perfect), and probably more objective than one administered across different schools as people have already said. I don't think it would be unreasonable to combine that with an interview. Teaching is a personal thing - academics will want to check that they can teach you and you're able to learn from them. The best university interviews I did were like a short teaching session and I came away with an opinion about who I wanted to learn from as much as I presume they had an opinion about who they wanted to spend their time teaching.

analog31 9y ago

One potential benefit of using high school grades is that it spreads out the opportunity to attend college, over a wider geographic area than just the affluent cities. In my state (Wisconsin), admission to the flagship state college is distributed in some similar way, though I don't know the details. Now of course we can ask if it's the best way to distribute this resource, and I think it depends to some extent on what you think is the mission of the college. At least in Wisconsin, there has been a traditional idea that the college exists to serve the development of the entire state. (The Wisconsin Idea). Politically, it gives people in the far corners of the state a stake in supporting public higher ed. Of course that's just one take on the mission of one school.

caseysoftware 9y ago
Texas is similar. If you're in the top 10% of your graduating class, you have a spot in the University of Texas system. Though UT-Austin (the flagship) uses the top 8%. Ref: http://catalog.utexas.edu/archive/2015-16/general-information/admission/undergraduate-admission/freshman-admission/ http://catalog.utexas.edu/archive/2015-16/general-informatio...

walshemj 9y ago

Seriously there is no national standard for high school tests in Canada?

cperciva 9y ago
Correct. Not even provincial standards in many cases.
- apricot 9y ago
  Quebec uses something called the R score to rank applicants to universities. The R score of a student is the sum of the student's Z score (how many standard deviations above the mean), plus the group's Z score relative to all students. https://en.wikipedia.org/wiki/R_score https://en.wikipedia.org/wiki/R_score

Bakary 9y ago

>I worry that we're moving back to a world where university admission will depend less on aptitude and more on having parents who have the money and the connections to get you the "right" extracurricular experience. We already moved back to this system and have done so for a long time, in my opinion. There's also a second round of this phenomenon with unpaid internships.