9 ms·

As somebody who has designed some of the tests Paul Graham is complaining about, it really is hard from the other side! For example, it's long been known in th

by knzhou 7y ago

As somebody who has designed some of the tests Paul Graham is complaining about, it really is hard from the other side! For example, it's long been known in the physics education research community that students come away from introductory courses with very little physical understanding, even if they can do the plug and chug problems on typical tests just fine. Students can all recite Newton's third law, but immediately afterward claim that when a truck hits a car, the truck exerts a bigger force. They know the law for the gravitational force, but can't explain what kept astronauts from falling off the moon, since "there's no gravity in space". Another common claim is that a table exerts no force on something sitting on it -- instead of "exerting a force" it's just "getting in the way". For research purposes, we measure physical understanding using a battery of tests, such as the Force Concept Inventory, containing only simple conceptual questions with unambiguous answers. So then everybody asks: if ordinary tests are so hackable, why not just switch to these conceptual ones? But that wouldn't work. There are less than ~100 distinct FCI-style questions. If these conceptual tests were the norm, students would just memorize the answers and parrot them back, with a flimsy understanding that crumples the second any follow-up question is asked. It would be just the same problem as before, except they would be worse computationally, too. The FCI only works as long as it doesn't count for a grade. The problem isn't tests, it's scale. If the people aren't motivated, any standardized measure will miss the mark -- even the entrepreneurship Paul Graham advocates for. God knows I've seen a lot of bullshit in that direction.

rglullis 7y ago
> If the people aren't motivated, any standardized measure will miss the mark Shouldn't that be a sign that trying to come up with standardized measures is a fool's errand? Short of blatant social engineering, I don't see how we can have a system that could improve people's motivation and drive for education. And if we don't have a way to actually change human nature, why do we keep trying to design this system that would magically work for so many different people?
- TeMPOraL 7y ago
  It's a trust issue. It comes from grades being used to evaluate people outside the particular schools they attend. We could give more autonomy to individual schools or teachers in terms of designing tests, but then we'd be facing two problems. One, the grades given by different schools/teachers would not be directly comparable. An A in one school could be awarded only for exceptional circumstances, and in another just for above-local-average. Two, once you start using those grades to judge people in a competitive situation like the job market, you'll have schools selling grades, teachers inflating grades to help their pupils compete against those from schools that sell grades, etc. and the whole system will collapse to uselessness. Standardized testing mitigates these problems by forcing everyone to grade by the same criteria, but in order to do that, the grading must be done on things that are extremely easy to measure - which removes both individual judgement of the teachers and any relation between testing and understanding of the material. The more I think about it, the more I feel it all boils down to scale. The grading system needs to be scaled up with the size of the market people compete on, otherwise it'll be either trivially gamed, or deeply unfair to unlucky people.
  - GCA10 7y ago
    The Big Four accounting firms pay huge attention to college grades and it works well for them. Ditto law firms. And in those fields, the mechanical pursuit of grades -- independent of actual knowledge -- models the actual demands of the job pretty well. Accounting and law firms are looking for people with the stamina and single-minded goal focus to get document-creation tasks done, one after another, to a high degree of "skill." Getting an A on the test and preparing an SEC-compliant bond prospectus are close cousins of each other. In fields where real-world mastery doesn't have much to do with the orderly routines of getting a 97 on the midterm, employers are less likely to hunt for people who got 97s on the midterm. (John Steinbeck stumbled around Stanford for six years and never graduated. But he got more accomplished as a writer than a whole decade worth of creative-writing majors with high GPAs.)
  - rglullis 7y ago
    > It comes from grades being used to evaluate people outside the particular schools they attend. Are they really, or is just this elaborate illusion supported by most decision makers to give some rationalization to their own selection processes? When you have Harvard being sued for discriminating against Asians, how can we keep a straight face and tell the kids that they need to study hard because the grades matter? I think this is part of what PG wants to defend in the essay: why do we actually care about grades at all?
- knzhou 7y ago
  We have to try, because the alternative to standardized testing isn't some egalitarian utopia, it's nepotism.
  - rglullis 7y ago
    Wait, what?! Why?! I am not asking for any utopia. What I am asking is "Instead of setting ourselves for failure every time coming up with some scheme for testing performance, why can't we just use actual performance on the activity that we want to evaluate candidates"? It works for people in sports. It works for trade professions. It works for traders. It works for performers, entertainers, even entrepreneurs. Why can't we have an educational system that does not put people on tracks to mold them and instead just coach them to follow whatever path they seem fit?
    - knzhou 7y ago
      Because proper evaluation takes time and a bad choice carries a huge cost. In the absence of any objective measures, an employer gets around this by just hiring the son of the guy they golf with. Connections. Sports works, but only because sports are extremely popular, with tens of millions of people enthusiastically participating on a regular basis, and accessible to anybody no matter their age and income. This is not true for, say, working at a law firm. You can’t take the best clerk from a pool of a thousand kids who have been clerking since elementary school. It’s either who you know, or who got good grades and test scores.
      - rglullis 7y ago
        > In the absence of any objective measures One would argue that this is the current status. How objective are, e.g, IQ tests when I can take two different tests in a span of two hours and get from 128 to 146? How my grades in Computer Engineering can be used to predict my performance as an Engineer in a country where 85% of the graduates would work in non-technical and more profitable fields? If a company looking for a Quant gets someone with a Master in Math and a good tracking record knocking on their door, would they stop and ask to see if they can get someone with a PhD instead? You are right that "proper evaluation takes time". What I am saying is that standardized tests should not count as proper evaluation.
- Kaiyou 7y ago
  It's pretty straightforward to motivate people. You just have to make that what you teach them has value that they can use immediately. For example, to motivate someone to train in self-defence, just beat them up every week. Now learning self-defence will have value in their life. If they don't get beat up weekly, what's the point of learning self-defence? If it's only once a year, then who cares. Spending time every day to counter something that only happens once a year isn't worth it. It's less trouble to get beat up once a year than it is to spend learning self-defence daily. There just has to be an immediate value in what you teach. If there isn't, there's no point in teaching it. Students will just forget it anyway after the test.
misnome 7y ago
Are there any resources good for finding out more about this and such studies? Myself, I went through this process in physics, and in many cases it was years afterwards that I properly understood the implications and context of the early courses, and only through repeat exposure could grok the actual connections in the problems. It seems especially hard to test and evaluate something that is planting the seeds for a more complete understanding, maybe years down the line.
- knzhou 7y ago
  I liked Randall Knight's Five Easy Lessons, which summarizes findings in the field and also comes with sample exercises and activities for the whole intro curriculum. There's also Mazur's Peer Instruction, which is a little older.
taeric 7y ago
It is amusing how many of my peers still hold to the notion that heavier things fall faster. Not a higher terminal velocity, but the intuition that they accelerate faster. Hilariously tough for folks to shake.
- GCA10 7y ago
  Add air resistance into the mix, and denser materials do fall faster. Coins vs. napkins. Plums vs. crumpled balls of paper I'd actually take the lesson the other way. It's hilariously tough for model-makers to acknowledge real-world noise that prevents their theoretically precise models from delivering reliable results.
  - knzhou 7y ago
    This is exactly why making more FCI-style questions is so hard. They shouldn't be sensitive to things like air resistance -- a question whose answer is completely changed by realistic air resistance isn't a very good one.
  - taeric 7y ago
    I covered that with terminal velocity. And it isn't just falling, but going down a hill. Folks commonly think heavier bikes will go faster down a hill. That isn't how that works.
    - marcosdumay 7y ago
      Faster bikes will go downhill faster. Both the acceleration of the bikes and their terminal velocity are determined by aerodynamics (on very steep hills) and rolling friction (on less steep ones), and those do not change much with bike weight.
    - deleted 7y ago
      [deleted]
  - noobermin 7y ago
    The reality I've learned is your intuition isn't necessarily wrong, it might just need more information. Arisotlian..."physics" isn't wrong neccessarily, it's just a model of motion under certain conditions (everyday conditions on Earth under an atmosphere or/and on friction-full surfaces) Newton's 1st can be thought of a generalization of Arisotle's (or anyone's) intuition...which to make things worse, physics' 101 students are not taught about. One of the things I think was the best ideas I learned in physics is that physics is all about approximations. Theorists tend to forget this annoyingly, but it helps keep naive scientists from using words like "fundamental" without qualifications, or saying garbage statements like "Newton's law X is wrong."
maxerickson 7y ago
Ever look at Keller Plan stuff? I did Physics 2 that way and really liked it. https://en.wikipedia.org/wiki/Keller_Plan https://en.wikipedia.org/wiki/Keller_Plan (also, wow, I just got that Principle Skinner in the Simpsons doesn't accidentally have that name. Face to palm.)
- cookie_monsta 7y ago
  You may be interested in Simon Singh's The Simpsons and their Mathematical Secrets which details the show's long tradition of hiring math geeks as writers and the plethora of math-based in-jokes that has resulted.
ken 7y ago
Science education is hard. I was constantly frustrated in science classes, especially physics, because every 2 minutes I wanted to say "but then why ___?" Every new thing you learn seems to contradict either common sense (this rock clearly falls faster than this paper!), or something else you've been taught (you mean F=ma was never actually true? and which other equations might turn out to need correcting?). I recognize it's not feasible for a class of students to constantly interrupt a lecture. The questions that other students had were never the same ones that I had, either. The problem is indeed scale, but I don't think motivation is a primary factor. Eventually you learn to be quiet and accept what the teacher and the textbook say.
- falcor84 7y ago
  > The problem is indeed scale Hear, hear! When people say that with better technology we could automate away teachers, I say that with better technology we could automate most everything else and spend the time mentoring one another.
contingencies 7y ago
> any standardized measure will miss the mark Goodhart's Law (Popular formulation): When a measure becomes a target, it ceases to be a good measure. ... via https://github.com/globalcitizen/taoup https://github.com/globalcitizen/taoup
gliese1337 7y ago
To illustrate just how hard: much of my work in academia was assisting in designing experiments or analyzing data for other people working on PhDs in "how to design decent tests" (i.e., Instructional Psychology). There was an entire department of the university dedicated to studying just that problem.
xmprt 7y ago
One of the best physics tests that I took was one where each question has a sense of progression. So the first part laid out a fundamental law of physics and the question was to prove something using it. Then using the result from that the problem was tweaked in a few ways for a few more questions until you were far removed from the original concept but could still explain why and how something worked. To take your gravity example, the first question might be something about calculating gravitational force on the ISS. Then explain that it's rotating in space and therefore has centripetal force that accounts for this gravity. Then compute how fast it must be rotating. And from there you can switch to a completely different topic in the same context. Maybe something about EM radiation from the Sun that falls on the space station.
mvaliente2001 7y ago
Yes, it's hard in both sides. I'm one of those people who got good grades in high school physics (I was able to solve problems), and still thought that satellites were kept in orbit because "there's not gravity" (or is too small). And I was genuinely interested in learning. Part of the problem is the lack of time. Both teachers and students need to cover all the topics in one semester/year, and it takes time to assimilate and integrate the knowledge. Add to that that as student, you don't know what's important, even when you're told so, if you're told at all. That's why we can hold contradictory models in our minds. They're good enough to solve some problems, but conceptually wrong. I think that a possible solution would be to keep the grades open. You get a B in physics-101, since you got a flawed model in your mind, you got to physics-201 where new experiences show the problems in your model and force you to correct it, and now you prove you achieved a knowledge that deserves an A. I know, that proposal it's impractical, and time consuming, but I think that's how most of us really learn.
pergadad 7y ago
I do work in a related field and give your answer really sad. The answer is simply that standardised tests are not a suitable measure of learning. They miss most of the important insights and even efforts to open up, like the attempt to measure 'problem solving'in PISA, are damned to stick to very narrow concepts. Standardized testing is good for: - measuring system performance against set standards - giving basic insights into student's average level of knowledge of a subject (if you use a thorough theoretical framework to conceptualise learning outcomes and link test items to this) - maybe measuring school performance against standards - maybe measuring teacher performance against standards But for students this individual measure of an exam is simply not a good measure of learning in any modern education system. Yes it is a useful measure in really bad systems where teachers are bad and teach wrong things, or where eg attendance and exam fraud are huge issues. But modern pedagogy is not about teaching book knowledge which is easily measured. In any Western education system we are mostly past that point. It's about helping students discover own interests and abilities, develop competence and confidence, experiment, fail and try again. This learning is simply too individual and hard to measure for a standardised test. The only option is formative assessment (essentially teachers grading students repeatedly throughout the year on things like effort, achievement of general learning aims, additional development, results produced, presentations, support to other students, etc - a bit how you would measure a good office worker). Or maybe let's do like Finland and Estonia and let's mostly eliminate the testing burden.
woodandsteel 7y ago
The problem is that so much of physics is quite counter-intuitive (that is one reason it took so long to discover). The basic principle of education is to start where the student is and then lead them step-by-step to what you want them to understand. That being the case, I would say physics should start with student's intuitive understanding and then explain, like through examples they can understand, why some of their intuitions are mistaken. Maybe this could even be turned into Socratic questioning.

by knzhou 7y ago

rglullis 7y ago

> If the people aren't motivated, any standardized measure will miss the mark Shouldn't that be a sign that trying to come up with standardized measures is a fool's errand? Short of blatant social engineering, I don't see how we can have a system that could improve people's motivation and drive for education. And if we don't have a way to actually change human nature, why do we keep trying to design this system that would magically work for so many different people?

TeMPOraL 7y ago
It's a trust issue. It comes from grades being used to evaluate people outside the particular schools they attend. We could give more autonomy to individual schools or teachers in terms of designing tests, but then we'd be facing two problems. One, the grades given by different schools/teachers would not be directly comparable. An A in one school could be awarded only for exceptional circumstances, and in another just for above-local-average. Two, once you start using those grades to judge people in a competitive situation like the job market, you'll have schools selling grades, teachers inflating grades to help their pupils compete against those from schools that sell grades, etc. and the whole system will collapse to uselessness. Standardized testing mitigates these problems by forcing everyone to grade by the same criteria, but in order to do that, the grading must be done on things that are extremely easy to measure - which removes both individual judgement of the teachers and any relation between testing and understanding of the material. The more I think about it, the more I feel it all boils down to scale. The grading system needs to be scaled up with the size of the market people compete on, otherwise it'll be either trivially gamed, or deeply unfair to unlucky people.
- GCA10 7y ago
  The Big Four accounting firms pay huge attention to college grades and it works well for them. Ditto law firms. And in those fields, the mechanical pursuit of grades -- independent of actual knowledge -- models the actual demands of the job pretty well. Accounting and law firms are looking for people with the stamina and single-minded goal focus to get document-creation tasks done, one after another, to a high degree of "skill." Getting an A on the test and preparing an SEC-compliant bond prospectus are close cousins of each other. In fields where real-world mastery doesn't have much to do with the orderly routines of getting a 97 on the midterm, employers are less likely to hunt for people who got 97s on the midterm. (John Steinbeck stumbled around Stanford for six years and never graduated. But he got more accomplished as a writer than a whole decade worth of creative-writing majors with high GPAs.)
- rglullis 7y ago
  > It comes from grades being used to evaluate people outside the particular schools they attend. Are they really, or is just this elaborate illusion supported by most decision makers to give some rationalization to their own selection processes? When you have Harvard being sued for discriminating against Asians, how can we keep a straight face and tell the kids that they need to study hard because the grades matter? I think this is part of what PG wants to defend in the essay: why do we actually care about grades at all?
knzhou 7y ago
We have to try, because the alternative to standardized testing isn't some egalitarian utopia, it's nepotism.
- rglullis 7y ago
  Wait, what?! Why?! I am not asking for any utopia. What I am asking is "Instead of setting ourselves for failure every time coming up with some scheme for testing performance, why can't we just use actual performance on the activity that we want to evaluate candidates"? It works for people in sports. It works for trade professions. It works for traders. It works for performers, entertainers, even entrepreneurs. Why can't we have an educational system that does not put people on tracks to mold them and instead just coach them to follow whatever path they seem fit?
  - knzhou 7y ago
    Because proper evaluation takes time and a bad choice carries a huge cost. In the absence of any objective measures, an employer gets around this by just hiring the son of the guy they golf with. Connections. Sports works, but only because sports are extremely popular, with tens of millions of people enthusiastically participating on a regular basis, and accessible to anybody no matter their age and income. This is not true for, say, working at a law firm. You can’t take the best clerk from a pool of a thousand kids who have been clerking since elementary school. It’s either who you know, or who got good grades and test scores.
    - rglullis 7y ago
      > In the absence of any objective measures One would argue that this is the current status. How objective are, e.g, IQ tests when I can take two different tests in a span of two hours and get from 128 to 146? How my grades in Computer Engineering can be used to predict my performance as an Engineer in a country where 85% of the graduates would work in non-technical and more profitable fields? If a company looking for a Quant gets someone with a Master in Math and a good tracking record knocking on their door, would they stop and ask to see if they can get someone with a PhD instead? You are right that "proper evaluation takes time". What I am saying is that standardized tests should not count as proper evaluation.
Kaiyou 7y ago
It's pretty straightforward to motivate people. You just have to make that what you teach them has value that they can use immediately. For example, to motivate someone to train in self-defence, just beat them up every week. Now learning self-defence will have value in their life. If they don't get beat up weekly, what's the point of learning self-defence? If it's only once a year, then who cares. Spending time every day to counter something that only happens once a year isn't worth it. It's less trouble to get beat up once a year than it is to spend learning self-defence daily. There just has to be an immediate value in what you teach. If there isn't, there's no point in teaching it. Students will just forget it anyway after the test.

misnome 7y ago

Are there any resources good for finding out more about this and such studies? Myself, I went through this process in physics, and in many cases it was years afterwards that I properly understood the implications and context of the early courses, and only through repeat exposure could grok the actual connections in the problems. It seems especially hard to test and evaluate something that is planting the seeds for a more complete understanding, maybe years down the line.

knzhou 7y ago
I liked Randall Knight's Five Easy Lessons, which summarizes findings in the field and also comes with sample exercises and activities for the whole intro curriculum. There's also Mazur's Peer Instruction, which is a little older.

taeric 7y ago

It is amusing how many of my peers still hold to the notion that heavier things fall faster. Not a higher terminal velocity, but the intuition that they accelerate faster. Hilariously tough for folks to shake.

GCA10 7y ago
Add air resistance into the mix, and denser materials do fall faster. Coins vs. napkins. Plums vs. crumpled balls of paper I'd actually take the lesson the other way. It's hilariously tough for model-makers to acknowledge real-world noise that prevents their theoretically precise models from delivering reliable results.
- knzhou 7y ago
  This is exactly why making more FCI-style questions is so hard. They shouldn't be sensitive to things like air resistance -- a question whose answer is completely changed by realistic air resistance isn't a very good one.
- taeric 7y ago
  I covered that with terminal velocity. And it isn't just falling, but going down a hill. Folks commonly think heavier bikes will go faster down a hill. That isn't how that works.
  - marcosdumay 7y ago
    Faster bikes will go downhill faster. Both the acceleration of the bikes and their terminal velocity are determined by aerodynamics (on very steep hills) and rolling friction (on less steep ones), and those do not change much with bike weight.
  - deleted 7y ago
    [deleted]
- noobermin 7y ago
  The reality I've learned is your intuition isn't necessarily wrong, it might just need more information. Arisotlian..."physics" isn't wrong neccessarily, it's just a model of motion under certain conditions (everyday conditions on Earth under an atmosphere or/and on friction-full surfaces) Newton's 1st can be thought of a generalization of Arisotle's (or anyone's) intuition...which to make things worse, physics' 101 students are not taught about. One of the things I think was the best ideas I learned in physics is that physics is all about approximations. Theorists tend to forget this annoyingly, but it helps keep naive scientists from using words like "fundamental" without qualifications, or saying garbage statements like "Newton's law X is wrong."

maxerickson 7y ago

Ever look at Keller Plan stuff? I did Physics 2 that way and really liked it. https://en.wikipedia.org/wiki/Keller_Plan https://en.wikipedia.org/wiki/Keller_Plan (also, wow, I just got that Principle Skinner in the Simpsons doesn't accidentally have that name. Face to palm.)

cookie_monsta 7y ago
You may be interested in Simon Singh's The Simpsons and their Mathematical Secrets which details the show's long tradition of hiring math geeks as writers and the plethora of math-based in-jokes that has resulted.

ken 7y ago

Science education is hard. I was constantly frustrated in science classes, especially physics, because every 2 minutes I wanted to say "but then why ___?" Every new thing you learn seems to contradict either common sense (this rock clearly falls faster than this paper!), or something else you've been taught (you mean F=ma was never actually true? and which other equations might turn out to need correcting?). I recognize it's not feasible for a class of students to constantly interrupt a lecture. The questions that other students had were never the same ones that I had, either. The problem is indeed scale, but I don't think motivation is a primary factor. Eventually you learn to be quiet and accept what the teacher and the textbook say.

falcor84 7y ago
> The problem is indeed scale Hear, hear! When people say that with better technology we could automate away teachers, I say that with better technology we could automate most everything else and spend the time mentoring one another.

contingencies 7y ago

> any standardized measure will miss the mark Goodhart's Law (Popular formulation): When a measure becomes a target, it ceases to be a good measure. ... via https://github.com/globalcitizen/taoup https://github.com/globalcitizen/taoup

gliese1337 7y ago

To illustrate just how hard: much of my work in academia was assisting in designing experiments or analyzing data for other people working on PhDs in "how to design decent tests" (i.e., Instructional Psychology). There was an entire department of the university dedicated to studying just that problem.

xmprt 7y ago

One of the best physics tests that I took was one where each question has a sense of progression. So the first part laid out a fundamental law of physics and the question was to prove something using it. Then using the result from that the problem was tweaked in a few ways for a few more questions until you were far removed from the original concept but could still explain why and how something worked. To take your gravity example, the first question might be something about calculating gravitational force on the ISS. Then explain that it's rotating in space and therefore has centripetal force that accounts for this gravity. Then compute how fast it must be rotating. And from there you can switch to a completely different topic in the same context. Maybe something about EM radiation from the Sun that falls on the space station.

mvaliente2001 7y ago

Yes, it's hard in both sides. I'm one of those people who got good grades in high school physics (I was able to solve problems), and still thought that satellites were kept in orbit because "there's not gravity" (or is too small). And I was genuinely interested in learning. Part of the problem is the lack of time. Both teachers and students need to cover all the topics in one semester/year, and it takes time to assimilate and integrate the knowledge. Add to that that as student, you don't know what's important, even when you're told so, if you're told at all. That's why we can hold contradictory models in our minds. They're good enough to solve some problems, but conceptually wrong. I think that a possible solution would be to keep the grades open. You get a B in physics-101, since you got a flawed model in your mind, you got to physics-201 where new experiences show the problems in your model and force you to correct it, and now you prove you achieved a knowledge that deserves an A. I know, that proposal it's impractical, and time consuming, but I think that's how most of us really learn.

pergadad 7y ago

I do work in a related field and give your answer really sad. The answer is simply that standardised tests are not a suitable measure of learning. They miss most of the important insights and even efforts to open up, like the attempt to measure 'problem solving'in PISA, are damned to stick to very narrow concepts. Standardized testing is good for: - measuring system performance against set standards - giving basic insights into student's average level of knowledge of a subject (if you use a thorough theoretical framework to conceptualise learning outcomes and link test items to this) - maybe measuring school performance against standards - maybe measuring teacher performance against standards But for students this individual measure of an exam is simply not a good measure of learning in any modern education system. Yes it is a useful measure in really bad systems where teachers are bad and teach wrong things, or where eg attendance and exam fraud are huge issues. But modern pedagogy is not about teaching book knowledge which is easily measured. In any Western education system we are mostly past that point. It's about helping students discover own interests and abilities, develop competence and confidence, experiment, fail and try again. This learning is simply too individual and hard to measure for a standardised test. The only option is formative assessment (essentially teachers grading students repeatedly throughout the year on things like effort, achievement of general learning aims, additional development, results produced, presentations, support to other students, etc - a bit how you would measure a good office worker). Or maybe let's do like Finland and Estonia and let's mostly eliminate the testing burden.

woodandsteel 7y ago

The problem is that so much of physics is quite counter-intuitive (that is one reason it took so long to discover). The basic principle of education is to start where the student is and then lead them step-by-step to what you want them to understand. That being the case, I would say physics should start with student's intuitive understanding and then explain, like through examples they can understand, why some of their intuitions are mistaken. Maybe this could even be turned into Socratic questioning.