4 ms·

1. See "competency-based education/learning". I have some experience doing this. First, it's really hard to do in American schools because of the grade progress

1. See "competency-based education/learning". I have some experience doing this. First, it's really hard to do in American schools because of the grade progression system enshrined in funding formulas through rigid scheduling of standardized tests by age (instead of by competency level). Second, software can keep track but definitely can't make the assessment. And keeping track isn't really valuable; teachers can do that today with a really simple Excel spreadsheet. So I don't really see a big role for software here. 2. There's a lot of research on extrinsic motivation in K-12 education. I'm skeptical, to say the least.

jay_kyburz 6y ago
Ok, so many things to comment on here. Firstly, there are a lot of other countries than America. Secondly, re grade progression system, the issue is not what broken system you have now, we should strive to improve the education of our children. There are clearly a million was that software can make assessment better. Software will eventually be able to read an essay and provide feedback on how well the argument was made, as well as the nuts and bolts of how sentences are put together. Software never gets tired after reading 30 other essays. Never grumpy, never phones it in. Teachers are human. We study intrinsic vs extrinsic motivation in the games industry as well, but to be clear, _almost all_ motivation in education is extrinsic now. Nobody is doing their math homework because its fun. They do it so they won't "fail". But I agree that just bolting some lame point system on top of exiting work is mostly a waist of time. The work and rewards need to be tightly coupled and the hard part is to blur the line between intrinsic and extrinsic motivations. Imagine for example the task is to convince an AI to buy you a pony. You have to write a persuasive essay as to why you should have a pony. The AI can respond to your essay with their own arguments. You can then respond to their augments and pushing the discussion forward. You could do it every day in 10th grade. How good would you be at persuasive writing?
- throwawaygh 6y ago
  Yes to everything about the US system having some serious problems and being fairly unique relative to the rest of the world. > Software will eventually be able to read an essay and provide feedback on how well the argument was made, as well as the nuts and bolts of how sentences are put together... Imagine for example the task is to convince an AI to buy you a pony. You have to write a persuasive essay as to why you should have a pony. The AI can... No, it can't. Giving good feedback on essays in high school English or History courses is AI-complete. "Some type of feedback" is already possible (you can go buy essay graders today -- go try one), but if we go that route we'll mostly just be teaching people how to optimize for a (very broken) grading algorithm. "'The AI' will then <do thing that we spend years teaching people how to do well>" is exactly how you end up with horribly shitty ML code destroying people's lives. TBH we're not even there with high school math yet, which is a lot easier than high school essays. I have a CS PhD, publish in AI conferences, and have taught high school. I am fundamentally absolutely certain that high-quality grading of high school English/History essays is not possible. I'm also certain that we could build "something" that "grades essays" and will have catastrophic negative outcomes for a generation of students if adopted. Students will learn how to write for one really dumb algorithm, and then there will be years of cleanup at universities and workplaces afterward teaching them how to actually write for humans. > but to be clear, _almost all_ motivation in education is extrinsic now. Nobody is doing their math homework because its fun. They do it so they won't "fail". My classroom experience suggests that a) this isn't true, and b) there are lots of different types of extrinsic motivation; grades are actually not the most important motivator for most students. People are really complex, and everyone has different motivators.
  - jay_kyburz 6y ago
    >high-quality grading of high school English/History essays is not possible. I didn't mean to imply we could do it now, but somebody will eventually spends hundred of millions of dollars on this and we'll get there. Machine Learning is not what I mean when I say AI. We need AI's that actually understand text in the same way we do. All jobs will go to machines eventually.
    - throwawaygh 6y ago
      > All jobs will go to machines eventually. IMO this claim is mostly indistinguishable from claims like "God exists".
  - mncharity 6y ago
    In addition to human graders, and purely automated graders, there's also the option of hybrid human-machine systems. Some aspects of the grading might be offloaded from humans, as the range of tasks and cases is often diverse. Or multiple less skilled humans substituted for more expensive ones. In math for instance, there's no need for a human to see a blank answer, or a clearly correct one. A system can assess graders as well as students. Might suggest to a professor which subset of gradings by graduate students could use a quality check. Using non-AI-complete statistics like "when this person grades late at night, and takes a long time on an answer, but doesn't write much, and the student is someone flagged as being at risk, then the odds increase that the professor will want to have seen it". Or perhaps particular cases are best handled by a particular grader. Or should always get an independent cross check of two graders, to reduce grading variance and bias. That a hybrid system might hypothetically support good grading, does not imply that that's the direction which often-unfortunate incentives will drive the tech. Years ago I saw a paper, about the post-WWII defense industry investment in CNC tech. One argument was there were other options, like recording and playing back a skilled machinist, which might have been widely deployed decades earlier. But getting rid of the dependency on skilled workmen was a management objective. Creating cheap automated grading may be far more incentivized that attempting harder and less cheap good grading.
    - 6y ago
- lmm 6y ago
  > Secondly, re grade progression system, the issue is not what broken system you have now, we should strive to improve the education of our children. Better assessments won't magically lead to better education if the system continues to ignore the results of those assessments. The US education system doesn't need better assessments, it needs to start actually using the assessment results that it already has.

jay_kyburz 6y ago

Ok, so many things to comment on here. Firstly, there are a lot of other countries than America. Secondly, re grade progression system, the issue is not what broken system you have now, we should strive to improve the education of our children. There are clearly a million was that software can make assessment better. Software will eventually be able to read an essay and provide feedback on how well the argument was made, as well as the nuts and bolts of how sentences are put together. Software never gets tired after reading 30 other essays. Never grumpy, never phones it in. Teachers are human. We study intrinsic vs extrinsic motivation in the games industry as well, but to be clear, _almost all_ motivation in education is extrinsic now. Nobody is doing their math homework because its fun. They do it so they won't "fail". But I agree that just bolting some lame point system on top of exiting work is mostly a waist of time. The work and rewards need to be tightly coupled and the hard part is to blur the line between intrinsic and extrinsic motivations. Imagine for example the task is to convince an AI to buy you a pony. You have to write a persuasive essay as to why you should have a pony. The AI can respond to your essay with their own arguments. You can then respond to their augments and pushing the discussion forward. You could do it every day in 10th grade. How good would you be at persuasive writing?

throwawaygh 6y ago
Yes to everything about the US system having some serious problems and being fairly unique relative to the rest of the world. > Software will eventually be able to read an essay and provide feedback on how well the argument was made, as well as the nuts and bolts of how sentences are put together... Imagine for example the task is to convince an AI to buy you a pony. You have to write a persuasive essay as to why you should have a pony. The AI can... No, it can't. Giving good feedback on essays in high school English or History courses is AI-complete. "Some type of feedback" is already possible (you can go buy essay graders today -- go try one), but if we go that route we'll mostly just be teaching people how to optimize for a (very broken) grading algorithm. "'The AI' will then <do thing that we spend years teaching people how to do well>" is exactly how you end up with horribly shitty ML code destroying people's lives. TBH we're not even there with high school math yet, which is a lot easier than high school essays. I have a CS PhD, publish in AI conferences, and have taught high school. I am fundamentally absolutely certain that high-quality grading of high school English/History essays is not possible. I'm also certain that we could build "something" that "grades essays" and will have catastrophic negative outcomes for a generation of students if adopted. Students will learn how to write for one really dumb algorithm, and then there will be years of cleanup at universities and workplaces afterward teaching them how to actually write for humans. > but to be clear, _almost all_ motivation in education is extrinsic now. Nobody is doing their math homework because its fun. They do it so they won't "fail". My classroom experience suggests that a) this isn't true, and b) there are lots of different types of extrinsic motivation; grades are actually not the most important motivator for most students. People are really complex, and everyone has different motivators.
- jay_kyburz 6y ago
  >high-quality grading of high school English/History essays is not possible. I didn't mean to imply we could do it now, but somebody will eventually spends hundred of millions of dollars on this and we'll get there. Machine Learning is not what I mean when I say AI. We need AI's that actually understand text in the same way we do. All jobs will go to machines eventually.
  - throwawaygh 6y ago
    > All jobs will go to machines eventually. IMO this claim is mostly indistinguishable from claims like "God exists".
- mncharity 6y ago
  In addition to human graders, and purely automated graders, there's also the option of hybrid human-machine systems. Some aspects of the grading might be offloaded from humans, as the range of tasks and cases is often diverse. Or multiple less skilled humans substituted for more expensive ones. In math for instance, there's no need for a human to see a blank answer, or a clearly correct one. A system can assess graders as well as students. Might suggest to a professor which subset of gradings by graduate students could use a quality check. Using non-AI-complete statistics like "when this person grades late at night, and takes a long time on an answer, but doesn't write much, and the student is someone flagged as being at risk, then the odds increase that the professor will want to have seen it". Or perhaps particular cases are best handled by a particular grader. Or should always get an independent cross check of two graders, to reduce grading variance and bias. That a hybrid system might hypothetically support good grading, does not imply that that's the direction which often-unfortunate incentives will drive the tech. Years ago I saw a paper, about the post-WWII defense industry investment in CNC tech. One argument was there were other options, like recording and playing back a skilled machinist, which might have been widely deployed decades earlier. But getting rid of the dependency on skilled workmen was a management objective. Creating cheap automated grading may be far more incentivized that attempting harder and less cheap good grading.
  - 6y ago
lmm 6y ago
> Secondly, re grade progression system, the issue is not what broken system you have now, we should strive to improve the education of our children. Better assessments won't magically lead to better education if the system continues to ignore the results of those assessments. The US education system doesn't need better assessments, it needs to start actually using the assessment results that it already has.