5 ms·

There's an even cooler thing you can do with automatic grading. I saw this idea on another HN comment once. The computer can figure out the answers without prog

There's an even cooler thing you can do with automatic grading. I saw this idea on another HN comment once. The computer can figure out the answers without programming them in. Or it can figure out mistakes in the real answer key. The idea is that answers are highly correlated. Better students are more likely to get all answers correct, worse students are more likely to make mistakes. So if you do something like PCA on all the students' answers, the first dimension will represent the quality of the student. And the weights will represent which answers are correct.

eanzenberg 10y ago
That's pretty cool, like automated collaborative grading on a curve.
- toomuchtodo 10y ago
  Machine learning using human expert trainers ;) I wonder if Khan Academy has tried this yet.
TD-Linux 10y ago
Classic multiple choice grading machines will print out the number of incorrect answers on a final sheet passed through, next to each question (where the checkmark normally is). A lot simpler than PCA, but still pretty effective in finding mistakes, or gaps in course material.
deleted 10y ago
[deleted]
tgb 10y ago
Interesting, but how would PCA work with a discretely-valued, unordered, metric-less data like multiple choice answers? (Actual question, not snark, I only know the basics of PCA.)
- gbrown 10y ago
  https://en.wikiversity.org/wiki/Dummy_variable_(statistics) https://en.wikiversity.org/wiki/Dummy_variable_(statistics)
- Houshalter 10y ago
  I would imagine you would assign each multiple choice answer a value of 0 or 1, based on whether they selected that answer or not. So if there was just one question, and the student answered 'A', it would produce a vector like [1, 0, 0, 0]. In fact I believe there are similar but better algorithms than PCA for handling mutually exclusive binary variables. I just don't know what they are called.
germinalphrase 10y ago
If automatic grading works well, could you also automatically assess the effectiveness of the question/answers themselves? For instance, a teacher creates a series of 10 different quizzes over the years that assess for the same knowledge/skill. Rather than simply re-using a static document, could you automatically generate a set of quizzes that assess students knowledge/skills at "basic", "intermediate", and "advanced" levels (distributed automatically based on [potentially] preparatory work scores). I would love to see an automated "quiz creation" engine (perhaps with some minimal instructor curation) that would both quiz/score students - but also allow teachers to integrate more assessment, more often at lower stakes. By significantly reducing the labor involved with assessment, teachers can more effectively create learning opportunities for students (teaching - with all it contains - is time constrained). Certainly, self-leveling assessments already exist and are in common use with cognitive tutors/online education; however, the knowledge/skills they assess must either be very general or specific to the content of the online course. If teachers were tagging the content of their courses - which are likely to teach knowledge/content that is being taught in hundreds/thousands of other school across the country - then such a self-generating assessment engine would be very useful if it could self-modify to accurate assess whatever is being taught in the course. Modifying learning materials and tasks for different student levels is a major pinpoint for teachers, but is very much a stated goal in almost all k12 schools. It is done poorly, if ever - and mostly when legally required by IEP/504 plans. The trend is very much in the direction of increased data/personalization in the classroom, but currently the big investments in data are being made on the administrative side of the k12 industry. There is so much potential for improving the classroom backend to support instructor workflow rather than constraining it with pre-conceived notions of how we work/what our goals are. I hope to see it happen.
- fnbr 10y ago
  > If automatic grading works well, could you also automatically assess the effectiveness of the question/answers themselves? That is an absolutely fascinating idea. I would love to see someone run with this as a company, or as an internal initiative at the Department of Education- that seems like a brilliant idea. Has anyone done this before? Is there any reason why this isn't a slam dunk?
Fiahil 10y ago
> Better students are more likely to get all answers correct, worse students are more likely to make mistakes. Yes! Until you brainfuck them, by putting the correct answer in the column B for the first 10 questions. Once the doubt is there, it's not gonna go anywhere.
patmcguire 10y ago
You'll have trouble with the hard questions, though, the ones toward the end where half the high quality students get it right and the other half fall for the trap answer.
ronald_raygun 10y ago
There is a paper on this! http://www.icml.cc/2012/papers/597.pdf http://www.icml.cc/2012/papers/597.pdf
fnbr 10y ago
Why would the first dimension necessarily represent the quality of the student? I don't see why that should be the case. Note- I'm not challenging you- I don't see why it shouldn't be the case- I just don't follow your reasoning (but I'm quite interested in the claim). Is it because the largest determinant in which answer is correct is the quality of the student?
zakki 10y ago
Doesn't it become a prejudice towards minority answer?

eanzenberg 10y ago

That's pretty cool, like automated collaborative grading on a curve.

toomuchtodo 10y ago
Machine learning using human expert trainers ;) I wonder if Khan Academy has tried this yet.

TD-Linux 10y ago

Classic multiple choice grading machines will print out the number of incorrect answers on a final sheet passed through, next to each question (where the checkmark normally is). A lot simpler than PCA, but still pretty effective in finding mistakes, or gaps in course material.

deleted 10y ago

[deleted]

tgb 10y ago

Interesting, but how would PCA work with a discretely-valued, unordered, metric-less data like multiple choice answers? (Actual question, not snark, I only know the basics of PCA.)

gbrown 10y ago
https://en.wikiversity.org/wiki/Dummy_variable_(statistics) https://en.wikiversity.org/wiki/Dummy_variable_(statistics)
Houshalter 10y ago
I would imagine you would assign each multiple choice answer a value of 0 or 1, based on whether they selected that answer or not. So if there was just one question, and the student answered 'A', it would produce a vector like [1, 0, 0, 0]. In fact I believe there are similar but better algorithms than PCA for handling mutually exclusive binary variables. I just don't know what they are called.

germinalphrase 10y ago

If automatic grading works well, could you also automatically assess the effectiveness of the question/answers themselves? For instance, a teacher creates a series of 10 different quizzes over the years that assess for the same knowledge/skill. Rather than simply re-using a static document, could you automatically generate a set of quizzes that assess students knowledge/skills at "basic", "intermediate", and "advanced" levels (distributed automatically based on [potentially] preparatory work scores). I would love to see an automated "quiz creation" engine (perhaps with some minimal instructor curation) that would both quiz/score students - but also allow teachers to integrate more assessment, more often at lower stakes. By significantly reducing the labor involved with assessment, teachers can more effectively create learning opportunities for students (teaching - with all it contains - is time constrained). Certainly, self-leveling assessments already exist and are in common use with cognitive tutors/online education; however, the knowledge/skills they assess must either be very general or specific to the content of the online course. If teachers were tagging the content of their courses - which are likely to teach knowledge/content that is being taught in hundreds/thousands of other school across the country - then such a self-generating assessment engine would be very useful if it could self-modify to accurate assess whatever is being taught in the course. Modifying learning materials and tasks for different student levels is a major pinpoint for teachers, but is very much a stated goal in almost all k12 schools. It is done poorly, if ever - and mostly when legally required by IEP/504 plans. The trend is very much in the direction of increased data/personalization in the classroom, but currently the big investments in data are being made on the administrative side of the k12 industry. There is so much potential for improving the classroom backend to support instructor workflow rather than constraining it with pre-conceived notions of how we work/what our goals are. I hope to see it happen.

fnbr 10y ago
> If automatic grading works well, could you also automatically assess the effectiveness of the question/answers themselves? That is an absolutely fascinating idea. I would love to see someone run with this as a company, or as an internal initiative at the Department of Education- that seems like a brilliant idea. Has anyone done this before? Is there any reason why this isn't a slam dunk?

Fiahil 10y ago

> Better students are more likely to get all answers correct, worse students are more likely to make mistakes. Yes! Until you brainfuck them, by putting the correct answer in the column B for the first 10 questions. Once the doubt is there, it's not gonna go anywhere.

patmcguire 10y ago

You'll have trouble with the hard questions, though, the ones toward the end where half the high quality students get it right and the other half fall for the trap answer.

ronald_raygun 10y ago

There is a paper on this! http://www.icml.cc/2012/papers/597.pdf http://www.icml.cc/2012/papers/597.pdf

fnbr 10y ago

Why would the first dimension necessarily represent the quality of the student? I don't see why that should be the case. Note- I'm not challenging you- I don't see why it shouldn't be the case- I just don't follow your reasoning (but I'm quite interested in the claim). Is it because the largest determinant in which answer is correct is the quality of the student?

zakki 10y ago

Doesn't it become a prejudice towards minority answer?