4 ms·

after running a few MOOC classes and contrasting them to brick and mortar classes the real problem in my opinion is assessment. the material we prepared was equ

by munin 12y ago

after running a few MOOC classes and contrasting them to brick and mortar classes the real problem in my opinion is assessment. the material we prepared was equal to what we would teach in a college class, but the assignment quality is not. this is I think a fundamental problem of scale. the only assessment tools that we have are multiple choice and regular expression matching short answer quiz questions. coming up with challenging yet fair multiple choice questions that let the learner express they have mastered a concept is extremely difficult. and to make the class rigorous, MOOC instructors would have to re-do that work for each class, which defeats another scaling point. I think that MOOCs solved the wrong problem and because of that aren't the solution to the class. the MOOC is an iteration on the textbook. a domain expert puts a lot of time into the layout, design and presentation of materials and then creates a relatively static asset out of it. it's less static than the old textbook, but you can't walk into the professors office and ask questions for an hour face to face either. we haven't figured out how to make assessment scale. we haven't moved far from the model we used for thousands of years, which is to have a student stand and talk to their instructors/mentors, and their mentors to make a judgement. how can you get a deep sense of a students capabilities this way, online? I don't think it can't be done, I just don't think that MOOCs have really tried, and my current cynical understanding is that they don't really want to try. Platforms like edX, coursera, and OCW seem to stop at "you got video and text what more you want", and stuff like minerva is using technology but keeping the same scaling factor between instructors and students.

benbreen 12y ago
This is spot on. In fact (having TAed a 400-student intro course once) I'm not even sure if assessment scales properly in the larger brick and mortar classes. The professor was using an iClicker for pseudo-pop quizzes at the beginning of class but I noticed that a lot of students would simply give the same answers as their friends next to them. Likewise for the short essay questions in the final and midterm, they had created a massive Google Docs database combining lecture notes which (while on some level showing admirable initiative) also resulted in a number of identical (and, indeed, identically-wrong) answers. I've never taught a MOOC but it seems to me that this sort of hive-mind tendency would be magnified.
- madcaptenor 12y ago
  I think the hive-mind tendency would be mitigated by the fact that in a typical MOOC the students don't know each other, so that sort of coordination is difficult.
- munin 12y ago
  > I'm not even sure if assessment scales properly in the larger brick and mortar classes you're right, it totally doesn't. the system lives with it because the huge intro/survey classes deal with such a superficial level of information that you can measure it with MC/short answer. upper level classes deal with it by increasing the staff/student ratio, either by lowering the number of students or increasing the number of staff. so the MOOC classes where the answers can't be graded by programs, MC or short answer, become this hybrid of advanced material with intro level assessments. I don't think this is to the students benefit. I also think that the MOOC process overall has been corrupted by the same force that corrupted universities, so I don't see the system as providing a more pro-student alternative anytime soon. I don't know what would.
henrik_w 12y ago
In some cases it works (beyond the multiple choice quizzes). In the algorithm courses I took, you have to implement a algorithm, and input a number that proves the implementation is correct. For example, on a graph with, say, 500,000 nodes, what are the sizes of the five largest strongly connected components. This is impossible to guess. I also did a course with peer grading of one assignment (the Financial Markets course from Yale). It actually worked surprisingly well. You asses 5 other students' 300-word summary of a paper (based on 3 fixed questions). I didn't think it would work, but I've changed my mind. Mind you, cheating is still a problem (e.g. somebody posts the correct answer online). And, an even bigger problem is proving that you completed the course, not somebody else.
kaitai 12y ago
"the MOOC is an iteration on the textbook." Spot on, and this is what politicians and administrators don't understand. How many kids learned calculus out of a textbook alone? Not that many. But it's cheap. MOOCs don't want to try, because the multiple-choice mode is cheap. That's the major appeal of MOOCs: cheap. A few alternatives: SPOCs (small private), with discussion weekly and weekly face-to-face meetings between profs and students. I know some folks who ran one; it was effective yet very time-consuming. Programs like NetMath at University of Illinois: self-paced, hires real people to correct & provide feedback on written student work. Online coaching services like CoachingActuaries have adaptive exams and diagnostics, which is at least a step up from the static multiple choice. None of these is free.
- Tenhundfeld 12y ago
  Adaptive exams can be a huge win. I know they can be hard/time-consuming to do really well, but it's fairly easy to make static multiple choice a little more dynamic. For example, imagine an end-of-section quiz with 5 multiple choice questions with 4 answers each. The static way is each student sees the same 5 questions with the same 4 answers to choose from. Instead, you could have a bank of 100 questions, in buckets of 20 covering the same material. So, each student sees one question from each bucket, drawn randomly. And then each question could have 10 answers, with 4 randomly displayed for any given instance of the question, sometimes with multiple right answers, sometimes none, etc. This would require more work, but effort spent on assessment should scale with the number of students, just not linearly. (Right?) The goal is not to make cheating impossible. The goal is to make cheating hard enough that it's easier to just learn the material and answer the question. Furthermore, once you have that bank of 100 questions, you can start running analytics on it to see which variations are harder or easier for students to understand, which topics might need a better explanation in the teaching section, etc. You could eventually be able to generate adaptive exams from this data. Sal Khan of the Khan Academy talks about all these things. I recommend his book: The One World Schoolhouse. (http://www.amazon.com/The-One-World-Schoolhouse-Reimagined/dp/1455508381 http://www.amazon.com/The-One-World-Schoolhouse-Reimagined/d...)
  - munin 12y ago

by munin 12y ago

benbreen 12y ago

This is spot on. In fact (having TAed a 400-student intro course once) I'm not even sure if assessment scales properly in the larger brick and mortar classes. The professor was using an iClicker for pseudo-pop quizzes at the beginning of class but I noticed that a lot of students would simply give the same answers as their friends next to them. Likewise for the short essay questions in the final and midterm, they had created a massive Google Docs database combining lecture notes which (while on some level showing admirable initiative) also resulted in a number of identical (and, indeed, identically-wrong) answers. I've never taught a MOOC but it seems to me that this sort of hive-mind tendency would be magnified.

madcaptenor 12y ago
I think the hive-mind tendency would be mitigated by the fact that in a typical MOOC the students don't know each other, so that sort of coordination is difficult.
munin 12y ago
> I'm not even sure if assessment scales properly in the larger brick and mortar classes you're right, it totally doesn't. the system lives with it because the huge intro/survey classes deal with such a superficial level of information that you can measure it with MC/short answer. upper level classes deal with it by increasing the staff/student ratio, either by lowering the number of students or increasing the number of staff. so the MOOC classes where the answers can't be graded by programs, MC or short answer, become this hybrid of advanced material with intro level assessments. I don't think this is to the students benefit. I also think that the MOOC process overall has been corrupted by the same force that corrupted universities, so I don't see the system as providing a more pro-student alternative anytime soon. I don't know what would.

henrik_w 12y ago

In some cases it works (beyond the multiple choice quizzes). In the algorithm courses I took, you have to implement a algorithm, and input a number that proves the implementation is correct. For example, on a graph with, say, 500,000 nodes, what are the sizes of the five largest strongly connected components. This is impossible to guess. I also did a course with peer grading of one assignment (the Financial Markets course from Yale). It actually worked surprisingly well. You asses 5 other students' 300-word summary of a paper (based on 3 fixed questions). I didn't think it would work, but I've changed my mind. Mind you, cheating is still a problem (e.g. somebody posts the correct answer online). And, an even bigger problem is proving that you completed the course, not somebody else.

kaitai 12y ago

"the MOOC is an iteration on the textbook." Spot on, and this is what politicians and administrators don't understand. How many kids learned calculus out of a textbook alone? Not that many. But it's cheap. MOOCs don't want to try, because the multiple-choice mode is cheap. That's the major appeal of MOOCs: cheap. A few alternatives: SPOCs (small private), with discussion weekly and weekly face-to-face meetings between profs and students. I know some folks who ran one; it was effective yet very time-consuming. Programs like NetMath at University of Illinois: self-paced, hires real people to correct & provide feedback on written student work. Online coaching services like CoachingActuaries have adaptive exams and diagnostics, which is at least a step up from the static multiple choice. None of these is free.

Tenhundfeld 12y ago
Adaptive exams can be a huge win. I know they can be hard/time-consuming to do really well, but it's fairly easy to make static multiple choice a little more dynamic. For example, imagine an end-of-section quiz with 5 multiple choice questions with 4 answers each. The static way is each student sees the same 5 questions with the same 4 answers to choose from. Instead, you could have a bank of 100 questions, in buckets of 20 covering the same material. So, each student sees one question from each bucket, drawn randomly. And then each question could have 10 answers, with 4 randomly displayed for any given instance of the question, sometimes with multiple right answers, sometimes none, etc. This would require more work, but effort spent on assessment should scale with the number of students, just not linearly. (Right?) The goal is not to make cheating impossible. The goal is to make cheating hard enough that it's easier to just learn the material and answer the question. Furthermore, once you have that bank of 100 questions, you can start running analytics on it to see which variations are harder or easier for students to understand, which topics might need a better explanation in the teaching section, etc. You could eventually be able to generate adaptive exams from this data. Sal Khan of the Khan Academy talks about all these things. I recommend his book: The One World Schoolhouse. (http://www.amazon.com/The-One-World-Schoolhouse-Reimagined/dp/1455508381 http://www.amazon.com/The-One-World-Schoolhouse-Reimagined/d...)
- munin 12y ago