3 ms·

If automatic grading works well, could you also automatically assess the effectiveness of the question/answers themselves? For instance, a teacher creates a ser

If automatic grading works well, could you also automatically assess the effectiveness of the question/answers themselves? For instance, a teacher creates a series of 10 different quizzes over the years that assess for the same knowledge/skill. Rather than simply re-using a static document, could you automatically generate a set of quizzes that assess students knowledge/skills at "basic", "intermediate", and "advanced" levels (distributed automatically based on [potentially] preparatory work scores). I would love to see an automated "quiz creation" engine (perhaps with some minimal instructor curation) that would both quiz/score students - but also allow teachers to integrate more assessment, more often at lower stakes. By significantly reducing the labor involved with assessment, teachers can more effectively create learning opportunities for students (teaching - with all it contains - is time constrained). Certainly, self-leveling assessments already exist and are in common use with cognitive tutors/online education; however, the knowledge/skills they assess must either be very general or specific to the content of the online course. If teachers were tagging the content of their courses - which are likely to teach knowledge/content that is being taught in hundreds/thousands of other school across the country - then such a self-generating assessment engine would be very useful if it could self-modify to accurate assess whatever is being taught in the course. Modifying learning materials and tasks for different student levels is a major pinpoint for teachers, but is very much a stated goal in almost all k12 schools. It is done poorly, if ever - and mostly when legally required by IEP/504 plans. The trend is very much in the direction of increased data/personalization in the classroom, but currently the big investments in data are being made on the administrative side of the k12 industry. There is so much potential for improving the classroom backend to support instructor workflow rather than constraining it with pre-conceived notions of how we work/what our goals are. I hope to see it happen.

fnbr 10y ago
> If automatic grading works well, could you also automatically assess the effectiveness of the question/answers themselves? That is an absolutely fascinating idea. I would love to see someone run with this as a company, or as an internal initiative at the Department of Education- that seems like a brilliant idea. Has anyone done this before? Is there any reason why this isn't a slam dunk?
- germinalphrase 10y ago
  In some capacity, I imagine this is a component in the creation of standardized tests. Likewise in the development of cognitive tutors (though I could probably follow up with an academic acquaintance that develops them at UW-Madison). The trick (and value) would be de-coupling it from specific curriculum. I would be also be intensely interested in exploring this as a domain side professional.
- Denzel 10y ago
  A company I cofounded back in 2013-2014 was trying to do this. Although our plan of attack showed promised, the company crashed and burned for "legal" reasons. For this to work, you need to do three things: (1) get the original exam, (2) tag every question with a series of categories, and (3) get the students' completed exams. You'd think professors/teachers would be excited by the idea of reducing their workload with an automated quiz/exam generator based upon their curriculum. It didn't seem like it. The professors can't be relied upon to tag their exam questions with categories, so the company has to handle #2. To solve problem #1, we reached out to hundreds of professors at multiple universities, explained what we were doing; and asked if they'd be willing to share their corpus of past (and hopefully present) exams. Nope! They didn't want to do it, and when pressed to give an explanation the lines of communication went cold. Not to mention, if we couldn't convince them to share a blank exam, there's no way they would be willing to share their students' completed exams. So, we thought of a different route. My cofounder had taken a year off from Vanderbilt University, where he participated in a fraternity. Not only did his frat have a massive test bank that we could use, he knew plenty of other frats that had even more. It was a start. But convincing frats at other schools to upload their exams was a no-go. They slammed their big Greek doors in our face. There was nowhere else to turn, except the student body itself. But no student wants to give away their exams without an incentive, and we didn't have the money to pay them. So, we setup a marketplace where students could upload their exams and sell them for a split of the sale. Finally, we got some traction. Some students uploaded their exams. Other students bought them. In our first week of release we made almost $1k. We were on track to finally make good on our promise; it looked like we'd be able to hire people to annotate the exams and begin work on our exam generation service. Until that fateful night, while we sat there trailing the logs, and we saw it. A Vandy professor had signed up and bought one of her own exams! (Just to verify they were authentic.) She immediately reported my cofounder to the university's disciplinary department. And they threatened to expel him with a permanent mark if he didn't immediately take down the website. After 3 years of classes, he'd leave with no credits, no diploma, and no other place to go. Like most young kids, he decided to lay his cards down under such massive pressure. After that, we shut the website down, and went our separate ways. Never able to achieve our ultimate vision. Maybe there's a different way to go about it. But without a strategy to compile an extensive corpus of completed exams with tagged questions, I don't know how you would go about it. If you have any ideas, I'd love to hear them. Because not a day goes by where I don't want to take another stab at this.

fnbr 10y ago

> If automatic grading works well, could you also automatically assess the effectiveness of the question/answers themselves? That is an absolutely fascinating idea. I would love to see someone run with this as a company, or as an internal initiative at the Department of Education- that seems like a brilliant idea. Has anyone done this before? Is there any reason why this isn't a slam dunk?

germinalphrase 10y ago
In some capacity, I imagine this is a component in the creation of standardized tests. Likewise in the development of cognitive tutors (though I could probably follow up with an academic acquaintance that develops them at UW-Madison). The trick (and value) would be de-coupling it from specific curriculum. I would be also be intensely interested in exploring this as a domain side professional.
Denzel 10y ago
A company I cofounded back in 2013-2014 was trying to do this. Although our plan of attack showed promised, the company crashed and burned for "legal" reasons. For this to work, you need to do three things: (1) get the original exam, (2) tag every question with a series of categories, and (3) get the students' completed exams. You'd think professors/teachers would be excited by the idea of reducing their workload with an automated quiz/exam generator based upon their curriculum. It didn't seem like it. The professors can't be relied upon to tag their exam questions with categories, so the company has to handle #2. To solve problem #1, we reached out to hundreds of professors at multiple universities, explained what we were doing; and asked if they'd be willing to share their corpus of past (and hopefully present) exams. Nope! They didn't want to do it, and when pressed to give an explanation the lines of communication went cold. Not to mention, if we couldn't convince them to share a blank exam, there's no way they would be willing to share their students' completed exams. So, we thought of a different route. My cofounder had taken a year off from Vanderbilt University, where he participated in a fraternity. Not only did his frat have a massive test bank that we could use, he knew plenty of other frats that had even more. It was a start. But convincing frats at other schools to upload their exams was a no-go. They slammed their big Greek doors in our face. There was nowhere else to turn, except the student body itself. But no student wants to give away their exams without an incentive, and we didn't have the money to pay them. So, we setup a marketplace where students could upload their exams and sell them for a split of the sale. Finally, we got some traction. Some students uploaded their exams. Other students bought them. In our first week of release we made almost $1k. We were on track to finally make good on our promise; it looked like we'd be able to hire people to annotate the exams and begin work on our exam generation service. Until that fateful night, while we sat there trailing the logs, and we saw it. A Vandy professor had signed up and bought one of her own exams! (Just to verify they were authentic.) She immediately reported my cofounder to the university's disciplinary department. And they threatened to expel him with a permanent mark if he didn't immediately take down the website. After 3 years of classes, he'd leave with no credits, no diploma, and no other place to go. Like most young kids, he decided to lay his cards down under such massive pressure. After that, we shut the website down, and went our separate ways. Never able to achieve our ultimate vision. Maybe there's a different way to go about it. But without a strategy to compile an extensive corpus of completed exams with tagged questions, I don't know how you would go about it. If you have any ideas, I'd love to hear them. Because not a day goes by where I don't want to take another stab at this.