3 ms·

Great question! There is a few different things we do. First of all, students can flag parts of the feedback for moderation by a teacher. Secondly we calculate

Great question! There is a few different things we do. First of all, students can flag parts of the feedback for moderation by a teacher. Secondly we calculate the inter-rater agreement between reviewers on a submission - that way teachers can look at submissions with a high disagreement. Of the slightly more complex parts, we have some algorithms that can detect biased students (similar to this https://arxiv.org/abs/0711.3964 https://arxiv.org/abs/0711.3964). Another simple trick is that if one reviewer says your work is good, another that it is bad and you don't flag the feedback saying your work is bad, then the review stating your work is good is probably too nice. The final thing we are working with is making sure that students are honest when they assess feedback quality. To do this, we use some natural language processing to estimate the quality of feedback so we can see if our model is in agreement with the student that reviewed the feedback. Basically, we do a lot of automated outlier detection and highlight outliers to teachers for manual moderation. Not all of this is in production yet (in the end, students are actually quite honest), but all of it is working in non-production environments on actual data.

lurker- 9y ago
I used it for one of my courses a little over a year ago; I don't recall seeing any flag, but maybe that's a newish feature. My primary problem was that the professor often created a question or two that 9/10 groups hadn't addressed, and failed to add questions addressing the value of their work. Meaning a group may have created a product worthy of a 95+/100 score, but failed to address some poorly made question by the professor. Although you could argue that flaw lies upon the professor rather than your product :) Another problem was that it was very easy to figure out the owner of the assignment, which basically meant that many of the students would give top score to their friends regardless of the quality of their work. Finally, I hated the psychology that played into effect when grading the peers. Especially because I couldn't help but think that the professor would be able to associate the comments/scores to me (resulting in me giving overly positive scores to terrible assignments).. That being said, I think what you're building is really great, and if I were a professor then I'd likely want to use it myself.
- utdiscant 9y ago
  Thanks for your comment! We love feedback (part of our DNA) ;). One of our biggest challenges is making a product that is flexible for teachers but which is hard to misuse. Bad assessment rubrics give bad student experiences, and unfortunately we can't make them for the teachers. We are in the process of writing a small booklet for teachers on how to make rubrics better. The challenge with anonymity is especially a problem in smaller classes. We make sure to strip metadata from the submissions, so if people don't put their names in the documents, the only way to determine the author is to know the content or the writing style. It sounds like the challenge in your course was that people were working on different projects and that you knew who was working on what? The psychology part is the most tricky. We want to make a setup where the incentives are correct, but where people don't feel scared of giving feedback to each other. The way we primarily do this is through feedback-on-feedback where receivers of feedback are asked to review the quality of the feedback they received, and then we let the teachers moderate this and students flag problems. I don't think we are completely there yet, but I can definitely see that we are making progress on solving this :).
  - lurker- 9y ago
    Yeah, it was a group project with a lot of presentations, so could instantly recognize who the project belonged to just by looking at group number/easily recognizable pictures. Not sure if there were other identifiable information.. but you say peergrade doesn't display this, so I guess here it's again the fault of the professor to let students upload entire documents rather than letting them copy parts of the text.. That being said, I do seem to recall that the uploading process was somewhat simplified, meaning that we had no choice but to upload everything together, rather than splitting the uploading part into different sections. I mean, if the assignment was a research paper, then instead of uploading it all together (usually in the form of a pdf), it could be divided into different sections, e.g. introduction, related works, etc. (with only text it would be much harder to recognize the author, and if person A reviewed introduction of PersonB and related works of PersonC then it would be even more difficult to identify).. this could also go a long way towards solving the poorly made questions by the professor since it would be guaranteed that all the questions were relevant and completed (not sure if that makes sense, and I wouldn't be surprised if I remember wrong and all of this is already possible ^_^) About feedback on feedback, it sounds good in theory, but I recall many wouldn't bother looking at it (especially because most would just leave "ok", "good", "I agree" etc as their grading comment), and those who did would just skim it without taking any actions to correct poor feedback where it was clear the person grading it hadn't bother to read the content in detail. But I of course have no idea if our behavior was the norm rather than the exception :) In fact, one of my biggest issues with grading in general (second only to being graded based on a stupid test/presentation at the end of the semester rather than the assignments completed throughout the course) is that teachers will often be biased, and I think you have a real shot at fixing this by ensuring that even the teachers don't know the author of the assignment they grade.

lurker- 9y ago

I used it for one of my courses a little over a year ago; I don't recall seeing any flag, but maybe that's a newish feature. My primary problem was that the professor often created a question or two that 9/10 groups hadn't addressed, and failed to add questions addressing the value of their work. Meaning a group may have created a product worthy of a 95+/100 score, but failed to address some poorly made question by the professor. Although you could argue that flaw lies upon the professor rather than your product :) Another problem was that it was very easy to figure out the owner of the assignment, which basically meant that many of the students would give top score to their friends regardless of the quality of their work. Finally, I hated the psychology that played into effect when grading the peers. Especially because I couldn't help but think that the professor would be able to associate the comments/scores to me (resulting in me giving overly positive scores to terrible assignments).. That being said, I think what you're building is really great, and if I were a professor then I'd likely want to use it myself.

utdiscant 9y ago
Thanks for your comment! We love feedback (part of our DNA) ;). One of our biggest challenges is making a product that is flexible for teachers but which is hard to misuse. Bad assessment rubrics give bad student experiences, and unfortunately we can't make them for the teachers. We are in the process of writing a small booklet for teachers on how to make rubrics better. The challenge with anonymity is especially a problem in smaller classes. We make sure to strip metadata from the submissions, so if people don't put their names in the documents, the only way to determine the author is to know the content or the writing style. It sounds like the challenge in your course was that people were working on different projects and that you knew who was working on what? The psychology part is the most tricky. We want to make a setup where the incentives are correct, but where people don't feel scared of giving feedback to each other. The way we primarily do this is through feedback-on-feedback where receivers of feedback are asked to review the quality of the feedback they received, and then we let the teachers moderate this and students flag problems. I don't think we are completely there yet, but I can definitely see that we are making progress on solving this :).
- lurker- 9y ago
  Yeah, it was a group project with a lot of presentations, so could instantly recognize who the project belonged to just by looking at group number/easily recognizable pictures. Not sure if there were other identifiable information.. but you say peergrade doesn't display this, so I guess here it's again the fault of the professor to let students upload entire documents rather than letting them copy parts of the text.. That being said, I do seem to recall that the uploading process was somewhat simplified, meaning that we had no choice but to upload everything together, rather than splitting the uploading part into different sections. I mean, if the assignment was a research paper, then instead of uploading it all together (usually in the form of a pdf), it could be divided into different sections, e.g. introduction, related works, etc. (with only text it would be much harder to recognize the author, and if person A reviewed introduction of PersonB and related works of PersonC then it would be even more difficult to identify).. this could also go a long way towards solving the poorly made questions by the professor since it would be guaranteed that all the questions were relevant and completed (not sure if that makes sense, and I wouldn't be surprised if I remember wrong and all of this is already possible ^_^) About feedback on feedback, it sounds good in theory, but I recall many wouldn't bother looking at it (especially because most would just leave "ok", "good", "I agree" etc as their grading comment), and those who did would just skim it without taking any actions to correct poor feedback where it was clear the person grading it hadn't bother to read the content in detail. But I of course have no idea if our behavior was the norm rather than the exception :) In fact, one of my biggest issues with grading in general (second only to being graded based on a stupid test/presentation at the end of the semester rather than the assignments completed throughout the course) is that teachers will often be biased, and I think you have a real shot at fixing this by ensuring that even the teachers don't know the author of the assignment they grade.