4 ms·

Everything in this thread just randomizes who doesn't get graded fairly.

by spullara 2y ago

jibe 2y ago
For a single assignment, yes. But at least randomization might mitigate the effect across a term.
karaterobot 2y ago
Is there a better solution? It's not for teachers to be perfect. Since that's not possible, it's not a solution.
- thaumasiotes 2y ago
  Yes, you can grade objectively.
- jtriangle 2y ago
  No, the solution is for the scoring to be handled by software that doesn't exist yet. Some things have easy, objective measures of correctness. STEM is mostly this way. Others, your humanities et al, are fairly subjective. You could probably cover most of this with an LLM, and access to a large body of graded material for a given course, provided said material was graded fairly. Generating that data would be time consuming, as, any given assignment would need to be graded by as many people as possible in order to find a fair average. From there, it's simple comparison between your sample work and the presented work. We're probably a decade from this really being viable en masse, but, it no doubt will happen, and for better or worse we'll likely end up with EDUAAS (education as a service).
  - CuriousCosmic 2y ago
    LLMs are not going to be a solution. LLMs have absolutely no concept of truth. And not everything has an objective solution. Even those that do often have a process associated with them and factoring in that work/process is an important part of grading. Reducing that subjective grading process to only objective solutions being right is grossly reductive and disproportionately punishes students who have the process right and understand the material but make small errors. That's exactly what you don't want to do. --- Instead the solution is to make sure each assignment gets multiple eyes on it and in a random order. Then to document biases and trends in biases so that the TAs and professors can be aware of them and mitigate them. It's a process problem that can only be solved by a process solution. Replacing the graders with technology or reducing problems to a binary right/wrong will never ever solve this and in many cases will end up being more harmful than the biases they claim to solve would be.
    - coredog64 2y ago
      The LLM can compile verbose prose down to a short summary. If the summaries of each chunk are consistent, then it’s at least structurally well written. Then you grade the summary itself.
      - brewdad 2y ago
        At that point you are grading the work of the LLM, not the student.
- CuriousCosmic 2y ago
  Probably it would be something like as follows: Have a group of N graders. And a parity of k. Let's say N is 6 and k is 2. Randomly shuffle the assignments and partition the assignments into N groups. Each grader gets assigned k of the N groups such that they share at most 1 overlap with any other grader and each group is assigned to k people. The assignment orders are shuffled for each grader. They mark up and then grade the assignments. Then for each of the N groups, randomly shuffle the group and equally distribute the assignments to the N-k graders. Now each grader reviews the assignment grades/markups (in random order) and assigns a grade based on the k grades/markups from the previous rounds along with a rationale for the grade assigned. From there the student receives the final assigned grade, the rationale for the grade, and the k markups. If they have a complaint they can go to the professor (who then can also see the k initial grades along with everything else) to dispute the grade for the assignment. --- This way each TA only has to mark up (class size * k / N) assignments, and review (class size / N) assignments to assign a final grade (which should take far less time to do than the initial markups). On top of that every assignment has a guaranteed (k + 1) separate eyes on it. And then the professors can serve as an unbiased arbiter while retaining all the context from the process. To take it an additional step further, the professors could sample a random subset of the assignments to verify the markup and grading is going properly. And those reviews/grade adjustments can then be recorded (along with the final grade/rationales) to document how a given TA's grading deviates from the final reviewed grade or the grade the professor assigns. Likewise for a TA's final assigned grade deviating from the professor's. This would allow deviations to be mitigated over time and major deviations to be identified.
  - spullara 2y ago
    This is amazing.
- stevage 2y ago
  Automatically unskew the results after grading based on this finding?
- spullara 2y ago
  Have an LLM at 0 temperature with the correct answers in context grade it?

Everything in this thread just randomizes who doesn't get graded fairly.

jibe 2y ago

For a single assignment, yes. But at least randomization might mitigate the effect across a term.

karaterobot 2y ago

Is there a better solution? It's not for teachers to be perfect. Since that's not possible, it's not a solution.

thaumasiotes 2y ago
Yes, you can grade objectively.
jtriangle 2y ago
No, the solution is for the scoring to be handled by software that doesn't exist yet. Some things have easy, objective measures of correctness. STEM is mostly this way. Others, your humanities et al, are fairly subjective. You could probably cover most of this with an LLM, and access to a large body of graded material for a given course, provided said material was graded fairly. Generating that data would be time consuming, as, any given assignment would need to be graded by as many people as possible in order to find a fair average. From there, it's simple comparison between your sample work and the presented work. We're probably a decade from this really being viable en masse, but, it no doubt will happen, and for better or worse we'll likely end up with EDUAAS (education as a service).
- CuriousCosmic 2y ago
  LLMs are not going to be a solution. LLMs have absolutely no concept of truth. And not everything has an objective solution. Even those that do often have a process associated with them and factoring in that work/process is an important part of grading. Reducing that subjective grading process to only objective solutions being right is grossly reductive and disproportionately punishes students who have the process right and understand the material but make small errors. That's exactly what you don't want to do. --- Instead the solution is to make sure each assignment gets multiple eyes on it and in a random order. Then to document biases and trends in biases so that the TAs and professors can be aware of them and mitigate them. It's a process problem that can only be solved by a process solution. Replacing the graders with technology or reducing problems to a binary right/wrong will never ever solve this and in many cases will end up being more harmful than the biases they claim to solve would be.
  - coredog64 2y ago
    The LLM can compile verbose prose down to a short summary. If the summaries of each chunk are consistent, then it’s at least structurally well written. Then you grade the summary itself.
    - brewdad 2y ago
      At that point you are grading the work of the LLM, not the student.
CuriousCosmic 2y ago
Probably it would be something like as follows: Have a group of N graders. And a parity of k. Let's say N is 6 and k is 2. Randomly shuffle the assignments and partition the assignments into N groups. Each grader gets assigned k of the N groups such that they share at most 1 overlap with any other grader and each group is assigned to k people. The assignment orders are shuffled for each grader. They mark up and then grade the assignments. Then for each of the N groups, randomly shuffle the group and equally distribute the assignments to the N-k graders. Now each grader reviews the assignment grades/markups (in random order) and assigns a grade based on the k grades/markups from the previous rounds along with a rationale for the grade assigned. From there the student receives the final assigned grade, the rationale for the grade, and the k markups. If they have a complaint they can go to the professor (who then can also see the k initial grades along with everything else) to dispute the grade for the assignment. --- This way each TA only has to mark up (class size * k / N) assignments, and review (class size / N) assignments to assign a final grade (which should take far less time to do than the initial markups). On top of that every assignment has a guaranteed (k + 1) separate eyes on it. And then the professors can serve as an unbiased arbiter while retaining all the context from the process. To take it an additional step further, the professors could sample a random subset of the assignments to verify the markup and grading is going properly. And those reviews/grade adjustments can then be recorded (along with the final grade/rationales) to document how a given TA's grading deviates from the final reviewed grade or the grade the professor assigns. Likewise for a TA's final assigned grade deviating from the professor's. This would allow deviations to be mitigated over time and major deviations to be identified.
- spullara 2y ago
  This is amazing.
stevage 2y ago
Automatically unskew the results after grading based on this finding?
spullara 2y ago
Have an LLM at 0 temperature with the correct answers in context grade it?