4 ms·

Is anything really harder to test in the age of deep learning classifiers ? If Bert can write an essay, it can grade Bart’s essay, or Lisa’s musical performance

Is anything really harder to test in the age of deep learning classifiers ? If Bert can write an essay, it can grade Bart’s essay, or Lisa’s musical performance too.

gipp 4y ago
Sorry, but this is incredibly naive.
eesmith 4y ago
Is your comment hand-waving optimism, or do you have evidence that 1) deep learning classifiers are better than teachers at evaluating student competency across a wide range of fields, and 2) that using those tests (vs, say, teacher-observed in-class participation) results in better alignment to the educational needs of the student? As https://curmudgucation.blogspot.com/2022/01/the-search-for-computerized-essay.html https://curmudgucation.blogspot.com/2022/01/the-search-for-c... points out, "no computer algorithm can, for instance, understand whether or not your supporting evidence provides good, er even accurate, support" for the thesis, which seems right to me. For example, https://www.wbur.org/hereandnow/2020/09/03/online-learning-algorithm https://www.wbur.org/hereandnow/2020/09/03/online-learning-a... has the story of student who answered the question "How did the location of Constantinople help it grow wealthy and prosperous?" with "It was between the Aegean and the Black seas, which made it a hub for boats of traders and passengers. It was also right between Europe and Asia Minor, which made it a huge hub for trade, and it was on many trade routes of the time. Profit Diversity Spain Gaul China India Africa" - and got a perfect score from the robograder. I'll switch to something that is more measurable - physical education. Why isn't physical education included in the PISA score? We can measure someone's speed, weight, height, jump distance, weight lifting ability, and a lot more. Do those give a good metric by which to judge someone's physical education grade? Or do we also need to include teamwork, improvement, and sportsmanship?
- naveen99 4y ago
  Deep learning classifiers can handle essays almost as cheaply as multiple choice questions. Teachers are inconsistent and expensive as graders.
  - eesmith 4y ago
    Where is your evidence? As related counter-evidence, the leaderboard (to "automatically segment texts and classify argumentative and rhetorical elements in essays written by 6th-12th grade students") at https://www.kaggle.com/competitions/feedback-prize-2021/leaderboard https://www.kaggle.com/competitions/feedback-prize-2021/lead... shows the score for the best team entrant is 0.753. Why isn't that very close to 1.0?
ironmagma 4y ago
We need somebody to yell at when things go wrong.

Is anything really harder to test in the age of deep learning classifiers ? If Bert can write an essay, it can grade Bart’s essay, or Lisa’s musical performance too.

gipp 4y ago

Sorry, but this is incredibly naive.

eesmith 4y ago

Is your comment hand-waving optimism, or do you have evidence that 1) deep learning classifiers are better than teachers at evaluating student competency across a wide range of fields, and 2) that using those tests (vs, say, teacher-observed in-class participation) results in better alignment to the educational needs of the student? As https://curmudgucation.blogspot.com/2022/01/the-search-for-computerized-essay.html https://curmudgucation.blogspot.com/2022/01/the-search-for-c... points out, "no computer algorithm can, for instance, understand whether or not your supporting evidence provides good, er even accurate, support" for the thesis, which seems right to me. For example, https://www.wbur.org/hereandnow/2020/09/03/online-learning-algorithm https://www.wbur.org/hereandnow/2020/09/03/online-learning-a... has the story of student who answered the question "How did the location of Constantinople help it grow wealthy and prosperous?" with "It was between the Aegean and the Black seas, which made it a hub for boats of traders and passengers. It was also right between Europe and Asia Minor, which made it a huge hub for trade, and it was on many trade routes of the time. Profit Diversity Spain Gaul China India Africa" - and got a perfect score from the robograder. I'll switch to something that is more measurable - physical education. Why isn't physical education included in the PISA score? We can measure someone's speed, weight, height, jump distance, weight lifting ability, and a lot more. Do those give a good metric by which to judge someone's physical education grade? Or do we also need to include teamwork, improvement, and sportsmanship?

naveen99 4y ago
Deep learning classifiers can handle essays almost as cheaply as multiple choice questions. Teachers are inconsistent and expensive as graders.
- eesmith 4y ago
  Where is your evidence? As related counter-evidence, the leaderboard (to "automatically segment texts and classify argumentative and rhetorical elements in essays written by 6th-12th grade students") at https://www.kaggle.com/competitions/feedback-prize-2021/leaderboard https://www.kaggle.com/competitions/feedback-prize-2021/lead... shows the score for the best team entrant is 0.753. Why isn't that very close to 1.0?

ironmagma 4y ago

We need somebody to yell at when things go wrong.