7 ms·

We are in an arms race now. Turnitin. Proctoring. Next? Airport style security scanners at the exam hall. I wrote about this arms race recently [1] and where it

We are in an arms race now. Turnitin. Proctoring. Next? Airport style security scanners at the exam hall. I wrote about this arms race recently [1] and where it will lead. The problem is really that, under conditions of self-commodification (reification), intrinsic motivation to learn and be a better person is replaced by extrinsic motivation to appear to be a better person. The experience (simulation) means more than the reality - which is a general trend in Western society now proved by the very existence of the company Meta. https://www.timeshighereducation.com/features/we-cant-teach-technological-dystopia https://www.timeshighereducation.com/features/we-cant-teach-...

jimbob45 5y ago
It seems to me that TurnItIn and proctoring take care of 99% of problems in exams if the exams are well-constructed such that consulting hidden notes wouldn't help much. The 1% of exams that remain problematic are the memorization-style exams that maybe shouldn't still exist in the modern age.
- jvanderbot 5y ago
  Right. there are plenty of exams in tech that require plain old practice and on-the-spot ingenuity. However, for medical and other fields, memorization and recall is key, so it's rife for this kind of cheating. But honestly, if someone were to tattoo the same information onto their arm, there's an argument to be made it was permanently accessible just as much as memory.
  - asdff 5y ago
    I mean it is kind of stupid even for medical. You are never in the middle of the desert having the memorize the mechanical properties of the inner ear. Every doctor there is has reference books in reality. Why not let people just bring those books to their exams if that's how the job actually works in reality? Learning how to consult a reference for information is just another mental offshoring tool like a calculator, so its a little silly when exams force you to work without it.
- BitwiseFool 5y ago
  To an extent, yes. As a programmer who can't remember the syntax for declaring a two dimensional array and populating it with hard-coded strings, I feel somewhat hypocritical when I say that Medicine seems to be valid domain for rote memorization.
  - nkrisc 5y ago
    When was the last time someone’s life depended on you recalling the syntax for a two dimensional array within the next few seconds? I think it’s OK to hold medical professionals to a higher standard.
    - asdff 5y ago
      How often do doctors even need to do this? Everything they do probably has a standard operating procedure printed out in a binder that can be dictated by a nurse. The training should be in learning how to take those instructions and turn it into action, not that as well as having to memorize all the instructions that are going to be referenceable anyhow. Maybe if the training were more like the former than the latter, fewer med students and residents would be pulling all nighters on some amphetamine. Personally, going into an ER I'd rather be met with well slept, focused hands that consult the relevant information, rather than a sleep deprived zombie barking out protocol they hypnotized themselves into remembering.
      - nkrisc 5y ago
        From what I know, not often. But more often than a developer.
- imoverclocked 5y ago
  Sometimes memorization itself is important to a role. How would you test for that, if not an exam that tests memorized facts? From recent personal experience: pilots need to know certain information so that it can be employed on a whim to help solve time-critical problems. The written exam (which precedes a practical exam) definitely requires memorization.
Fogest 5y ago
Some courses seem to put far too much emphasis on the memorization aspect of the courses, which has always seemed silly to me. For so many jobs you are not required to memorize a ton of information and instead can refer to manuals, textbooks, the internet, etc... for those kinds of things. These are also typically the things people forget shortly after the exam, which begs the question of how useful it even is in the first place. For example, I tried out an "Intro to sociology" class. It was a horrible class, and over half the exams questions were things like "Which person came up with xyz theory". Or "What year was xyz theory proposed". So many questions were purely memorization and did little to actually test your understanding of those theories. What I remember out of most courses is not the things I had to memorize, but instead it is the practical components. Those should be the takeaways and things that get tested. Many people also just sucked at memorizing things like that, and I can totally see why cheating is such a problem on these kinds of tests. Some of the bests tests I've taken where I've remembered the most material have been open book tests that allowed me to use all my textbooks and notes that I had taken. They were hard tests which really tested the understanding of the course material.
- zozbot234 5y ago
  Memorization of "raw" facts has become quite easy with spaced-repetition tools like Anki. The best performance on the job always comes from putting together raw memory and deep conceptual understanding. Neither of these is useful without the other.
- andrewzah 5y ago
  Intro classes (i.e. 101 or just 100-level classes) are designed to weed out people or fill cross-class requirements for majors. My degree required some "aesthetic" classes so I took Dance 101; the class was comprised of a diversity of majors + athletes looking for an easy course. It's also a -lot- of work to grade 150-200 students' tests, and they might change their major anyways. Nearly every course I took at a 300/400+ level was about demonstrating a fundamental understanding instead of strict memorization. Open-book tests were more common. And it went down from 150+ students to 20 or 4. Considering my degree is a b.a. in media arts, some senior courses had minimal or no testing at all and instead had large projects. That said, perhaps one of the hardest classes I took was a 300 or 400 level one on Tarantino films, probably to weed out people who were just looking for an easy class. On top of demonstrating understanding through analysis via storyboarding or papers, the quizzes/tests had multiple choice of 5 answers, which always included "all of the above" or "none of the above", and short/long answers/prompts. You really had to have studied each lecture and actually have watched the relevant films. You wouldn't believe how many people didn't watch the films... in a film analysis class. But all of that aside, it was thoroughly interesting and memorable.
robocat 5y ago
I can understand one motivation to cheat: exam grading is very unfair at the individual level because your performance has a high variation depending on factors outside of your control. Those factors are random from the perspective of a test taker: the exact questions, whether you had a flu, family circumstances, and hundreds of other factors that are independent of your ability. Apart from the fact that exams strongly measure the skill of passing exams, yet often poorly measure actual ability for the subject. The unfairness depends on how steep the cutoff is: does 89.9% mean you miss out on an important life goal, whereas 90.1% means you win? I did a quick google to find facts on expected individual exam mark volatility, but couldn’t find anything - what keywords do I need? One pattern to the results I did read is that individual volatility is not even acknowledged - the unwritten assumption is that exams are completely fair and volatility has other dominant causes. Edit: I am an engineer type with some spectrum attributes and I loath cheats, but over time I have seen how important some “dark” skills are such as: deception, judging when to ignore rules, and meta-games. In some roles or countries, perhaps cheating is a good quality? There is a reason the best card games are about deception. Cheaters are also risk takers, and taking appropriate risks is associated with entrepreneurship. Risks with extreme downsides are interesting.
- CrazyStat 5y ago
  Based on my experience teaching I don't think individual variability is that large. In a semester-long course with 3 or 4 exams the top students and bottom students are mostly consistent from exam to exam. That said, of course individual variability does exist. In one graduate-level math class I misremembered one theorem which caused me to immediately lose 40/100 points--there were only 5 questions on the exam, and that theorem was central to two of them. Had those two questions been different I probably would have scored much better. I consider that a poorly-written exam, though, since so much of the score was dependent on recalling one theorem correctly. I somehow still ended up with an A in that course, I can only assume through either creative accounting or a generous grading curve. In any case, I don't know any other method of evaluation which is more fair than exams. Every method of evaluation is subject to similar sources of individual variability and some have other issues in addition. "Fair" in such cases is a mirage. It's not like there's some other objective evaluation that we can use instead of exams, just ones with different biases.
criddell 5y ago
It's also worth thinking about why we use tests and what they actually measure. Malcolm Gladwell (yes, I know) did a podcast[1] on the LSAT that's pretty good. In that podcast, he makes a pretty compelling argument that what the test measures isn't very useful. [1]: https://www.pushkin.fm/episode/the-tortoise-and-the-hare/ https://www.pushkin.fm/episode/the-tortoise-and-the-hare/
- jcranberry 5y ago
  The LSAT is probably one of the few tests where what it measures is directly useful to a prospective lawyer. I find it hard to believe that he can show that reading comprehension, logical ability and analytical ability aren't useful to law school admissions.
  - criddell 5y ago
    His criticism was mostly about the timed nature of it. The time restriction is a very artificial constraint and they end up filtering a lot of potentially great people.
908B64B197 5y ago
> We are in an arms race now. Turnitin. Proctoring. Next? Airport style security scanners at the exam hall. I wrote about this arms race recently [1] and where it will lead. This is completely self-inflicted. When the map becomes the territory and exams are lazily written to reward rote, you are no longer testing for reasoning and understanding. See Caltech as the exact counterpoint with its culture of take home exams and honesty.
dikei 5y ago
Not the full body scanner used by the America's TSA, but the TOEIC testing centers in my country already require candidates to pass through a metal detector when entering the testing room.

jimbob45 5y ago

It seems to me that TurnItIn and proctoring take care of 99% of problems in exams if the exams are well-constructed such that consulting hidden notes wouldn't help much. The 1% of exams that remain problematic are the memorization-style exams that maybe shouldn't still exist in the modern age.

jvanderbot 5y ago
Right. there are plenty of exams in tech that require plain old practice and on-the-spot ingenuity. However, for medical and other fields, memorization and recall is key, so it's rife for this kind of cheating. But honestly, if someone were to tattoo the same information onto their arm, there's an argument to be made it was permanently accessible just as much as memory.
- asdff 5y ago
  I mean it is kind of stupid even for medical. You are never in the middle of the desert having the memorize the mechanical properties of the inner ear. Every doctor there is has reference books in reality. Why not let people just bring those books to their exams if that's how the job actually works in reality? Learning how to consult a reference for information is just another mental offshoring tool like a calculator, so its a little silly when exams force you to work without it.
BitwiseFool 5y ago
To an extent, yes. As a programmer who can't remember the syntax for declaring a two dimensional array and populating it with hard-coded strings, I feel somewhat hypocritical when I say that Medicine seems to be valid domain for rote memorization.
- nkrisc 5y ago
  When was the last time someone’s life depended on you recalling the syntax for a two dimensional array within the next few seconds? I think it’s OK to hold medical professionals to a higher standard.
  - asdff 5y ago
    How often do doctors even need to do this? Everything they do probably has a standard operating procedure printed out in a binder that can be dictated by a nurse. The training should be in learning how to take those instructions and turn it into action, not that as well as having to memorize all the instructions that are going to be referenceable anyhow. Maybe if the training were more like the former than the latter, fewer med students and residents would be pulling all nighters on some amphetamine. Personally, going into an ER I'd rather be met with well slept, focused hands that consult the relevant information, rather than a sleep deprived zombie barking out protocol they hypnotized themselves into remembering.
    - nkrisc 5y ago
      From what I know, not often. But more often than a developer.
imoverclocked 5y ago
Sometimes memorization itself is important to a role. How would you test for that, if not an exam that tests memorized facts? From recent personal experience: pilots need to know certain information so that it can be employed on a whim to help solve time-critical problems. The written exam (which precedes a practical exam) definitely requires memorization.

Fogest 5y ago

Some courses seem to put far too much emphasis on the memorization aspect of the courses, which has always seemed silly to me. For so many jobs you are not required to memorize a ton of information and instead can refer to manuals, textbooks, the internet, etc... for those kinds of things. These are also typically the things people forget shortly after the exam, which begs the question of how useful it even is in the first place. For example, I tried out an "Intro to sociology" class. It was a horrible class, and over half the exams questions were things like "Which person came up with xyz theory". Or "What year was xyz theory proposed". So many questions were purely memorization and did little to actually test your understanding of those theories. What I remember out of most courses is not the things I had to memorize, but instead it is the practical components. Those should be the takeaways and things that get tested. Many people also just sucked at memorizing things like that, and I can totally see why cheating is such a problem on these kinds of tests. Some of the bests tests I've taken where I've remembered the most material have been open book tests that allowed me to use all my textbooks and notes that I had taken. They were hard tests which really tested the understanding of the course material.

zozbot234 5y ago
Memorization of "raw" facts has become quite easy with spaced-repetition tools like Anki. The best performance on the job always comes from putting together raw memory and deep conceptual understanding. Neither of these is useful without the other.
andrewzah 5y ago
Intro classes (i.e. 101 or just 100-level classes) are designed to weed out people or fill cross-class requirements for majors. My degree required some "aesthetic" classes so I took Dance 101; the class was comprised of a diversity of majors + athletes looking for an easy course. It's also a -lot- of work to grade 150-200 students' tests, and they might change their major anyways. Nearly every course I took at a 300/400+ level was about demonstrating a fundamental understanding instead of strict memorization. Open-book tests were more common. And it went down from 150+ students to 20 or 4. Considering my degree is a b.a. in media arts, some senior courses had minimal or no testing at all and instead had large projects. That said, perhaps one of the hardest classes I took was a 300 or 400 level one on Tarantino films, probably to weed out people who were just looking for an easy class. On top of demonstrating understanding through analysis via storyboarding or papers, the quizzes/tests had multiple choice of 5 answers, which always included "all of the above" or "none of the above", and short/long answers/prompts. You really had to have studied each lecture and actually have watched the relevant films. You wouldn't believe how many people didn't watch the films... in a film analysis class. But all of that aside, it was thoroughly interesting and memorable.

robocat 5y ago

I can understand one motivation to cheat: exam grading is very unfair at the individual level because your performance has a high variation depending on factors outside of your control. Those factors are random from the perspective of a test taker: the exact questions, whether you had a flu, family circumstances, and hundreds of other factors that are independent of your ability. Apart from the fact that exams strongly measure the skill of passing exams, yet often poorly measure actual ability for the subject. The unfairness depends on how steep the cutoff is: does 89.9% mean you miss out on an important life goal, whereas 90.1% means you win? I did a quick google to find facts on expected individual exam mark volatility, but couldn’t find anything - what keywords do I need? One pattern to the results I did read is that individual volatility is not even acknowledged - the unwritten assumption is that exams are completely fair and volatility has other dominant causes. Edit: I am an engineer type with some spectrum attributes and I loath cheats, but over time I have seen how important some “dark” skills are such as: deception, judging when to ignore rules, and meta-games. In some roles or countries, perhaps cheating is a good quality? There is a reason the best card games are about deception. Cheaters are also risk takers, and taking appropriate risks is associated with entrepreneurship. Risks with extreme downsides are interesting.

CrazyStat 5y ago
Based on my experience teaching I don't think individual variability is that large. In a semester-long course with 3 or 4 exams the top students and bottom students are mostly consistent from exam to exam. That said, of course individual variability does exist. In one graduate-level math class I misremembered one theorem which caused me to immediately lose 40/100 points--there were only 5 questions on the exam, and that theorem was central to two of them. Had those two questions been different I probably would have scored much better. I consider that a poorly-written exam, though, since so much of the score was dependent on recalling one theorem correctly. I somehow still ended up with an A in that course, I can only assume through either creative accounting or a generous grading curve. In any case, I don't know any other method of evaluation which is more fair than exams. Every method of evaluation is subject to similar sources of individual variability and some have other issues in addition. "Fair" in such cases is a mirage. It's not like there's some other objective evaluation that we can use instead of exams, just ones with different biases.

criddell 5y ago

It's also worth thinking about why we use tests and what they actually measure. Malcolm Gladwell (yes, I know) did a podcast[1] on the LSAT that's pretty good. In that podcast, he makes a pretty compelling argument that what the test measures isn't very useful. [1]: https://www.pushkin.fm/episode/the-tortoise-and-the-hare/ https://www.pushkin.fm/episode/the-tortoise-and-the-hare/

jcranberry 5y ago
The LSAT is probably one of the few tests where what it measures is directly useful to a prospective lawyer. I find it hard to believe that he can show that reading comprehension, logical ability and analytical ability aren't useful to law school admissions.
- criddell 5y ago
  His criticism was mostly about the timed nature of it. The time restriction is a very artificial constraint and they end up filtering a lot of potentially great people.

908B64B197 5y ago

> We are in an arms race now. Turnitin. Proctoring. Next? Airport style security scanners at the exam hall. I wrote about this arms race recently [1] and where it will lead. This is completely self-inflicted. When the map becomes the territory and exams are lazily written to reward rote, you are no longer testing for reasoning and understanding. See Caltech as the exact counterpoint with its culture of take home exams and honesty.

dikei 5y ago

Not the full body scanner used by the America's TSA, but the TOEIC testing centers in my country already require candidates to pass through a metal detector when entering the testing room.