3 ms·

Speaking as someone who just took a take-home exam and used ChatGPT to complete it (documented, checked with the prof before, etc.) I may have some insight here

Speaking as someone who just took a take-home exam and used ChatGPT to complete it (documented, checked with the prof before, etc.) I may have some insight here. Also relevant: my PhD mentor is leading a synchronous/asynchronous discussion among the local academia folks regarding this specific concern. The short takeaway is that it’s a huge problem when it comes to test taking. Untimed power tests are the gold standard for assessing student knowledge. The epitome of this kind of test (in any discipline) is a take home exam with extensive short/long answer questions. The test is open-internet, open notes, open everything, except for collaboration. It has been proven for a long time that this is the best way to assess whether or not a student has learned the material. The worst alternative is a stressful in-person exam that is closed-everything. This alternative produces many false negatives. Anxiety, slow test taking, etc. cause students who know the material to perform poorly. The issue is detecting cheating. It’s very easy for teachers to administer said worst alternative. An untimed power test, on the other hand, is extremely labor-intensive to produce/mark. Also, cheating is detected by comparing answers between students. This adds another level of complexity compared to multiple choice or the simpler short-answer questions that are delivered during a timed exam. ChatGPT introduces a layer when it comes to detecting cheating that is currently looking like it’s going to hurt students. On its current track, it’s going to make untimed power tests much harder to produce and administer. They’re already so difficult to create that most professors just opt for the simpler timed exam. In an entry-level research-oriented graduate class, the point is to learn the foundational material so that you can progress to more abstract levels. ChatGPT is making it much harder to assess these classes. As far as testing whether someone can solve a more technically-oriented problem that would be seen in industry, I’m with your interpretation.

nickelcitymario 4y ago
I think we need to ask ourselves what we're testing and why. An untimed at-home test has very little value to me, because it's unlike anything you would ever be asked to do for work. And let's not forget the real reason we have schools is to prepare us for work. Never in my entire career have I been sent home to work on something without a deadline. It's always "get this done by noon from your desk". What we SHOULD be testing is people's abilities to find answers, not memorize them. I want someone with amazing Google skills. Because Google knows far more than any one person could ever hope to even scratch the surface of. We need to think about the Internet and computers as tools. We wouldn't ask a carpenter to build a house with nails and a hammer and a saw (etc). Not even on a test, just to see if he could do it. Tradespeople are trained with their tools in hand. For knowledge workers, the Internet is our biggest tool. AI is a tool. If I was a teacher, I'd want to test my students on their ability to make the most of the tools at their disposal, not how they perform in artificially constrained scenarios. EDIT: I could imagine some exceptions that prove the rule. For example, a surgeon should know everything they need to know, because they can't just pull out a phone and Google it when they're elbows deep inside of someone. But to me this proves the rule, because it's about a real world application: If you're testing prospective surgeons, make sure they only have access to the information and tools they would have on the job.
- comfypotato 4y ago
  I appreciate your thoughtful response. I failed to communicate my point. An untimed power test has a deadline. It’s also not about memorization. You use all of the resources at your disposal to complete the test by the deadline. The only rule is that you aren’t allowed to collaborate, and this rule is enforced by comparing the students’ answers after submission of the test. As an aside: I’m not sure if you were trying to argue that a UPT isn’t appropriate for industry. Whether or not a university’s job has to do with industry (or if it’s being done appropriately) is a different discussion I have no interest in pursuing here or elsewhere on HN. That being said, it’s been proven by decades of research universally saying that the UPT is the ultimate tool for ensuring that the universities have done their job. The point of my response was to communicate that ChatGPT has made it much harder to administer these types of exams because it’s so difficult to detect cheating. This is very different than saying ChatGPT is cheating. It’s not; it just makes students’ who use ChatGPT have such similar answers that you can’t tell if they cheated. This forces exams back into in-person, timed settings (without open internet access). It’s a miserable environment for someone with the aforementioned problems (anxiety, slow test-taking, dyslexia, etc.).

nickelcitymario 4y ago

I think we need to ask ourselves what we're testing and why. An untimed at-home test has very little value to me, because it's unlike anything you would ever be asked to do for work. And let's not forget the real reason we have schools is to prepare us for work. Never in my entire career have I been sent home to work on something without a deadline. It's always "get this done by noon from your desk". What we SHOULD be testing is people's abilities to find answers, not memorize them. I want someone with amazing Google skills. Because Google knows far more than any one person could ever hope to even scratch the surface of. We need to think about the Internet and computers as tools. We wouldn't ask a carpenter to build a house with nails and a hammer and a saw (etc). Not even on a test, just to see if he could do it. Tradespeople are trained with their tools in hand. For knowledge workers, the Internet is our biggest tool. AI is a tool. If I was a teacher, I'd want to test my students on their ability to make the most of the tools at their disposal, not how they perform in artificially constrained scenarios. EDIT: I could imagine some exceptions that prove the rule. For example, a surgeon should know everything they need to know, because they can't just pull out a phone and Google it when they're elbows deep inside of someone. But to me this proves the rule, because it's about a real world application: If you're testing prospective surgeons, make sure they only have access to the information and tools they would have on the job.

comfypotato 4y ago
I appreciate your thoughtful response. I failed to communicate my point. An untimed power test has a deadline. It’s also not about memorization. You use all of the resources at your disposal to complete the test by the deadline. The only rule is that you aren’t allowed to collaborate, and this rule is enforced by comparing the students’ answers after submission of the test. As an aside: I’m not sure if you were trying to argue that a UPT isn’t appropriate for industry. Whether or not a university’s job has to do with industry (or if it’s being done appropriately) is a different discussion I have no interest in pursuing here or elsewhere on HN. That being said, it’s been proven by decades of research universally saying that the UPT is the ultimate tool for ensuring that the universities have done their job. The point of my response was to communicate that ChatGPT has made it much harder to administer these types of exams because it’s so difficult to detect cheating. This is very different than saying ChatGPT is cheating. It’s not; it just makes students’ who use ChatGPT have such similar answers that you can’t tell if they cheated. This forces exams back into in-person, timed settings (without open internet access). It’s a miserable environment for someone with the aforementioned problems (anxiety, slow test-taking, dyslexia, etc.).