6 ms·

Can I ask what’s the point of testing students in a scenario where they don’t have access to AI tools? As a software developer I use these tools daily, and if I

Can I ask what’s the point of testing students in a scenario where they don’t have access to AI tools? As a software developer I use these tools daily, and if I or my colleagues stopped our productivity would suffer - it sounds like the same genre of thinking that leads to having students write out code by hand because using an IDE is “cheating”. If it would be possible to take a bottom-tier CS student and turn them into a decent programmer using AI tools, the hiring landscape would be /very/ different. These tools aren’t magic, and to me it sounds like the tests are failing at measuring the students competence if they can so easily be gamed by using an AI/old fashioned googling. Maybe the solution should be to move towards a style of exam/grading that actually measures the competence of the student in a situation closer to what a professional developer will be in, rather than an old fashioned artificial exam setting?

yuliyp 3y ago
An evaluation for a course should measure how much of the subjects of the course the students learned. It's not a job success simulator, nor a measure of unrelated skills.
- hxugufjfjf 3y ago
  So the students are disallowed or discouraged from learning how to use tools like AI in the courses they take?
  - watwut 3y ago
    Student are neither disallowed or discouraged from learning them. The are not allowed to use them during unrelated test.
    - hxugufjfjf 3y ago
      Why is the test unrelated to learning methods?
      - noitpmeder 3y ago
        Should we give grade students calculators on their tests that ask for the result of 1+1? Or 10x3?
  - pclmulqdq 3y ago
    This follows the same logic as banning calculators when teaching people basic arithmetic. When something is a replacement for your learning, it should be banned, and when it is an augmentation, it should be encouraged. You may think it's ridiculous, but many people today can't calculate a 20% tip or add two prices together without a calculator because the constant use of calculators has caused their arithmetic skills to atrophy. This seems like a silly gripe, except a lot of other skills are built around these basic ones: for arithmetic, the ability to estimate the cost or time taken to do something is all based on tricks you learn when you are trying to learn mental math. Society is not worse off for this (enough of us still know how to do these things), but many people are poorer, both intellectually and monetarily, due to a lack of arithmetic skills. And no, this lost knowledge of arithmetic is not replaced by a knowledge of higher math - it tends to come with a fear of it. The same applies with text-generating AI. In terms of writing, if you don't learn to write dumb essays about books, skills like learning to construct an argument are much harder to pick up. For people writing code, learning to slog through writing and debugging a doubly linked list (something ChatGPT can reliably generate for you today) leads you to later being able to slog through debugging B-trees or lock-free queues (which ChatGPT definitely cannot write for you). I think there is a very compelling argument along these lines for low-level courses to ban AI tools. However, higher-level courses probably should allow students to add them to their repertoire, where they are an aid and not a crutch. This follows how mathematicians and engineers learn to use calculators and computer algebra tools, which seems to work well.
    - hxugufjfjf 3y ago
      I don’t think it ridiculous at all. I’m not sure if your comment was an argument, a counter-argument or just a comment.
    - Baeocystin 3y ago
      >people today can't calculate a 20% tip Earlier this week, I spent several minutes talking one of my co-workers through how I 'did math in my head' to figure out my half of the 20% tip we were leaving for lunch. Even something as rudimentary as moving the decimal place over one and doubling the result seemed like wizardry to them. They aren't dumb. They've literally just never thought one second past reaching for a calculator. Which is kind of scary, because it means they have no way of sanity checking any numbers they come up with.
codegladiator 3y ago
A person who knows how to code without access to AI tools means when they have access to AI tools their productivity would increase but they will also be able to point out mistakes, if any, in the generated code. A person who does not know how to code without access to AI tools means they will consistently push bugs generated by AI, affecting the teams productivity, if any.
- mdorazio 3y ago
  If this is the case, wouldn't you expect their grades to reflect this as well due to the generated code answering exam questions having errors?
  - jackson1442 3y ago
    The exam questions are likely simpler than an entire application would be; using AI to write bubble sort is trivial but getting it to understand the full context of your application is much less so. ChatGPT can solve most LeetCode questions, but I don't think most people here are ready to replace their junior developers with it.
  - Thiez 3y ago
    Not on a class that teaches programming to beginners, no. Would you?
jlkuester7 3y ago
I at least partially share this sentiment for hiring interviews. In that setting, the goal of any kind of coding exercise is to evaluate a persons coding skills in a real-world environment. For something like that, it does not make much sense to hobble the developer by artificially limiting their access to coding resources that are normally employed while on the job. However, I feel like an academic exam is a bit different. The goal of these exams is to evaluate what students have _learned_. Unless the class was about how to find helpful code examples on SO, it does not make much sense to allow AI/internet-searching during the exam.
JohnBooty 3y ago
Maybe the solution should be to move towards a style of exam/grading that actually measures the competence of the student in a situation closer to what a professional developer will be in, rather than an old fashioned artificial exam setting? Most "real world" software development work involves understanding existing code and/or choosing solutions. AI can help you write a linked list, but can it help you know that you need a linked list vs. a queue or a binary tree? Can it help you to make more architecture-y decisions like deciding whether you need a document database, a traditional SQL RDBMS, or maybe something else? I'm not in a teaching role but those are the kinds of things I'd like to see examinations cover -- what tool or data structure are you choosing, and why is it the appropriate tool for this specific case? Or, similarly, can the student understand existing code and fix bugs or make performance improvements? Bottom line, though: professors should just absolutely accept the reality of AI. Assume all students are using AI tools. Actively mandate or at least encourage their use in order to ensure a level playing field.
- PixyMisa 3y ago
  And then fail them all because none of them have learned anything.
suid 3y ago
> Can I ask what’s the point of testing students in a scenario where they don’t have access to AI tools? It's not just the issue of "what if they didn't have access to ChatGPT?" Just as an example: ChatGPT is literally the world's biggest bullshitter. It vacuums up vast stores of "data" (note: NOT "knowledge"), and just pulls out words that it thinks could likely follow from previous words given a context. Quite often is produces silly, or even dangerously wrong, answers. You need to be pretty well skilled in the art to catch it at its bullshit. It's just like when you hire assistants to do something critical for you. You have to be able to verify their work, or you have to be able to trust them blindly.
WalterBright 3y ago
Think of it this way. I use a calculator to do less trivial arithmetic. However, knowing how to multiply means I understand what multiplication is rather than just a number that inexplicably appears on the calculator screen. I've known engineers who did not understand the analytical tools they were using. They misused them constantly.
jeroenhd 3y ago
This is like asking "what's the point of not allowing students to use Google during their Maths exams" or "why bother with English exams when Grammarly exists". The point is to gain some knowledge yourself. Obviously you'll use new tools and methods later when you eventually get a job, but the point of most courses is to teach you the concepts, not how to command the IDE. Whether you decide to hand-craft assembly code or become professional a Copilot suggestion approver doesn't really matter.
BeetleB 3y ago
> Can I ask what’s the point of testing students in a scenario where they don’t have access to AI tools? It's a question of: Are you testing for understanding or ability? Both have their merits, but often the goal is the former. If it were a project, it is generally the latter.
PixyMisa 3y ago
> Can I ask what’s the point of testing students in a scenario where they don’t have access to AI tools? So that they can tell when the AI tools get things wrong. Which is always.

yuliyp 3y ago

An evaluation for a course should measure how much of the subjects of the course the students learned. It's not a job success simulator, nor a measure of unrelated skills.

hxugufjfjf 3y ago
So the students are disallowed or discouraged from learning how to use tools like AI in the courses they take?
- watwut 3y ago
  Student are neither disallowed or discouraged from learning them. The are not allowed to use them during unrelated test.
  - hxugufjfjf 3y ago
    Why is the test unrelated to learning methods?
    - noitpmeder 3y ago
      Should we give grade students calculators on their tests that ask for the result of 1+1? Or 10x3?
- pclmulqdq 3y ago
  This follows the same logic as banning calculators when teaching people basic arithmetic. When something is a replacement for your learning, it should be banned, and when it is an augmentation, it should be encouraged. You may think it's ridiculous, but many people today can't calculate a 20% tip or add two prices together without a calculator because the constant use of calculators has caused their arithmetic skills to atrophy. This seems like a silly gripe, except a lot of other skills are built around these basic ones: for arithmetic, the ability to estimate the cost or time taken to do something is all based on tricks you learn when you are trying to learn mental math. Society is not worse off for this (enough of us still know how to do these things), but many people are poorer, both intellectually and monetarily, due to a lack of arithmetic skills. And no, this lost knowledge of arithmetic is not replaced by a knowledge of higher math - it tends to come with a fear of it. The same applies with text-generating AI. In terms of writing, if you don't learn to write dumb essays about books, skills like learning to construct an argument are much harder to pick up. For people writing code, learning to slog through writing and debugging a doubly linked list (something ChatGPT can reliably generate for you today) leads you to later being able to slog through debugging B-trees or lock-free queues (which ChatGPT definitely cannot write for you). I think there is a very compelling argument along these lines for low-level courses to ban AI tools. However, higher-level courses probably should allow students to add them to their repertoire, where they are an aid and not a crutch. This follows how mathematicians and engineers learn to use calculators and computer algebra tools, which seems to work well.
  - hxugufjfjf 3y ago
    I don’t think it ridiculous at all. I’m not sure if your comment was an argument, a counter-argument or just a comment.
  - Baeocystin 3y ago
    >people today can't calculate a 20% tip Earlier this week, I spent several minutes talking one of my co-workers through how I 'did math in my head' to figure out my half of the 20% tip we were leaving for lunch. Even something as rudimentary as moving the decimal place over one and doubling the result seemed like wizardry to them. They aren't dumb. They've literally just never thought one second past reaching for a calculator. Which is kind of scary, because it means they have no way of sanity checking any numbers they come up with.

codegladiator 3y ago

A person who knows how to code without access to AI tools means when they have access to AI tools their productivity would increase but they will also be able to point out mistakes, if any, in the generated code. A person who does not know how to code without access to AI tools means they will consistently push bugs generated by AI, affecting the teams productivity, if any.

mdorazio 3y ago
If this is the case, wouldn't you expect their grades to reflect this as well due to the generated code answering exam questions having errors?
- jackson1442 3y ago
  The exam questions are likely simpler than an entire application would be; using AI to write bubble sort is trivial but getting it to understand the full context of your application is much less so. ChatGPT can solve most LeetCode questions, but I don't think most people here are ready to replace their junior developers with it.
- Thiez 3y ago
  Not on a class that teaches programming to beginners, no. Would you?

jlkuester7 3y ago

I at least partially share this sentiment for hiring interviews. In that setting, the goal of any kind of coding exercise is to evaluate a persons coding skills in a real-world environment. For something like that, it does not make much sense to hobble the developer by artificially limiting their access to coding resources that are normally employed while on the job. However, I feel like an academic exam is a bit different. The goal of these exams is to evaluate what students have _learned_. Unless the class was about how to find helpful code examples on SO, it does not make much sense to allow AI/internet-searching during the exam.

JohnBooty 3y ago

Maybe the solution should be to move towards a style of exam/grading that actually measures the competence of the student in a situation closer to what a professional developer will be in, rather than an old fashioned artificial exam setting? Most "real world" software development work involves understanding existing code and/or choosing solutions. AI can help you write a linked list, but can it help you know that you need a linked list vs. a queue or a binary tree? Can it help you to make more architecture-y decisions like deciding whether you need a document database, a traditional SQL RDBMS, or maybe something else? I'm not in a teaching role but those are the kinds of things I'd like to see examinations cover -- what tool or data structure are you choosing, and why is it the appropriate tool for this specific case? Or, similarly, can the student understand existing code and fix bugs or make performance improvements? Bottom line, though: professors should just absolutely accept the reality of AI. Assume all students are using AI tools. Actively mandate or at least encourage their use in order to ensure a level playing field.

PixyMisa 3y ago
And then fail them all because none of them have learned anything.

suid 3y ago

> Can I ask what’s the point of testing students in a scenario where they don’t have access to AI tools? It's not just the issue of "what if they didn't have access to ChatGPT?" Just as an example: ChatGPT is literally the world's biggest bullshitter. It vacuums up vast stores of "data" (note: NOT "knowledge"), and just pulls out words that it thinks could likely follow from previous words given a context. Quite often is produces silly, or even dangerously wrong, answers. You need to be pretty well skilled in the art to catch it at its bullshit. It's just like when you hire assistants to do something critical for you. You have to be able to verify their work, or you have to be able to trust them blindly.

WalterBright 3y ago

Think of it this way. I use a calculator to do less trivial arithmetic. However, knowing how to multiply means I understand what multiplication is rather than just a number that inexplicably appears on the calculator screen. I've known engineers who did not understand the analytical tools they were using. They misused them constantly.

jeroenhd 3y ago

This is like asking "what's the point of not allowing students to use Google during their Maths exams" or "why bother with English exams when Grammarly exists". The point is to gain some knowledge yourself. Obviously you'll use new tools and methods later when you eventually get a job, but the point of most courses is to teach you the concepts, not how to command the IDE. Whether you decide to hand-craft assembly code or become professional a Copilot suggestion approver doesn't really matter.

BeetleB 3y ago

> Can I ask what’s the point of testing students in a scenario where they don’t have access to AI tools? It's a question of: Are you testing for understanding or ability? Both have their merits, but often the goal is the former. If it were a project, it is generally the latter.

PixyMisa 3y ago

> Can I ask what’s the point of testing students in a scenario where they don’t have access to AI tools? So that they can tell when the AI tools get things wrong. Which is always.