4 ms·

> 2. Non-technical users (teachers) treat this like a 100% certainty This is the part that needs to be addressed the most. Teachers can't offload their critic

> 2. Non-technical users (teachers) treat this like a 100% certainty This is the part that needs to be addressed the most. Teachers can't offload their critical reasoning to the computer. They should ask their students to write things in class and get a feeling for what those individual students are capable of. Then those that turn in essays written at 10x their normal writing level will be obvious, without the use of any automated cheat detectors. I was once accused of cheating by a computer; my friend and I both turned in assignments that used do-while loops, which the computer thought was so statistically unlikely that we surely must have worked together on the assignment. But the explanation was straight forward; I had been evangelizing the aesthetic virtue of do-while loops to anybody that would listen to me, and my friend had been persuaded. Thankfully the professor understood this once he compared the two submissions himself and realized we didn't even use the do-while loop in the same part of the program. There was almost no similarity between the two submissions besides the statistically unlikely but completely innocuous use of do-while loops. It's a good thing my professor used common sense instead of blindly trusting the computer.

londons_explore 4y ago
> blindly trusting the computer. Professors blindly trust the computer not out of laziness, but to protect themselves from accusations of unfairness... "The work was detected as plagiarism, but the professor overrode it for the pretty girl in class, but not for me"
- mitchdoogle 4y ago
  Seems like something like this should only be used as a first-level filter. If the writing doesn't pass, it warrants more investigation. If no proof of plagiarism is found, then there's nothing else to do and professor must pass the student
  - TchoBeer 4y ago
    with a 26% true positive rate that seems flawed.
asah 4y ago
seems like this is the future... 1. first day of class, write a N word essay and sign a release permitting this to be used to detect cheating. The essay topic is chosen at random. 2. digitize & feed to learning model, which detects that YOU are cheating. upside: this also helps detect students who are getting help (e.g. parents) downside: arms race as students feed their cheat-essays (memorize their essays?) into AI-detection models that are similarly trained.
- kaibee 4y ago
  The funniest implication here is that the student's writing skill isn't expected to improve.
  - eh9 4y ago
    I was just asking my partner who’s a writer if it would even be fair to train a model based on a student at Nth grade if the whole point is to measure growth. Would there be enough “stylistic tokens” developed in a young person’s writing style?
    - deleted 4y ago
      [deleted]
    - ask_b123 4y ago
      Personally, I feel mildly embarrassed when reading my essays from years prior. And I probably still count as a 'young person'. That said, there's no need to consider changes in years when stylistic choices can change from one day to another depending on one's mood, recent thoughts, relationship with the teacher, etc. That's why I've always been a little confused about how some (philologists?) treat certain ancient texts as not being written by some authors due to the text's style, as if ancient people could not significantly deviate from their usual style.
- Aransentin 4y ago
  > first day of class, write a N word essay Initially I thought you meant having the student write an essay about slurs, as the AI will refuse to output anything like that. Then I realized you meant "N" as in "Number of words". Still, that first idea might actually work; make the students write about hotwiring cars or something that's controversial enough for the AI to ban it but not controversial enough that anybody will actually care.
geph2021 4y ago
ask their students to write things in class and get a feeling for what those individual students are capable of. Then those that turn in essays written at 10x their normal writing level will be obvious I think that's a flawed approach. Plenty of people simply don't perform or think well under imposed time-limited situations. I believe I can write close to 10x better with 10x the time. To be clear, I don't mean writing more, or a longer essay, given more time. Personally, the hardest part of writing is distilling your thoughts down to the most succinct, cogent and engaging text.
- runarberg 4y ago
  The point is for the teacher to get a sense of the students style and capabilities. Even if your home essay is 10x better and 10x more concise as your in class work, a good teacher that knows you—unlike an inference model—will be able to extrapolate and spot commonalities. Also a good teacher (that isn’t overworked) will also talk to students and get a sense of their style and capabilities that way, this allows them to extrapolate even better then a computer could ever hope to.
  - zopa 4y ago
    Sure, but what about all the students with mediocre and/or overworked teachers? If our plan assumes the best-case scenario, we're going to have problems.
    - runarberg 4y ago
      Honestly if we can’t have nice things and we keep skimping out on education, I’d rather we just accept the fact that some will students cheat, then to introduce another subpar technical solution to a societal problem.
- deepspace 4y ago
  > Plenty of people simply don't perform or think well under imposed time-limited situations From first-hand experience, the difference between poor stress-related performance and a total lack of knowledge is night and day. I have personally witnessed students who could not speak or understand the simplest English, and were unable to come up with two coherent sentences in a classroom situation, but turned in graduate level essays. The difference is blindingly obvious.
TheDudeMan 4y ago
You are asking teachers to be good at their job. But is teaching a merit-based profession?
runarberg 4y ago
So the computer’s evaluation model assumed that each student’s learning is independent? That seems like a ludicrous assumption to put in a model like this, unless the model authors have never been in a class setting (which I doubt).
munificent 4y ago
I think you're misunderstanding the primary purpose of essays. Teachers don't have the time to do deep critical reasoning about each student's essay. An essay is only partially an evaluation tool. The primary purpose of an essay is that the act of writing an essay teaches the student critical reasoning and structured thought. Essays would be an effective tool even if they weren't graded at all. Just writing them is most of the value. A big part of the reason they're graded at all is just to force students to actually write them. The main problem with AI generated essays isn't that teachers will lose out on the ability to evaluate their students. It's that students won't do the work and learn the skills they get from doing the work itself. It's like building a robot to do push ups for you. Not only does the teacher no longer know how many push ups you can do, you're no longer exercising your muscles.
- YeGoblynQueenne 4y ago
  >> The primary purpose of an essay is that the act of writing an essay teaches the student critical reasoning and structured thought. Essays would be an effective tool even if they weren't graded at all. Just writing them is most of the value. A big part of the reason they're graded at all is just to force students to actually write them. That's our problem, I think. Education keeps failing to convince students of the need to be educated.
  - deleted 4y ago
    [deleted]
  - BurningFrog 4y ago
    The value of a degree is very clear. The value of an education is much less clear. I'm saying the students are probably right.
    - deleted 4y ago
      [deleted]
  - hndamien 4y ago
    Seem like it would be fairly trivial to make a document writer that measured if a human was doing the typing such that it was much more likely to have been written by a human sitting and thinking at a keyboard. We do it in ad fraud detection all the time at scale with much less willing participants.
busyant 4y ago
I asked chatgpt to write an essay as if it were written by a mediocre 10th grader. It did a reasonably good job. It threw in a little bit of slang and wasn’t particularly formal. Edit. I sometimes tell my students “if you’re going to cheat, don’t give yourself a perfect score, especially if you’ve failed the first exam. It fires off alarm bells.” But the students who struggle usually can’t calibrate a non-suspicious performance. I guess the same applies here.
- Baeocystin 4y ago
  You've touched upon a central issue that is not often addressed in these conversations. People who have difficulty comprehending and composing essays also struggle to work with repeated prompts in AI systems like ChatGPT to reach a solution. I've found in practice that when showing someone how prompting works, their understanding either clicks instantly, or they fail to grasp it at all. There appears to be very little in between.
userbinator 4y ago
Programming is fortunately one of those subjects where there's something objectively close to a correct/optimal solution. A trivial example is that there aren't very many sane ways to write a "Hello world" program, but this seems to hold for more complex tasks too. In fact, in my experience, the ones who cheat and get it wrong are the most obvious. Unfortunately, the software industry also has plenty of literal tools who are far too trusting of what the computer says (or authority in general, but that's another rant...)
Gigachad 4y ago
I once got called up because my work was flagged as 100% copied. I had uploaded it, made a mistake so I deleted it and uploaded a new file. Second file was flagged as copied. Was able to explain it by pointing at the screen that was claiming I plagiarized my own name.

londons_explore 4y ago

> blindly trusting the computer. Professors blindly trust the computer not out of laziness, but to protect themselves from accusations of unfairness... "The work was detected as plagiarism, but the professor overrode it for the pretty girl in class, but not for me"

mitchdoogle 4y ago
Seems like something like this should only be used as a first-level filter. If the writing doesn't pass, it warrants more investigation. If no proof of plagiarism is found, then there's nothing else to do and professor must pass the student
- TchoBeer 4y ago
  with a 26% true positive rate that seems flawed.

asah 4y ago

seems like this is the future... 1. first day of class, write a N word essay and sign a release permitting this to be used to detect cheating. The essay topic is chosen at random. 2. digitize & feed to learning model, which detects that YOU are cheating. upside: this also helps detect students who are getting help (e.g. parents) downside: arms race as students feed their cheat-essays (memorize their essays?) into AI-detection models that are similarly trained.

kaibee 4y ago
The funniest implication here is that the student's writing skill isn't expected to improve.
- eh9 4y ago
  I was just asking my partner who’s a writer if it would even be fair to train a model based on a student at Nth grade if the whole point is to measure growth. Would there be enough “stylistic tokens” developed in a young person’s writing style?
  - deleted 4y ago
    [deleted]
  - ask_b123 4y ago
    Personally, I feel mildly embarrassed when reading my essays from years prior. And I probably still count as a 'young person'. That said, there's no need to consider changes in years when stylistic choices can change from one day to another depending on one's mood, recent thoughts, relationship with the teacher, etc. That's why I've always been a little confused about how some (philologists?) treat certain ancient texts as not being written by some authors due to the text's style, as if ancient people could not significantly deviate from their usual style.
Aransentin 4y ago
> first day of class, write a N word essay Initially I thought you meant having the student write an essay about slurs, as the AI will refuse to output anything like that. Then I realized you meant "N" as in "Number of words". Still, that first idea might actually work; make the students write about hotwiring cars or something that's controversial enough for the AI to ban it but not controversial enough that anybody will actually care.

geph2021 4y ago

ask their students to write things in class and get a feeling for what those individual students are capable of. Then those that turn in essays written at 10x their normal writing level will be obvious I think that's a flawed approach. Plenty of people simply don't perform or think well under imposed time-limited situations. I believe I can write close to 10x better with 10x the time. To be clear, I don't mean writing more, or a longer essay, given more time. Personally, the hardest part of writing is distilling your thoughts down to the most succinct, cogent and engaging text.

runarberg 4y ago
The point is for the teacher to get a sense of the students style and capabilities. Even if your home essay is 10x better and 10x more concise as your in class work, a good teacher that knows you—unlike an inference model—will be able to extrapolate and spot commonalities. Also a good teacher (that isn’t overworked) will also talk to students and get a sense of their style and capabilities that way, this allows them to extrapolate even better then a computer could ever hope to.
- zopa 4y ago
  Sure, but what about all the students with mediocre and/or overworked teachers? If our plan assumes the best-case scenario, we're going to have problems.
  - runarberg 4y ago
    Honestly if we can’t have nice things and we keep skimping out on education, I’d rather we just accept the fact that some will students cheat, then to introduce another subpar technical solution to a societal problem.
deepspace 4y ago
> Plenty of people simply don't perform or think well under imposed time-limited situations From first-hand experience, the difference between poor stress-related performance and a total lack of knowledge is night and day. I have personally witnessed students who could not speak or understand the simplest English, and were unable to come up with two coherent sentences in a classroom situation, but turned in graduate level essays. The difference is blindingly obvious.

TheDudeMan 4y ago

You are asking teachers to be good at their job. But is teaching a merit-based profession?

runarberg 4y ago

So the computer’s evaluation model assumed that each student’s learning is independent? That seems like a ludicrous assumption to put in a model like this, unless the model authors have never been in a class setting (which I doubt).

munificent 4y ago

I think you're misunderstanding the primary purpose of essays. Teachers don't have the time to do deep critical reasoning about each student's essay. An essay is only partially an evaluation tool. The primary purpose of an essay is that the act of writing an essay teaches the student critical reasoning and structured thought. Essays would be an effective tool even if they weren't graded at all. Just writing them is most of the value. A big part of the reason they're graded at all is just to force students to actually write them. The main problem with AI generated essays isn't that teachers will lose out on the ability to evaluate their students. It's that students won't do the work and learn the skills they get from doing the work itself. It's like building a robot to do push ups for you. Not only does the teacher no longer know how many push ups you can do, you're no longer exercising your muscles.

YeGoblynQueenne 4y ago
>> The primary purpose of an essay is that the act of writing an essay teaches the student critical reasoning and structured thought. Essays would be an effective tool even if they weren't graded at all. Just writing them is most of the value. A big part of the reason they're graded at all is just to force students to actually write them. That's our problem, I think. Education keeps failing to convince students of the need to be educated.
- deleted 4y ago
  [deleted]
- BurningFrog 4y ago
  The value of a degree is very clear. The value of an education is much less clear. I'm saying the students are probably right.
  - deleted 4y ago
    [deleted]
- hndamien 4y ago
  Seem like it would be fairly trivial to make a document writer that measured if a human was doing the typing such that it was much more likely to have been written by a human sitting and thinking at a keyboard. We do it in ad fraud detection all the time at scale with much less willing participants.

busyant 4y ago

I asked chatgpt to write an essay as if it were written by a mediocre 10th grader. It did a reasonably good job. It threw in a little bit of slang and wasn’t particularly formal. Edit. I sometimes tell my students “if you’re going to cheat, don’t give yourself a perfect score, especially if you’ve failed the first exam. It fires off alarm bells.” But the students who struggle usually can’t calibrate a non-suspicious performance. I guess the same applies here.

Baeocystin 4y ago
You've touched upon a central issue that is not often addressed in these conversations. People who have difficulty comprehending and composing essays also struggle to work with repeated prompts in AI systems like ChatGPT to reach a solution. I've found in practice that when showing someone how prompting works, their understanding either clicks instantly, or they fail to grasp it at all. There appears to be very little in between.

userbinator 4y ago

Programming is fortunately one of those subjects where there's something objectively close to a correct/optimal solution. A trivial example is that there aren't very many sane ways to write a "Hello world" program, but this seems to hold for more complex tasks too. In fact, in my experience, the ones who cheat and get it wrong are the most obvious. Unfortunately, the software industry also has plenty of literal tools who are far too trusting of what the computer says (or authority in general, but that's another rant...)

Gigachad 4y ago

I once got called up because my work was flagged as 100% copied. I had uploaded it, made a mistake so I deleted it and uploaded a new file. Second file was flagged as copied. Was able to explain it by pointing at the screen that was claiming I plagiarized my own name.