6 ms·

The trouble with percentages like this is that the last <1% are difficult to achieve but provide the most value. If you're a teacher looking to prevent cheating

The trouble with percentages like this is that the last <1% are difficult to achieve but provide the most value. If you're a teacher looking to prevent cheating by students, you can't take the chance of falsely accusing one student out of 100.

TuringNYC 3y ago
I'll take your point further. If you have a college class of 1000 students and they submit 5 papers a year, that is 5000 papers. 1% miss would be 50 student assignmentss accused of plagiarism. This isnt detecting cats, this is about lives being seriously affected. Further, i'm curious how universal the accuracy is...do certain subsets incur more of the burden?
cabalamat 3y ago
The simple fix is don't determine grade based on coursework (because cheating is too easy), instead use exams.
- sublinear 3y ago
  The best part is that this is already how grades are weighted at any accredited school.
  - jwestbury 3y ago
    Very much education level and field dependent. It's certainly the case that high schools will grade on coursework more. And, further, many fields primarily grade based on papers, because these are the currency of knowledge, and research skills are difficult to test in an exam setting.
    - sublinear 3y ago
      > research skills are difficult to test in an exam setting I understand you're not saying it's impossible, but is it really that much of a pain for a real human to just read the paper?
chaxor 3y ago
Is not even that - this entire idea just misses the point entirely. It doesn't matter if a very particular model $A$ under particular conditions can be identified with another particular model $C$. That can very easily be circumvented by using some other model (Vicuna, GPT-4, Claude, etc) in different conditions to fool the model $C$. It's a constant race that will never be won and shouldn't even really be attempted. One idea for determining if the content is generated by OpenAI that requires far less work is to for OpenAI to just make a database table of (document_id_hash, text_shingle_hash, shingle_weight_by_frequency) to make a system like old plagiarism detection. It's incredibly fast and simple, so uses far less compute than a big ML model to do an approximate but worse method. Yes, yes, the API, other models, etc. But that was already a problem (see above).

TuringNYC 3y ago

I'll take your point further. If you have a college class of 1000 students and they submit 5 papers a year, that is 5000 papers. 1% miss would be 50 student assignmentss accused of plagiarism. This isnt detecting cats, this is about lives being seriously affected. Further, i'm curious how universal the accuracy is...do certain subsets incur more of the burden?

cabalamat 3y ago

The simple fix is don't determine grade based on coursework (because cheating is too easy), instead use exams.

sublinear 3y ago
The best part is that this is already how grades are weighted at any accredited school.
- jwestbury 3y ago
  Very much education level and field dependent. It's certainly the case that high schools will grade on coursework more. And, further, many fields primarily grade based on papers, because these are the currency of knowledge, and research skills are difficult to test in an exam setting.
  - sublinear 3y ago
    > research skills are difficult to test in an exam setting I understand you're not saying it's impossible, but is it really that much of a pain for a real human to just read the paper?

chaxor 3y ago

Is not even that - this entire idea just misses the point entirely. It doesn't matter if a very particular model $A$ under particular conditions can be identified with another particular model $C$. That can very easily be circumvented by using some other model (Vicuna, GPT-4, Claude, etc) in different conditions to fool the model $C$. It's a constant race that will never be won and shouldn't even really be attempted. One idea for determining if the content is generated by OpenAI that requires far less work is to for OpenAI to just make a database table of (document_id_hash, text_shingle_hash, shingle_weight_by_frequency) to make a system like old plagiarism detection. It's incredibly fast and simple, so uses far less compute than a big ML model to do an approximate but worse method. Yes, yes, the API, other models, etc. But that was already a problem (see above).