3 ms·
Has anyone tried using an LLM as an LLM detector yet?
by moffkalast 4y ago
Has anyone tried using an LLM as an LLM detector yet?
- minimaxir 4y agoThat is the implementation that many of these approaches are doing. It is a very, very hard problem.
- jacobsenscott 4y agoIt is an impossible problem. An llm can produce any combination of words. A human can produce any combination of words.
- flatiron 4y agoIt works…ish. GPT4 is pretty good at detecting what it wrote but I was able to get a false positive with the United States constitution. Or maybe we can go deeper and say maybe it was AI generated?
- jandrese 4y agoSo if you submitted the US Constitution word for word for your assignment you wouldn't expect the cheating detector to flag it? How isn't this flagrant plagiarism?
- function_seven 4y agoI would expect a plagiarism detector to flag it. I would not expect an "AI wrote this" to flag it, because it was not written by AI.
- jandrese 4y agoAn AI detector is basically a plagiarism detector trained on AI datasets. Fundamentally it's about discovering writing that is too similar to existing writing, which is why I'm concerned about false positives.
- function_seven 4y agoYeah, which is why "AI detection" is crap. It's basically the plagiarism detector with the false-positive dampening turned off. (I know, I know, it may not be exactly that under the hood, but I think that's what it amounts to in practice). I think you and I agree. I'm just saying that the "AI wrote this" flag on something that's notoriously not AI-written should be enough to reject outright the use of these detectors. Like a mechanic who warns me that my Tesla doesn't have a muffler. They're technically correct, but I wouldn't trust them to diagnose anything in the future.