8 ms·
the paper abstract is generated by ai: https://www.pangram.com/history/c480c94f-2d19-4cc3-95e0-c4faa0936cdc?ucc=Bkykg1IArnm https://www.pangram.com/history/c480
by owenpayton 2mo ago
the paper abstract is generated by ai: https://www.pangram.com/history/c480c94f-2d19-4cc3-95e0-c4faa0936cdc?ucc=Bkykg1IArnm https://www.pangram.com/history/c480c94f-2d19-4cc3-95e0-c4fa...
- estearum 2mo agoIt may well be, but also Pangram is ridiculously transparent pseudoscience. Let's not play pretend that it's authoritative, please. Edit: Just for fun I just defeated it with AI-generated text that it reported as 100% human-written. Great stuff. Truly genius scam.
- porridgeraisin 2mo agoIt is not pseudoscience. Have you read about how it works?
- estearum 2mo agoYes, quite a bit actually. It's directly analogous to e.g. "Red sky at night, sailors' delight", i.e. a pseudo-scientific approach that probably has a better-than-random success rate.
- robotresearcher 2mo agoSo it’s a heuristic. Lots of very valuable heuristics exist.
- estearum 2mo agoNone of which is authoritative, per my initial comment.
- Tostino 2mo agoAnd I'd agree if that's what GP said.
- deleted 2mo ago[deleted]
- robotresearcher 2mo agoWhat name do you use for an approach that probably gets a better than random success rate (without guaranteeing that)? I’m leaving out the ‘pseudo-scientific’ part of GP’s description, since it doesn’t entail anything technically.
- Tostino 2mo agoI was talking about the root comment that just said "it's AI because this website said so", not the one about treating it like a heuristic. I agree with that.
- ekelsen 2mo agoI never found a piece of text that pangram claims is 100% AI that to me seems like purely human written text. The only counter example is when a human composed a piece deliberately trying to mimic AI slop. So call it an AI slop detector if you want, but either way it accurately identifies stuff I don't wan't to read.
- estearum 2mo ago> I never found a piece of text that pangram claims is 100% AI that to me seems like purely human written text. Is that the level of sensitivity we're looking for? More than literally 0%? Do you know how crazy it'd be to achieve that, even if they were trying to?
- snitty 2mo agoWhat were the text, your prompt, and model?
- estearum 2mo agoIt was actual work output I generated a few weeks back with Sonnet 5. Not posting here for obvious reasons, but it was nothing special or engineered for Pangram deception.
- ekelsen 2mo agoThe real reason being that it doesn't exist?
- estearum 2mo agoYa bro it's totally not possible to defeat a detection system that, in its own marketing on contrived, game-able benchmarks that they created, claims only a 99.6% true negative rate. It has literally never been done. What a hilarious inversion of credulity.
- ekelsen 2mo agoSo show us the prompt
- estearum 2mo agoAs stated, it's work product. Not a toy "adversarial prompt," so I won't be posting it here. And as stated, even Pangram's own marketing material claims at least a 0.4% false negative rate on their own contrived corpus. A one in 250 event is hardly even rare. You seem quite defensive about Pangram, and it shows up in a few of your comments. Is there a reason for that? If it's extending to outright denial of Pangram's own stated error rates... sheesh... very weird indeed (unless they're paying you, obviously).
- 2mo ago
- dwaltrip 2mo agoPangram 4?
- estearum 2mo agoWhatever is deployed on their website 13 hours ago
- jefftk 2mo agoYes? The author is very clear in the blog post about having used AI very heavily: Up until now you have been reading the words of Anthropic’s Claude, in particular Fable 5. In part, this is because I simply could not have written this piece. ... Although it might seem strange I thought it wrong to impose too much upon the machine, changing its voice to emulate mine. Richard Sutton’s bitter lesson would likely advise as much. Instead it has its unique voice, grating to some perhaps but altogether fitting that it should be able to keep it, and I only gave it advice on what considerations during writing would bridge the gap between its understanding and that of a reader.
- ekelsen 2mo agoI don't know if putting a disclaimer at the very end of the blog, that you also disclaim, should be considered "very clear". "Here is, for those that have read this far, the acknowledgements that used to be in the formal paper but we decided to remove it, mostly because the paper has been reworked and rewritten so many times by us."