3 ms·
Didn't Pangram claim to have a >99% success rate? https://news.ycombinator.com/item?id=48667761 https://news.ycombinator.com/item?id=48667761 From their homep
by andai 4mo ago
Didn't Pangram claim to have a >99% success rate?
https://news.ycombinator.com/item?id=48667761 https://news.ycombinator.com/item?id=48667761
From their homepage:
> Detect AI-generated content with 99.98% accuracy.
Are they wrong?
Though I ran the numbers, and even with a 0.02% false positive rate, that works out to about 6000 students falsely accused every semester, per university.
- lelanthran 4mo ago> > Detect AI-generated content with 99.98% accuracy. > Though I ran the numbers, and even with a 0.02% false positive rate They don't say that the false positive is 0.02%, only that the accuracy is 0.02%. All we know for certain is that the false positive and false negatives added together result in 0.02%.
- andai 4mo agoYeah, my bad. Well they don't exactly guard against that interpretation. "Up to 6000 falsely accused per semester!" doesn't exactly ease the mind!
- Chu4eeno 4mo agoIt's also confirmed in third-party studies like this https://arxiv.org/html/2501.15654v1 https://arxiv.org/html/2501.15654v1 (which also confirms that humans that have experience with LLMs are very good at spotting them).