2 ms·

The false positive rate for Pangram 4 is something like one in 24,000.[0] To put that in perspective, the wrongful-conviction (false positive rate) for death-se

by droidjj 19d ago

The false positive rate for Pangram 4 is something like one in 24,000.[0] To put that in perspective, the wrongful-conviction (false positive rate) for death-sentenced defendants in the US is estimated conservatively to be around 4.1%.[1] The FP rate for death-sentence convictions is 1,000 times bigger than Pangram’s FP rate. Now, the US criminal system is not a great yardstick for justice. But it goes to show you Pangram is really good evidence that something was LLM generated. It can be an amazing tool for enforcing AI policies in schools, and there ought to be ways to use it with caveats for the rare but inevitable false positives (appeals, etc). [0]: see page 15 https://arxiv.org/pdf/2607.27183 https://arxiv.org/pdf/2607.27183 [1]: see https://pmc.ncbi.nlm.nih.gov/articles/PMC4034186/ https://pmc.ncbi.nlm.nih.gov/articles/PMC4034186/

emil-lp 19d ago
> The false positive rate for Pangram 4 is something like one in 24,000. First of all: says them.. Do you think they might have an incentive to boost their numbers? Second, this is on existing text, which might be in their training data, no?
lucb1e 19d ago
> The false positive rate for Pangram 4 is something like one in 24,000. Gotta suck to be one of the 8B/24k=~300k people in the world whose writing pattern is falsely labelled as slop by this tool that people say is so accurate so customers are going to feel really sure about your alleged dishonesty about writing your own texts This false positive rate is a double-edged sword. Please still be careful when accusing people
runako 19d ago
There are a couple of statistical errors in your argument here. First is frequency. Even using Pangram's claimed numbers, the University of Georgia should expect to see several false positives every week. Remember that the metric is # of assignments run through Pangram, not number of students. A campus of 40k students will see many more than 40k assignments every week, and so should expect honest students to be accused of cheating with some high degree of frequency. You're comparing infrequent events (death penalty sentences) to high-frequency events (students submitting assignments). And obviously, you are citing a company marketing document as fact, of which we should all be suspicious. (There are also obvious problems with the eval dataset that the paper does not address.) Second, you're using the upper bound for Pangram's claimed numbers and the lower bound cited in the NIH publication. > at least 4.1% would be exonerated. We conclude that this is a conservative estimate of the proportion of false conviction among death sentences in the United States. The one commonality is that

by droidjj 19d ago

emil-lp 19d ago

> The false positive rate for Pangram 4 is something like one in 24,000. First of all: says them.. Do you think they might have an incentive to boost their numbers? Second, this is on existing text, which might be in their training data, no?

lucb1e 19d ago

> The false positive rate for Pangram 4 is something like one in 24,000. Gotta suck to be one of the 8B/24k=~300k people in the world whose writing pattern is falsely labelled as slop by this tool that people say is so accurate so customers are going to feel really sure about your alleged dishonesty about writing your own texts This false positive rate is a double-edged sword. Please still be careful when accusing people

runako 19d ago

There are a couple of statistical errors in your argument here. First is frequency. Even using Pangram's claimed numbers, the University of Georgia should expect to see several false positives every week. Remember that the metric is # of assignments run through Pangram, not number of students. A campus of 40k students will see many more than 40k assignments every week, and so should expect honest students to be accused of cheating with some high degree of frequency. You're comparing infrequent events (death penalty sentences) to high-frequency events (students submitting assignments). And obviously, you are citing a company marketing document as fact, of which we should all be suspicious. (There are also obvious problems with the eval dataset that the paper does not address.) Second, you're using the upper bound for Pangram's claimed numbers and the lower bound cited in the NIH publication. > at least 4.1% would be exonerated. We conclude that this is a conservative estimate of the proportion of false conviction among death sentences in the United States. The one commonality is that