4 ms·
Is there a longer-form paper on this yet? TPR (P(T|AI)) and FPR (P(T|H)) are useful, but what I really want is the probability that a piece flagged as AI-genera
by claytonjy 4y ago
Is there a longer-form paper on this yet? TPR (P(T|AI)) and FPR (P(T|H)) are useful, but what I really want is the probability that a piece flagged as AI-generated is indeed AI-generated, i.e. P(AI|T). Per Bayes rule I'm missing P(AI), the portion of the challenger set that was produced by AI.
If we assume the challenger set is evenly split 50-50, that means
P(AI|T) = P(T|AI)P(AI)/P(T) = (0.26)(0.5)/(0.26+0.09) ~ 37%
So slightly better than a 1/3 chance of the flagged text actually being AI-generated.
They say the web-app uses a confidence threshold to keep the FPR low, so maybe these numbers get a bit better, but very far from being used as a detector anywhere it matters.
- TchoBeer 4y ago>Per Bayes rule I'm missing P(AI), the portion of the challenger set that was produced by AI This will obviously depend on your circumstances.
- drc500free 4y agoPrecision is impossible to calculate without knowing P(AI), which is use-case specific. Source: Spent 10 years trying to explain this to government people who insisted that someone tell them Precision based purely on the classifier accuracy without considering usage.