While the general conclusion of this paper seems perfectly plausible, I'm not sure how much weight I can put into the exact numbers they come up with. They onl

While the general conclusion of this paper seems perfectly plausible, I'm not sure how much weight I can put into the exact numbers they come up with. They only gather 46 summaries from MTurks, and they train a predictor on some known-good and known-synthetic data, and use that to decide when the crowd workers have submitted LLM-generated text. I'm not totally sold on the idea that the good performance on the validation set is going to translate to this new dataset. In general, I'm just really skeptical of results that say "we can detect generated text with 99% accuracy!". It would be very interesting to see a study that asks crowd workers afterwards if they used an LLM to complete the task. Obviously you'd be pretty worried about lying, but maybe there's a way to structure the situation so that the workers know their response is anonymous and won't affect whether they get paid.