2 ms·
No it isn't. I've put human written material in there (my own unpublished material) and it says 90%+ AI. Then I put some AI material in and it said 30% chance
by jedberg 25d ago
No it isn't. I've put human written material in there (my own unpublished material) and it says 90%+ AI. Then I put some AI material in and it said 30% chance of AI.
I only tested it with the two pieces, but was not impressed.
- ekelsen 25d agoCan you share your human piece that got flagged? And the AI piece that didn't? (Or snippets?) I've found pangram to be very accurate, so I'm curious.
- jedberg 25d agoI just posted it above: https://news.ycombinator.com/item?id=49524889 https://news.ycombinator.com/item?id=49524889 Edit: It was removed. Here is a gist: https://gist.github.com/jedberg/e24124e577c0da1c8445c22798eb796c https://gist.github.com/jedberg/e24124e577c0da1c8445c22798eb...
- matsemann 25d agoEverything has false positives or negatives. 2 examples doesn't tell anything about how good or bad it is.
- pixl97 25d agoBeing that Panagram lies about its effectiveness in their presentations while hiding it's actual capabilities and testing methods rather deeply I have little faith in it. (1 in 10000 wrong in presentations verses 2-4% wrong in testing). For example they have a corpus of older pre-llm text and use that as the example their current model doesn't misclassify human written text. It shouldn't take much thinking to realize why this is a fucking stupid benchmark. Every day humans use LLMs and read LLM content Panagram becomes more useless because it forces languages to have a stopping point sometime around 2020. If you adopt any LLMism or are one of those unlucky people that already talked like an LLM before LLMs then all your shit is getting marked even though it was created by the human mind and written by human hands.
- calmoo 25d agoI’ve never actually seen someone give an example of what they say pangram get wrong, so please go ahead and share.
- deleted 25d ago[deleted]
- jedberg 25d agoYou can read it here: https://gist.github.com/jedberg/e24124e577c0da1c8445c22798eb796c https://gist.github.com/jedberg/e24124e577c0da1c8445c22798eb... I posted here on HN but it got removed.
- calmoo 25d agoI'll take your word that it's human written, but reading that gist, it absolutely reads like Claudeslop / GPT slop, it doesn't surprise me in the slightest that Pangram flagged it as AI when it reads identically to LLM output - this feels like a very acceptable edge case to me (assuming you are telling the truth). Are you absolutely sure you wrote this by hand? If so it's kind of remarkable how close to an LLM you write like.
- jedberg 25d agoIt’s important to remember that LLMs were trained on well written human text. People who write well are going to sound like an LLM. Especially if it’s a marketing message for a website. I’ve been accused of being an LLM multiple times here on HN too. I know you have no way to know for sure other than trusting that I’m not using an LLM to write. But it’s pretty frustrating that people jump right to LLM accusations.
- calmoo 24d agoI’ll be honest, i’ve never read text that is human written that reads as close to an LLM as your sample sounds. I think discounting Pangram’s accuracy based on that sample isn’t very reasonable. Really nobody writes like that other than LLMs!