2 ms·

Can you show some examples of Pangram being "hot worthless garbage"? This reputation is true for most other detectors, but Pangram spends considerable effort to

Can you show some examples of Pangram being "hot worthless garbage"? This reputation is true for most other detectors, but Pangram spends considerable effort to have an extremely low false positive rate. I think you just used some of them in 2023 and now think that they can't be made better.

Twiin 2mo ago
I put your initial comment into Pangram and it said "We believe that this entire text is AI." So.
- pixl97 2mo ago
  The fact that anyone as any faith in an AI language detector at all blows my mind. It seems that every AI detector that gets productized gets as bad as the AI CEOs themselves and starts making claims their products cannot accomplish. They try to sell it as a black and white detection product while covering up the error bars are huge.
- Tiberium 2mo ago
  ... Come on now. This is obviously because I quoted the LLM written parts from the article. Remove the quoted parts and try again. Do people not understand what LLM detectors do? And, to be fair, short texts are one of the areas where even Pangram struggles, with a longer text it would be able to identify it as a mix of human and AI written text. They clearly disclose that short text detection is less reliable.
pixl97 2mo ago
Language is a living thing. At one time saying 'cool' to mean something was neat wasn't a thing. Then someone used it that way and it spread throughout humanity (or least English speakers in the US). This means as people use AI and read AI we'll start speaking like AI just as much as AI speaks like us. ESL people tend to have it even worse as they commonly speak a more proper subset of English and get identified as AI quite often. The Panagram CEO was saying some rather dumb shit recently that they have high detection rates of AI because they can put in documents from X years ago and they are not detected as AI. This tells you absolutely nothing about if a recently written document is human or AI written at all! Total snake oil.
- Tiberium 2mo ago
  Okay, I have a test for you. It's a bit undocumented, but I'm pretty sure HN is using Pangram or a similar tool to auto flag AI written comments. Can you copy the entire text from the linked article and reply with it here, and see what happens to your reply, would it get flagged or not? I think this will show you what reality we live in.
  - timcobb 2mo ago
    Pangram is just some BS man can you please stop
    - Tiberium 2mo ago
      Can you actually just do the test I explained? You seem to be basing your evaluation of Pangram on some old experiences. It's a very reliable service, used by research conferences and universities. It's baffling that people would rather call BS on me than try the actual tool being talked about.
      - pixl97 2mo ago
        It is not a reliable service. The CEO likes to say that it failed 1 time out of 10,000. In Panagrams own research the failure rate was shown at 2%. That's a pretty massive difference. How about you do this, write your own words on a topic, but use AI-isms in it. A few 'not X but Y' an a few EM dashes and watch it say you're text is AI written. Writing using AI stylisms has nothing to do if a text is actually written by AI or not. The tests are mostly bullshit because just adding some informalisms in the text corpus tends to get it counted as non-AI. It's not detecting what you think it is.
      - timcobb 2mo ago
        Who cares about Pangram? Why do you care about Pangram? Why not leverage our own intelligence? How does this post read to you? To me, it does not read like it was written by AI.

Twiin 2mo ago

I put your initial comment into Pangram and it said "We believe that this entire text is AI." So.

pixl97 2mo ago
The fact that anyone as any faith in an AI language detector at all blows my mind. It seems that every AI detector that gets productized gets as bad as the AI CEOs themselves and starts making claims their products cannot accomplish. They try to sell it as a black and white detection product while covering up the error bars are huge.
Tiberium 2mo ago
... Come on now. This is obviously because I quoted the LLM written parts from the article. Remove the quoted parts and try again. Do people not understand what LLM detectors do? And, to be fair, short texts are one of the areas where even Pangram struggles, with a longer text it would be able to identify it as a mix of human and AI written text. They clearly disclose that short text detection is less reliable.

pixl97 2mo ago

Language is a living thing. At one time saying 'cool' to mean something was neat wasn't a thing. Then someone used it that way and it spread throughout humanity (or least English speakers in the US). This means as people use AI and read AI we'll start speaking like AI just as much as AI speaks like us. ESL people tend to have it even worse as they commonly speak a more proper subset of English and get identified as AI quite often. The Panagram CEO was saying some rather dumb shit recently that they have high detection rates of AI because they can put in documents from X years ago and they are not detected as AI. This tells you absolutely nothing about if a recently written document is human or AI written at all! Total snake oil.

Tiberium 2mo ago
Okay, I have a test for you. It's a bit undocumented, but I'm pretty sure HN is using Pangram or a similar tool to auto flag AI written comments. Can you copy the entire text from the linked article and reply with it here, and see what happens to your reply, would it get flagged or not? I think this will show you what reality we live in.
- timcobb 2mo ago
  Pangram is just some BS man can you please stop
  - Tiberium 2mo ago
    Can you actually just do the test I explained? You seem to be basing your evaluation of Pangram on some old experiences. It's a very reliable service, used by research conferences and universities. It's baffling that people would rather call BS on me than try the actual tool being talked about.
    - pixl97 2mo ago
      It is not a reliable service. The CEO likes to say that it failed 1 time out of 10,000. In Panagrams own research the failure rate was shown at 2%. That's a pretty massive difference. How about you do this, write your own words on a topic, but use AI-isms in it. A few 'not X but Y' an a few EM dashes and watch it say you're text is AI written. Writing using AI stylisms has nothing to do if a text is actually written by AI or not. The tests are mostly bullshit because just adding some informalisms in the text corpus tends to get it counted as non-AI. It's not detecting what you think it is.
    - timcobb 2mo ago
      Who cares about Pangram? Why do you care about Pangram? Why not leverage our own intelligence? How does this post read to you? To me, it does not read like it was written by AI.