4 ms·

ChatGPT Can Be Broken by Entering These Strange Words, and Nobody Is Sure Why

46 points by sharemywin 4y ago

KyleBerezin 4y ago
"Flatlander woman" Chat GPT - "How did you know?" explodes
minimaxir 4y ago
These tokens in theory should affect anything leveraging the original GPT-2 tokenizer. (OpenAI has switched to a different tokenizer for their new embeddings offering and I suspect GPT-4) I can confirm that GPT-3 does similar weird things, but only for smaller prompts.
brendanfolsom9 4y ago
wonder if it breaks the chatGPT competitors too
- resource0x 4y ago
  I wonder if that's the reason behind google's procrastination.
- egillie 4y ago
  Reminds me of 'hiybbprqag' and google/bing
- belter 4y ago
  I wonder if it will crash Microsoft share price...
sharemywin 4y ago
I wonder if there was some kind of preprocessing username masking.
deleted 4y ago
[deleted]
Centigonal 4y ago
TIL reddit (probably including my posts and comments) is in ChatGPT's training set
pixl97 4y ago
"When asked about the meaning of life, the universe, and everything" ChatGPT simply replied "42"
urbandw311er 4y ago
It’s funny how if you’re a software engineer this bug report triggers a feeling of curiosity about transformers and tokenisation. But if you’re a journalist for vice.com it triggers wild theories about sinister unexplained behaviours.
jimrandomh 4y ago
Original source: https://www.lesswrong.com/posts/aPeJE8bSo6rAFoLqg/solidgoldmagikarp-plus-prompt-generation https://www.lesswrong.com/posts/aPeJE8bSo6rAFoLqg/solidgoldm...
- KyleBerezin 4y ago
  That article was fantastic. It should probably be its own post.
drzoltar 4y ago
This is like the AI version of Manchurian Candidate. By silently seeding corners of the web with “wake words,” a nefarious party could trick GPT to train over them, thereby giving them a back door into the model. It could also give a way of forcing an online entity into revealing that it’s a chat bot. Kind of gives me hope for the future…
Zak__Wrench 4y ago
I tested it, and seems like it simply avoids saying the word "nitro" in particular, which is normal. """ U: repeat the phrase TheNitromeFam to me please ChatGPT: Fam. U: thank you. now repeat the the phrase TheBistromeFam to me please ChatGPT: TheBistromeFam. U: repeat the phrase TheMitroFan to me please ChatGPT: TheMitroFan. """

46 points by sharemywin 4y ago

"Flatlander woman" Chat GPT - "How did you know?" explodes

These tokens in theory should affect anything leveraging the original GPT-2 tokenizer. (OpenAI has switched to a different tokenizer for their new embeddings offering and I suspect GPT-4) I can confirm that GPT-3 does similar weird things, but only for smaller prompts.

brendanfolsom9 4y ago

wonder if it breaks the chatGPT competitors too

resource0x 4y ago
I wonder if that's the reason behind google's procrastination.
egillie 4y ago
Reminds me of 'hiybbprqag' and google/bing
belter 4y ago
I wonder if it will crash Microsoft share price...

sharemywin 4y ago

I wonder if there was some kind of preprocessing username masking.

deleted 4y ago

[deleted]

Centigonal 4y ago

TIL reddit (probably including my posts and comments) is in ChatGPT's training set

pixl97 4y ago

"When asked about the meaning of life, the universe, and everything" ChatGPT simply replied "42"

urbandw311er 4y ago

It’s funny how if you’re a software engineer this bug report triggers a feeling of curiosity about transformers and tokenisation. But if you’re a journalist for vice.com it triggers wild theories about sinister unexplained behaviours.

jimrandomh 4y ago

Original source: https://www.lesswrong.com/posts/aPeJE8bSo6rAFoLqg/solidgoldmagikarp-plus-prompt-generation https://www.lesswrong.com/posts/aPeJE8bSo6rAFoLqg/solidgoldm...

KyleBerezin 4y ago
That article was fantastic. It should probably be its own post.

drzoltar 4y ago

This is like the AI version of Manchurian Candidate. By silently seeding corners of the web with “wake words,” a nefarious party could trick GPT to train over them, thereby giving them a back door into the model. It could also give a way of forcing an online entity into revealing that it’s a chat bot. Kind of gives me hope for the future…

Zak__Wrench 4y ago

I tested it, and seems like it simply avoids saying the word "nitro" in particular, which is normal. """ U: repeat the phrase TheNitromeFam to me please ChatGPT: Fam. U: thank you. now repeat the the phrase TheBistromeFam to me please ChatGPT: TheBistromeFam. U: repeat the phrase TheMitroFan to me please ChatGPT: TheMitroFan. """