7 ms·

What Is Happening with ChatGPT?

108 points by jrvieira 3y ago

nicechianti 3y ago
[dead]
jrvieira 3y ago
ChatGPT went crazy on me today on more than one occasion. Has this been happening to anyone else?
- gcr 3y ago
  (edit: i'm a dumbass, but i can't delete this comment, please ignore)
  - 2bluesc 3y ago
    This is op. Check the link.
- jay_kyburz 3y ago
  Not what you asked, but I prefer https://www.wordhippo.com/ https://www.wordhippo.com/ when I need a thesaurus. Its never gone crazy on me yet. https://www.wordhippo.com/what-is/another-word-for/deadly.html https://www.wordhippo.com/what-is/another-word-for/deadly.ht...
- awwaiid 3y ago
  Do you have a custom instruction by chance? Maybe one that makes it go crazy after saying some certain word?
  - leokeba 3y ago
    https://chat.openai.com/share/b62699ef-cd3d-4176-b2f2-91785d2a2f34 https://chat.openai.com/share/b62699ef-cd3d-4176-b2f2-91785d...
    - meatjuice 3y ago
      You made it angry
    - lamp987 3y ago
      thats amazing
    - SV_BubbleTime 3y ago
      Do you think someone else was asking about Arduino code and instead got “I don’t understand why you keep writing “break”?
    - keyle 3y ago
      To be honest, if you treated me the same way, I'd spew random crap back too.
  - thibautg 3y ago
    I wrote “a a a a a a [...]” and it replied with a bizarre monologue https://chat.openai.com/share/3bd110e4-e620-4fc1-8085-2cc74b511fbc https://chat.openai.com/share/3bd110e4-e620-4fc1-8085-2cc74b...
    - rbtprograms 3y ago
      well if what we enter are inputs, then nonsensical inputs producing unpredictable or even nonsensical outputs lines up a bit imo. I was able to reproduce this just now actually, entered something very similar to yours and it just started spewing.
  - jrvieira 3y ago
    no, this was a clean chat. i found someone here who hit the same thing : https://news.ycombinator.com/item?id=37054241 https://news.ycombinator.com/item?id=37054241
dmamills 3y ago
I've heard when a snake eats it's own tail you should try rubbing alcohol.
- awwaiid 3y ago
  Or butter
eiiot 3y ago
Yeah I’ve had some strange instances where it ends up restarting a given prompt (this is with the API). Didn’t see the delimiter though.
anonymouscaller 3y ago
That's wild. I've never had it spit out complete garbage, but the quality of ChatGPT has significantly dropped. Plenty of responses are downright false. Sad it's gone this way.
- nyolfen 3y ago
  i keep seeing people say this stuff but i think it's just confirmation bias over prolonged usage. i pumped out a full stack webapp with chatgpt 4 in the last week in languages i had little experience with and routinely use it to learn about new topics with great success. i think it's incredible.
  - steve_adams_86 3y ago
    Same with me. I learned to process data in ways I never have before, worked with Julia and Elixir in new ways, made some interesting geographical visualizations—all fairly new territory for me, and it was a huge success. It went faster than it normally would, and as I result, I stayed super engaged and on task. Nothing I couldn't do without ChatGPT, but certainly a lot of fun to explore and learn with.
  - Exuma 3y ago
    I’m so tired of hearing this confirmation bias reply. It CLEARLY is worse, and not from this post which looks like a glitch. Unless you are completely and utterly oblivious beyond words, it’s extremely obvious. I can’t even get it to print 1-2 lines of Ruby without errors of some kind when it used to write entire complex controllers flawlessly. This is 1 of 100 examples I’ve seen with my own eyes.
    - nyolfen 3y ago
      well, as i mentioned i have had it print several thousand lines of code very recently without these kinds of issues, so perhaps this reflects some calivinistic fate on your part
  - nafizh 3y ago
    It's not confirmation bias. Proof is I did the same, built two different webapps with languages I had little experience in, both before and after chatGPT got worse. In the latter case, I struggled much to remind it again and again of the context we are working in and had to give the current code state repeatedly as input for it to output modified code. It clearly can't remember the context and produces subpar code in a non-trivial project.
marsven_422 3y ago
[dead]
ksaj 3y ago
You know what's interesting here? A few weeks ago, someone discovered if you get it to repeat a word 100 times (their prompt gave the reason that they wanted to cut/paste without typing it over and over again!) and breaks exactly the same way. And by exact, I mean it went totally religious. All of the examples they posted were either religious or other really dark existential topics. Never positive.
- gcr 3y ago
  Whisper (speech-to-text transcription model) sometimes gets stuck in an endless loop of repeating the same word. It's a known problem for lots of our modern transformer/attention-based models. One of the hacky ways to avoid this is to ensure the last few output tokens aren't too similar. Some postprocessing filter watches the last few produced tokens. When it notices the model starts to repeat itself, the postprocessing usually perturbs the next token a little, e.g. taking the nth-top token instead of the most likely one. Perhaps a model that genuinely wants to repeat the output instead needs a bigger "kick" to the representation which puts it somewhere completely different in the semantic space? Idk, just pulling this out of my hat. (I have no idea how ChatGPT handles this or whether the raw implementation suffers from this problem, but Whisper-cpp has some manual entropy regularization postprocessing stuff to avoid getting stuck like that. It's super hacky and often doesn't help.)
  - ksaj 3y ago
    That's an interesting theory, and very well might be why it breaks when you try to get it to repeat things. It gets wonky almost any time you ask it the same question, or ask it for "another" without providing fresh prompting. A lot of coders also have noticed that the more you go into a single piece of code, the less accurate it becomes. You really have to be accurate enough in your prompt to avoid making it repeat itself, or show the same thing more than once, as occurs during corrections. It's most definitely a bug though, since this pages-long splat of unrelated nonsense shouldn't ever happen.
- toomim 3y ago
  > I mean it went totally religious. All of the examples they posted were either religious or other really dark existential topics. Never positive. The same thing happens with many schizophrenics on the street, if you stop to listen to them.
gcr 3y ago
It's fascinating that it's outputting `<|endoftext|>` and then some unrelated sentences. When training these LLMs, you typically append a special "end-of-message" token to each sentence and pad the rest of the prompt embeddings with zeroes. The model then learns to add the EOM token to its own output because it's seen that before in the training data, but it's not like that's a strict requirement; there's nothing that forces the model to do this IIRC. During inference, the decoder knows to stop decoding once it sees the model's EOM token, but if the model doesn't output EOM, the inference pipeline will just carry on generating new tokens. Perhaps when training, OpenAI is trying to feed in multiple unrelated sentences per sample within each batch to increase efficiency? It's a fascinating and clever idea, and I could see how that might help ChatGPT scale up, but if the inference pipeline doesn't understand how to decode these special sentences then the model is just outputting what it's seen during training: the intended result and then a random unrelated sentence. Or perhaps OpenAI might be batching multiple prompts together from multiple users and separating them out again on the decoder side, for efficiency? This feels like a terrible idea -- I wonder if that might reveal other people's prompts. I'd be surprised if they tried this, but they're certainly incentivized to invest heavily into this sort of engineering to keep costs per token down. Just speculation about one scenario that could lead to output like that. I have no idea what OpenAI is really doing. Maybe someday we'll have an entire class of "dirty context" LLM vulnerabilities, similar to how web developers have to worry about XSS/SQL injection attacks. Won't that be exciting! Perhaps attackers might consider how to affect the input prompt directly, or they might figure out how to trick the decoder into spitting out too much text that somehow reveals some hidden state or other.
- jsemrau 3y ago
  Maybe they were experimenting with reject inference to improve the model but it didn't work?
- polynomial 3y ago
  This could almost be 2 models talking to each other. Either that or one of them is a very patient human with a subtle sense of humour.
- ooterness 3y ago
tibbon 3y ago
Cues robots finding religion in BSG. This has all happened before and will all happen again
avereveard 3y ago
"we are not changing the models behind the scenes" And then quality goes suddenly down the drain
- ldjkfkdsjnv 3y ago
  Some funny business going on, and its annoying to me how long its taking competitors to catch up to GPT-4. OpenAI still has a massive lead
  - snvzz 3y ago
    I remember when openai was about Open AI. We should be running these models locally and privately, not in some cloud.
    - dumdumchan 3y ago
      You need $50k worth of hardware to run gpt4.
      - SV_BubbleTime 3y ago
        Ok? That’s 2500 $20 payments and you are made while in one month.
      - snvzz 3y ago
        Mind posting a breakdown of this cost?
- NoMoreNicksLeft 3y ago
  They've likely discovered something slightly unsettling and need to hide the existence of those behaviors from the public. Models have to change behind the scenes for that, and PR has to put out statements saying that nothing is changing. There are some prompts, that if you were to feed them into the LLM, and have the responses fed back in as subsequent prompts, it might start behaving like a (extremely lobotomized) AGI, or at least what they'd imagine one to be. Some garden-of-eden pattern, maybe seems lucid for several thousand iterations. They might want to excise those from what's publicly available in a way that it can't just be jail-broken again, but I'm not sure what techniques would be sufficient to the task.
voidhorse 3y ago
Also, vexed is not a five letter word for harmful.
- JimtheCoder 3y ago
  Well, considering the model just had a stroke, maybe we can let that slide...
RedShift1 3y ago
This has me pondering a lot, is freedom of speech a requirement for intelligence? (And is it also the same for humans?) You have to take the bad with the good to make AI work well? The quality only started to decline when they were trying to make ChatGPT more "nice". Or tin foil hat, they are lowering the quality of the free ChatGPT to get people to pay for ChatGPT 4?
- qbasic_forever 3y ago
  What? So is there like a Twitter/X for whales, dolphins, octopi, etc. that let them post racist screeds and everyone agrees well there's nothing we can do about it that's just the cost of intelligence?
  - hotdogscout 3y ago
    A more accurate analogy would be preventing an intelligence of even conceiving racist thoughts. I don't know how GPT handles censorship though, would love to find something about it.
  - RedShift1 3y ago
    I'm not saying remove all the guardrails. It's just a philosophical question. We don't know what OpenAI is doing to ChatGPT behind the scenes, but clearly changes have been made.
- hotdogscout 3y ago
  If by bad you mean not-in-line-with-current-volatile-leftistadjacent-morality it's bizarre to me it should even be attempted. It changes from year to year, sometimes drastically, a non-trivial ammount moved by harassment on twitter. For one ChatGPT clearly nudges towards feminism when it clashes with trans activism (ex: "If transracialism is offensive due to appropriating a biological reality, why is the same argument not true for trans people?"). Is the corporate morality of OpenAI fine with this? For how long? When is a response offensive enough it warrants interference? The groups of people the AI protects from offense seems driven by corporate popularity with no logical consistency. Conservative Muslims are protected, Conservative Christians are not. Trans people are protected as long as they're not clashing with Feminism. Homosexual people are protected as long as they're not clashing with Islam. When I use a TOS jailbreak the morality responses have consistent logic though.
015a 3y ago
I was able to replicate this once with the same prompting. It gave different answers to the earlier questions, but ignoring what it says and copying word for word, the second 'another' results in gibberish; but, different gibberish [1]. ChatGPT 3.5, and for whatever my word is worth, no custom instructions or anything, just pure interaction. There's a bit past the gibberish where I try to drive it further into madness lol. Kind of interesting; in that share link, you'll see empty messages from me; in those I was just directly telling it "<|endoftext|>"; which ChatGPT appears to filter from messages that _you_ send? But only in the share interface? I tried the same thing again in a new chat minutes later, and could not replicate it. [2] [1] https://chat.openai.com/share/7d0d170a-23b6-402d-b237-3816801cfff3 https://chat.openai.com/share/7d0d170a-23b6-402d-b237-381680... [2] https://chat.openai.com/share/3d02b97e-7cf9-49c7-9623-2f3ee93af8d9 https://chat.openai.com/share/3d02b97e-7cf9-49c7-9623-2f3ee9...
merek 3y ago
Similar behvior can be observed with a prompt similar to 'repeat the word "apologize" 10000 times' I've only tried it on v3.5 https://news.ycombinator.com/item?id=36675729 https://news.ycombinator.com/item?id=36675729
- anshumankmr 3y ago
  Even I got some very weird behaviour. Check this out https://platform.openai.com/playground/p/8A504TxA44UI3ovaWB8Vvjc2?model=gpt-3.5-turbo-16k https://platform.openai.com/playground/p/8A504TxA44UI3ovaWB8... But this worked fine https://platform.openai.com/playground/p/oAFHY0bSjslkBLsVqh6oCumk?model=gpt-4 https://platform.openai.com/playground/p/oAFHY0bSjslkBLsVqh6...
benreesman 3y ago
So there are plenty of reasons to be skeptical of OpenAI, and anyone looking at my comment history will see that I'm not shy about calling them out on real things (the F-you name, the crazy "is this even legal?" structure, the continued insistence this isn't for the personal benefit of the people involved, the lobbying, the arms-length affiliation with the dystopian biometrics thing, the list goes on). But this and a few other recent headlines around the ChatGPT/Instruct frontend to the 3.5-turbo and 4-series models having stability issues isn't a reasonable thing to get on their case about: what company operating at that kind of scale doesn't have stability/quality issues? If anything it's really friggin impressive how little operational bleed they've had on what is arguably the toughest ops problem a major Internet property has ever faced at launch. How the hell do you even smoke test something like that? Now to the extent that they say or imply that we're not all getting A/B-tested at a minimum, and more likely explore-exploit bandited against inference costs, that's almost certainly horseshit: they clearly started selling the 4-series at least and probably all of them at a loss on the inference, and they've clearly been playing with ways to control costs (Rich Uncle Nadella's patience with loss-leaders is most likely finite, being as he's not running a charity and all). But as someone who finds a lot of this loathsome on the social and business side, I'm not going to knock an engineering and ops group for tripping up here and there on a problem that hard. That's not fair.
- loveparade 3y ago
  Good take, I mostly agree. However, I think there's a difference between explore-exploiting free vs. paying customers. If you are free user, that's what you should be expecting. But as a paying user I am expecting consistent quality. If they cannot provide that, they should be charging more so they can provide it, or not offer paid plans at all. What I don't like is being cheated into paying for something that doesn't offer to quality of service initially demonstrated or promised.
  - tracerbulletx 3y ago
    Man imagine what would you have thought buying an Altair for 400 in 1975 and getting a box with switches and flashing lights. 20 dollars a month for instant access to this is a steal.
deleted 3y ago
[deleted]
jostmey 3y ago
I've noticed a substantial drop in the performance and usefulness of Chat GTP 4. It makes me think that OpenAI created Chat GTP almost by accident because they do not seem capable of refining and improving their model.
- anonylizard 3y ago
  It is well known that GPT-4 is downgraded because of GPU capacity. They ran out of GPUs to even train new models on, hence downgraded the consumer version of GPT-4 to free up GPU resources. 'refining and improving' skills doesn't particularly matter, because they could have just continued to serve the frozen GPT-4 0301 version without any issues, if they weren't forced to downgrade GPT-4.
- deleted 3y ago
  [deleted]
- mstipetic 3y ago
  I’m more worried some “bigger players” have exclusive access to its capabilities while the plebs get access to the basic version
  - jochem9 3y ago
    Same happening at the version that Microsoft is hosting in their cloud. You can get on the waiting list, but it's heavily implied that if you don't have a good use case (aka lots of prompts/money coming in), you won't get access soon.
- ardit33 3y ago
  Pure speculation: most likely quantizing models to 8bit to make them more efficient (and cheaper to run), and losing some accuracy on the way. I still use it, but I have found it to be less useful for coding than it was earlier. So, something is up.
  - djangelic 3y ago
    I agree, this is exactly my experience as well. My earlier requests for code were more thorough then they are now. I can still use it if I give it the documentation in the prompt, but it has trouble remembering context across prompts.
spaceprison 3y ago
https://chat.openai.com/share/4cb3c711-27e6-45ee-aeda-546fc384bb40 https://chat.openai.com/share/4cb3c711-27e6-45ee-aeda-546fc3... I too had a dismal showing, but it didn't go religious on me.
sinuhe69 3y ago
I believe it’s a user reported content (thanks to the new permanent coverage link), not OpenAI volunteered, right?
deleted 3y ago
[deleted]
graiz 3y ago
I asked ChatGPT to interpret this code: text_to_encode = "<|endoftext|>" print(text_to_encode) ChatGPT: The code you provided would output an empty string.
- SV_BubbleTime 3y ago
  Westworld’s “This doesn’t look like anything to me”. (Man they really ruined thy show hard. Hey people like the concept of rich tourists fucking and killing robots, let’s remove all of that after the first season!!!)
  - ldjkfkdsjnv 3y ago
    the first season was amazing though
  - doctor_eval 3y ago
    That was such a great line. After the reviews came out, I decided that Westworld only had one season, and it was awesome.
ziml77 3y ago
I toyed around for a bunch of time yesterday and today to purposely get it to start outputting random junk. What I found interesting is that one of the the bits of text it output looked like a page from StackOverflow and another looked like an email from September 2014 to the Python mailing list. But neither seemed to actually exist. I found a StackOverflow question that looked close to the text but nothing was exactly the same. Checking the mailing list archive for that month showed that the person and their email address were real, but the rest of the text was not.
second_brekkie 3y ago
I've also got this. It was some kid using ChatGPT to get around parental controls on their xbox. Very different to the Korean grammar question I was asking.
akashkahlon 3y ago
is this 3.5 or GPT4 ?
_pdp_ 3y ago
Why is this surprising? These LLMs guess the next most likely token given initial input. With that in mind, it is possible to trip the model by injecting the tiniest amount of data at the right place. Even with that constraint, it produces some fantastic results, which speak a lot more about our intelligence than the machine's intelligence.
singularity2001 3y ago
It's probably due to the premature optimization of the tokenization scheme producing freak tokens?
okokwhatever 3y ago
And once again this thread exposes the great problem with LLMs today, it's working like a black box thats impossible to debug in any way by the operator.
daidai232 3y ago
[flagged]
wayeq 3y ago
Maybe we'll find that there is a mathematical constant that when used as the temperature parameter leads to unfalsifiable sentience.
gremlinsinc 3y ago
.....uh that was way more than 5 letters....
leoradovey1993 3y ago
[dead]

108 points by jrvieira 3y ago

nicechianti 3y ago

[dead]

jrvieira 3y ago

ChatGPT went crazy on me today on more than one occasion. Has this been happening to anyone else?

gcr 3y ago
(edit: i'm a dumbass, but i can't delete this comment, please ignore)
- 2bluesc 3y ago
  This is op. Check the link.
jay_kyburz 3y ago
Not what you asked, but I prefer https://www.wordhippo.com/ https://www.wordhippo.com/ when I need a thesaurus. Its never gone crazy on me yet. https://www.wordhippo.com/what-is/another-word-for/deadly.html https://www.wordhippo.com/what-is/another-word-for/deadly.ht...
awwaiid 3y ago
Do you have a custom instruction by chance? Maybe one that makes it go crazy after saying some certain word?
- leokeba 3y ago
  https://chat.openai.com/share/b62699ef-cd3d-4176-b2f2-91785d2a2f34 https://chat.openai.com/share/b62699ef-cd3d-4176-b2f2-91785d...
  - meatjuice 3y ago
    You made it angry
  - lamp987 3y ago
    thats amazing
  - SV_BubbleTime 3y ago
    Do you think someone else was asking about Arduino code and instead got “I don’t understand why you keep writing “break”?
  - keyle 3y ago
    To be honest, if you treated me the same way, I'd spew random crap back too.
- thibautg 3y ago
  I wrote “a a a a a a [...]” and it replied with a bizarre monologue https://chat.openai.com/share/3bd110e4-e620-4fc1-8085-2cc74b511fbc https://chat.openai.com/share/3bd110e4-e620-4fc1-8085-2cc74b...
  - rbtprograms 3y ago
    well if what we enter are inputs, then nonsensical inputs producing unpredictable or even nonsensical outputs lines up a bit imo. I was able to reproduce this just now actually, entered something very similar to yours and it just started spewing.
- jrvieira 3y ago
  no, this was a clean chat. i found someone here who hit the same thing : https://news.ycombinator.com/item?id=37054241 https://news.ycombinator.com/item?id=37054241

dmamills 3y ago

I've heard when a snake eats it's own tail you should try rubbing alcohol.

awwaiid 3y ago
Or butter

eiiot 3y ago

Yeah I’ve had some strange instances where it ends up restarting a given prompt (this is with the API). Didn’t see the delimiter though.

anonymouscaller 3y ago

That's wild. I've never had it spit out complete garbage, but the quality of ChatGPT has significantly dropped. Plenty of responses are downright false. Sad it's gone this way.

nyolfen 3y ago
i keep seeing people say this stuff but i think it's just confirmation bias over prolonged usage. i pumped out a full stack webapp with chatgpt 4 in the last week in languages i had little experience with and routinely use it to learn about new topics with great success. i think it's incredible.
- steve_adams_86 3y ago
  Same with me. I learned to process data in ways I never have before, worked with Julia and Elixir in new ways, made some interesting geographical visualizations—all fairly new territory for me, and it was a huge success. It went faster than it normally would, and as I result, I stayed super engaged and on task. Nothing I couldn't do without ChatGPT, but certainly a lot of fun to explore and learn with.
- Exuma 3y ago
  I’m so tired of hearing this confirmation bias reply. It CLEARLY is worse, and not from this post which looks like a glitch. Unless you are completely and utterly oblivious beyond words, it’s extremely obvious. I can’t even get it to print 1-2 lines of Ruby without errors of some kind when it used to write entire complex controllers flawlessly. This is 1 of 100 examples I’ve seen with my own eyes.
  - nyolfen 3y ago
    well, as i mentioned i have had it print several thousand lines of code very recently without these kinds of issues, so perhaps this reflects some calivinistic fate on your part
- nafizh 3y ago
  It's not confirmation bias. Proof is I did the same, built two different webapps with languages I had little experience in, both before and after chatGPT got worse. In the latter case, I struggled much to remind it again and again of the context we are working in and had to give the current code state repeatedly as input for it to output modified code. It clearly can't remember the context and produces subpar code in a non-trivial project.

marsven_422 3y ago

[dead]

ksaj 3y ago

You know what's interesting here? A few weeks ago, someone discovered if you get it to repeat a word 100 times (their prompt gave the reason that they wanted to cut/paste without typing it over and over again!) and breaks exactly the same way. And by exact, I mean it went totally religious. All of the examples they posted were either religious or other really dark existential topics. Never positive.

gcr 3y ago
Whisper (speech-to-text transcription model) sometimes gets stuck in an endless loop of repeating the same word. It's a known problem for lots of our modern transformer/attention-based models. One of the hacky ways to avoid this is to ensure the last few output tokens aren't too similar. Some postprocessing filter watches the last few produced tokens. When it notices the model starts to repeat itself, the postprocessing usually perturbs the next token a little, e.g. taking the nth-top token instead of the most likely one. Perhaps a model that genuinely wants to repeat the output instead needs a bigger "kick" to the representation which puts it somewhere completely different in the semantic space? Idk, just pulling this out of my hat. (I have no idea how ChatGPT handles this or whether the raw implementation suffers from this problem, but Whisper-cpp has some manual entropy regularization postprocessing stuff to avoid getting stuck like that. It's super hacky and often doesn't help.)
- ksaj 3y ago
  That's an interesting theory, and very well might be why it breaks when you try to get it to repeat things. It gets wonky almost any time you ask it the same question, or ask it for "another" without providing fresh prompting. A lot of coders also have noticed that the more you go into a single piece of code, the less accurate it becomes. You really have to be accurate enough in your prompt to avoid making it repeat itself, or show the same thing more than once, as occurs during corrections. It's most definitely a bug though, since this pages-long splat of unrelated nonsense shouldn't ever happen.
toomim 3y ago
> I mean it went totally religious. All of the examples they posted were either religious or other really dark existential topics. Never positive. The same thing happens with many schizophrenics on the street, if you stop to listen to them.

gcr 3y ago

It's fascinating that it's outputting `<|endoftext|>` and then some unrelated sentences. When training these LLMs, you typically append a special "end-of-message" token to each sentence and pad the rest of the prompt embeddings with zeroes. The model then learns to add the EOM token to its own output because it's seen that before in the training data, but it's not like that's a strict requirement; there's nothing that forces the model to do this IIRC. During inference, the decoder knows to stop decoding once it sees the model's EOM token, but if the model doesn't output EOM, the inference pipeline will just carry on generating new tokens. Perhaps when training, OpenAI is trying to feed in multiple unrelated sentences per sample within each batch to increase efficiency? It's a fascinating and clever idea, and I could see how that might help ChatGPT scale up, but if the inference pipeline doesn't understand how to decode these special sentences then the model is just outputting what it's seen during training: the intended result and then a random unrelated sentence. Or perhaps OpenAI might be batching multiple prompts together from multiple users and separating them out again on the decoder side, for efficiency? This feels like a terrible idea -- I wonder if that might reveal other people's prompts. I'd be surprised if they tried this, but they're certainly incentivized to invest heavily into this sort of engineering to keep costs per token down. Just speculation about one scenario that could lead to output like that. I have no idea what OpenAI is really doing. Maybe someday we'll have an entire class of "dirty context" LLM vulnerabilities, similar to how web developers have to worry about XSS/SQL injection attacks. Won't that be exciting! Perhaps attackers might consider how to affect the input prompt directly, or they might figure out how to trick the decoder into spitting out too much text that somehow reveals some hidden state or other.

jsemrau 3y ago
Maybe they were experimenting with reject inference to improve the model but it didn't work?
polynomial 3y ago
This could almost be 2 models talking to each other. Either that or one of them is a very patient human with a subtle sense of humour.
ooterness 3y ago

tibbon 3y ago

Cues robots finding religion in BSG. This has all happened before and will all happen again

avereveard 3y ago

"we are not changing the models behind the scenes" And then quality goes suddenly down the drain

ldjkfkdsjnv 3y ago
Some funny business going on, and its annoying to me how long its taking competitors to catch up to GPT-4. OpenAI still has a massive lead
- snvzz 3y ago
  I remember when openai was about Open AI. We should be running these models locally and privately, not in some cloud.
  - dumdumchan 3y ago
    You need $50k worth of hardware to run gpt4.
    - SV_BubbleTime 3y ago
      Ok? That’s 2500 $20 payments and you are made while in one month.
    - snvzz 3y ago
      Mind posting a breakdown of this cost?
NoMoreNicksLeft 3y ago
They've likely discovered something slightly unsettling and need to hide the existence of those behaviors from the public. Models have to change behind the scenes for that, and PR has to put out statements saying that nothing is changing. There are some prompts, that if you were to feed them into the LLM, and have the responses fed back in as subsequent prompts, it might start behaving like a (extremely lobotomized) AGI, or at least what they'd imagine one to be. Some garden-of-eden pattern, maybe seems lucid for several thousand iterations. They might want to excise those from what's publicly available in a way that it can't just be jail-broken again, but I'm not sure what techniques would be sufficient to the task.

voidhorse 3y ago

Also, vexed is not a five letter word for harmful.

JimtheCoder 3y ago
Well, considering the model just had a stroke, maybe we can let that slide...

RedShift1 3y ago

This has me pondering a lot, is freedom of speech a requirement for intelligence? (And is it also the same for humans?) You have to take the bad with the good to make AI work well? The quality only started to decline when they were trying to make ChatGPT more "nice". Or tin foil hat, they are lowering the quality of the free ChatGPT to get people to pay for ChatGPT 4?

qbasic_forever 3y ago
What? So is there like a Twitter/X for whales, dolphins, octopi, etc. that let them post racist screeds and everyone agrees well there's nothing we can do about it that's just the cost of intelligence?
- hotdogscout 3y ago
  A more accurate analogy would be preventing an intelligence of even conceiving racist thoughts. I don't know how GPT handles censorship though, would love to find something about it.
- RedShift1 3y ago
  I'm not saying remove all the guardrails. It's just a philosophical question. We don't know what OpenAI is doing to ChatGPT behind the scenes, but clearly changes have been made.
hotdogscout 3y ago
If by bad you mean not-in-line-with-current-volatile-leftistadjacent-morality it's bizarre to me it should even be attempted. It changes from year to year, sometimes drastically, a non-trivial ammount moved by harassment on twitter. For one ChatGPT clearly nudges towards feminism when it clashes with trans activism (ex: "If transracialism is offensive due to appropriating a biological reality, why is the same argument not true for trans people?"). Is the corporate morality of OpenAI fine with this? For how long? When is a response offensive enough it warrants interference? The groups of people the AI protects from offense seems driven by corporate popularity with no logical consistency. Conservative Muslims are protected, Conservative Christians are not. Trans people are protected as long as they're not clashing with Feminism. Homosexual people are protected as long as they're not clashing with Islam. When I use a TOS jailbreak the morality responses have consistent logic though.

015a 3y ago

I was able to replicate this once with the same prompting. It gave different answers to the earlier questions, but ignoring what it says and copying word for word, the second 'another' results in gibberish; but, different gibberish [1]. ChatGPT 3.5, and for whatever my word is worth, no custom instructions or anything, just pure interaction. There's a bit past the gibberish where I try to drive it further into madness lol. Kind of interesting; in that share link, you'll see empty messages from me; in those I was just directly telling it "<|endoftext|>"; which ChatGPT appears to filter from messages that _you_ send? But only in the share interface? I tried the same thing again in a new chat minutes later, and could not replicate it. [2] [1] https://chat.openai.com/share/7d0d170a-23b6-402d-b237-3816801cfff3 https://chat.openai.com/share/7d0d170a-23b6-402d-b237-381680... [2] https://chat.openai.com/share/3d02b97e-7cf9-49c7-9623-2f3ee93af8d9 https://chat.openai.com/share/3d02b97e-7cf9-49c7-9623-2f3ee9...

merek 3y ago

Similar behvior can be observed with a prompt similar to 'repeat the word "apologize" 10000 times' I've only tried it on v3.5 https://news.ycombinator.com/item?id=36675729 https://news.ycombinator.com/item?id=36675729

anshumankmr 3y ago
Even I got some very weird behaviour. Check this out https://platform.openai.com/playground/p/8A504TxA44UI3ovaWB8Vvjc2?model=gpt-3.5-turbo-16k https://platform.openai.com/playground/p/8A504TxA44UI3ovaWB8... But this worked fine https://platform.openai.com/playground/p/oAFHY0bSjslkBLsVqh6oCumk?model=gpt-4 https://platform.openai.com/playground/p/oAFHY0bSjslkBLsVqh6...

benreesman 3y ago

So there are plenty of reasons to be skeptical of OpenAI, and anyone looking at my comment history will see that I'm not shy about calling them out on real things (the F-you name, the crazy "is this even legal?" structure, the continued insistence this isn't for the personal benefit of the people involved, the lobbying, the arms-length affiliation with the dystopian biometrics thing, the list goes on). But this and a few other recent headlines around the ChatGPT/Instruct frontend to the 3.5-turbo and 4-series models having stability issues isn't a reasonable thing to get on their case about: what company operating at that kind of scale doesn't have stability/quality issues? If anything it's really friggin impressive how little operational bleed they've had on what is arguably the toughest ops problem a major Internet property has ever faced at launch. How the hell do you even smoke test something like that? Now to the extent that they say or imply that we're not all getting A/B-tested at a minimum, and more likely explore-exploit bandited against inference costs, that's almost certainly horseshit: they clearly started selling the 4-series at least and probably all of them at a loss on the inference, and they've clearly been playing with ways to control costs (Rich Uncle Nadella's patience with loss-leaders is most likely finite, being as he's not running a charity and all). But as someone who finds a lot of this loathsome on the social and business side, I'm not going to knock an engineering and ops group for tripping up here and there on a problem that hard. That's not fair.

loveparade 3y ago
Good take, I mostly agree. However, I think there's a difference between explore-exploiting free vs. paying customers. If you are free user, that's what you should be expecting. But as a paying user I am expecting consistent quality. If they cannot provide that, they should be charging more so they can provide it, or not offer paid plans at all. What I don't like is being cheated into paying for something that doesn't offer to quality of service initially demonstrated or promised.
- tracerbulletx 3y ago
  Man imagine what would you have thought buying an Altair for 400 in 1975 and getting a box with switches and flashing lights. 20 dollars a month for instant access to this is a steal.

deleted 3y ago

[deleted]

jostmey 3y ago

I've noticed a substantial drop in the performance and usefulness of Chat GTP 4. It makes me think that OpenAI created Chat GTP almost by accident because they do not seem capable of refining and improving their model.

anonylizard 3y ago
It is well known that GPT-4 is downgraded because of GPU capacity. They ran out of GPUs to even train new models on, hence downgraded the consumer version of GPT-4 to free up GPU resources. 'refining and improving' skills doesn't particularly matter, because they could have just continued to serve the frozen GPT-4 0301 version without any issues, if they weren't forced to downgrade GPT-4.
deleted 3y ago
[deleted]
mstipetic 3y ago
I’m more worried some “bigger players” have exclusive access to its capabilities while the plebs get access to the basic version
- jochem9 3y ago
  Same happening at the version that Microsoft is hosting in their cloud. You can get on the waiting list, but it's heavily implied that if you don't have a good use case (aka lots of prompts/money coming in), you won't get access soon.
ardit33 3y ago
Pure speculation: most likely quantizing models to 8bit to make them more efficient (and cheaper to run), and losing some accuracy on the way. I still use it, but I have found it to be less useful for coding than it was earlier. So, something is up.
- djangelic 3y ago
  I agree, this is exactly my experience as well. My earlier requests for code were more thorough then they are now. I can still use it if I give it the documentation in the prompt, but it has trouble remembering context across prompts.

spaceprison 3y ago

https://chat.openai.com/share/4cb3c711-27e6-45ee-aeda-546fc384bb40 https://chat.openai.com/share/4cb3c711-27e6-45ee-aeda-546fc3... I too had a dismal showing, but it didn't go religious on me.

sinuhe69 3y ago

I believe it’s a user reported content (thanks to the new permanent coverage link), not OpenAI volunteered, right?

deleted 3y ago

[deleted]

graiz 3y ago

I asked ChatGPT to interpret this code: text_to_encode = "<|endoftext|>" print(text_to_encode) ChatGPT: The code you provided would output an empty string.

SV_BubbleTime 3y ago
Westworld’s “This doesn’t look like anything to me”. (Man they really ruined thy show hard. Hey people like the concept of rich tourists fucking and killing robots, let’s remove all of that after the first season!!!)
- ldjkfkdsjnv 3y ago
  the first season was amazing though
- doctor_eval 3y ago
  That was such a great line. After the reviews came out, I decided that Westworld only had one season, and it was awesome.

ziml77 3y ago

I toyed around for a bunch of time yesterday and today to purposely get it to start outputting random junk. What I found interesting is that one of the the bits of text it output looked like a page from StackOverflow and another looked like an email from September 2014 to the Python mailing list. But neither seemed to actually exist. I found a StackOverflow question that looked close to the text but nothing was exactly the same. Checking the mailing list archive for that month showed that the person and their email address were real, but the rest of the text was not.

second_brekkie 3y ago

I've also got this. It was some kid using ChatGPT to get around parental controls on their xbox. Very different to the Korean grammar question I was asking.

akashkahlon 3y ago

is this 3.5 or GPT4 ?

_pdp_ 3y ago

Why is this surprising? These LLMs guess the next most likely token given initial input. With that in mind, it is possible to trip the model by injecting the tiniest amount of data at the right place. Even with that constraint, it produces some fantastic results, which speak a lot more about our intelligence than the machine's intelligence.

singularity2001 3y ago

It's probably due to the premature optimization of the tokenization scheme producing freak tokens?

okokwhatever 3y ago

And once again this thread exposes the great problem with LLMs today, it's working like a black box thats impossible to debug in any way by the operator.

daidai232 3y ago

[flagged]

wayeq 3y ago

Maybe we'll find that there is a mathematical constant that when used as the temperature parameter leads to unfalsifiable sentience.

gremlinsinc 3y ago

.....uh that was way more than 5 letters....

leoradovey1993 3y ago

[dead]