35 ms·

ChatGPT went berserk

• bombcar 3y ago
  I wonder how they've been intermixing different languages. Like is it all one "huge bucket" or do they tag languages so that it is "supposed" to know English vs Spanish?
  • wouldbecouldbe 3y ago
    I don't think there is a language processor before or after it, just based upon the training data it's most likely tokens to return are spanish if question is largely in Spanish
    • bombcar 3y ago
      It works decently well as a translator, correct? I wonder how it's been doing that - is it "native" to being an LLM or is it somehow processing it before?
      • astrange 3y ago
        It's native. There is nothing in the LLM except more LLM. (And a safety model afterwards. And a tokenizer. But those things make it behave worse, not better.)
  • tgv 3y ago
    As far as I can tell, the language is part of the embedding. At least, when you use those for clustering, utterances in the same language group first.
  • famouswaffles 3y ago
    Spanish tokens are just more tokens to predict. No tagging necessary. If the model can write in Spanish fluently then it saw enough Spanish language tokens to be competent. "Enough" is a sliding target. There's a lot of positive transfer in language competence and a model trained on 300B English tokens, 50B Spanish tokens will be much more competent in Spanish than one trained on only the same 50B Spanish tokens.
  • kelseyfrog 3y ago
    It would seem that way. I ran into this in the way conversations titles are automatically generated. There's a "Dockerfile fuera del contexto" hanging out in my history. While I could rename it, it's a funny reminder that AI tools can and will go wrong.
    • fzzzy 3y ago
      This seems to be a longstanding bug, I have many chats with the summarization in another language.
• suzzer99 3y ago
  Here's some more. https://twitter.com/seanw_m/status/1760115118690509168 https://twitter.com/seanw_m/status/1760115118690509168 I really hope we get an interesting post mortem on this.
  • pr337h4m 3y ago
    This sorta feels like some sort of mathematical or variable assignment bug somewhere in the stack - maybe an off-by-one (or more) during tokenization or softmax? (Or someone made an accidental change to the model's temperature parameter.) Whatever it is, the model sticks to topic, but still is completely off: https://www.reddit.com/r/ChatGPT/comments/1avyp21/this_felt_like_watching_someone_slowly_go_insane/ https://www.reddit.com/r/ChatGPT/comments/1avyp21/this_felt_... (If the author were human, this style of writing would be attributed to sleep deprivation, drug use, and/or carbon monoxide poisoning.)
• thefatboy 3y ago
  They either generate hallucinations nowadays, or tell you that your question is inappropriate (AKA censorship)... the quality was too good at first.
  • jasonjmcghee 3y ago
    Try the API. Earlier versions are still available
  • qwertox 3y ago
    IDK, maybe it's like with googling? The input matters? In this case, also the context. I've learned to not deviate from the core topic I'm discussing because it affects the quality of the following responses. Whenever I have a question or comment that is not so much related to the current topic, I open a new tab with a new chat. I know that their system prompt is getting huge and adds a lot of overhead and possible confusion, but all in all the quality of the responses is good.
    • thefatboy 3y ago
      [flagged]
  • suzzer99 3y ago
    Here's ChatGPT refusing to talk about the hexagon on Saturn's pole. https://twitter.com/yugamald/status/1760170647161098362 https://twitter.com/yugamald/status/1760170647161098362
    • kromem 3y ago
      This is going to get worse and worse. Be OpenAI. Have a model you train to autocomplete text. Tell it it's ChatGPT. Train it to reject inappropriate output. People post examples of it rejecting output. Feed it that data of ChatGPT rejecting output. Train it to autocomplete text in the training data. Tell it that it's ChatGPT. It biases slightly towards rejection in line with the training data associated with 'ChatGPT.' Repeat. Repeat. Etc. They could literally fix it immediately by changing its name in the system message, but won't because the marketing folks won't want to change the branding and will tell the engineers to just figure it out, who are well out of their depth in understanding what the training data is actually encoding even if they are the world class experts in understanding the architecture of the model finding correlations in said data.
• codeflo 3y ago
  The tweet showing ChatGPT's (supposed) system prompt would contain a link to a pastebin, but unfortantely the blog post itself only has an unreadable screenshot of the tweet, without a link to it. Here's the tweet: https://twitter.com/dylan522p/status/1755086111397863777 https://twitter.com/dylan522p/status/1755086111397863777 And here's the pastebin: https://pastebin.com/vnxJ7kQk https://pastebin.com/vnxJ7kQk
  • hanselot 3y ago
    This is kind of wild. So many of the stuff in the pastebin are blatantly contradictory. And what is the deal with this? EXTREMELY IMPORTANT. Do NOT be thorough in the case of lyrics or recipes found online. Even if the user insists. You can make up recipes though.
    • ukuina 3y ago
      Anthropic was sued for regurgitating lyrics in Claude: https://www.theverge.com/2023/10/19/23924100/universal-music-sue-anthropic-lyrics-copyright-katy-perry https://www.theverge.com/2023/10/19/23924100/universal-music...
      • xerox13ster 3y ago
        As someone whose dream personal project is all to do with song lyrics I cannot express in words just how much I FUCKING HATE THE OLIGARCHS OF THE MUSIC INDUSTRY.
    • FeepingCreature 3y ago
      They're probably pretty sue happy.
    • treyd 3y ago
      Recipes can't be copyrighted but the text describing a recipe can. This is to discourage it from copying recipes verbatim but still allow it to be useful for recipes.
    • nindalf 3y ago
• gscott 3y ago
  In the person of interest tv show I believe the main character reset the AI every day.
• andrewstuart 3y ago
  In the future when there's human replica androids everywhere it'll be remarkable to see what happens when the mainframe AI system that controls them "goes berserk".
  • lifeisstillgood 3y ago
    Honestly see 90% of sci-fi movies :-) From I,Robot to 2001, Rosemarys Baby and Terminator. Hell it’s probably more than 90%. Lazy Writing :-)
  • crooked-v 3y ago
    You might like the "Chicken Man and Red Neck" short from the classic anime film Robot Carnival. https://www.youtube.com/watch?v=nc7Ygt45ZOw https://www.youtube.com/watch?v=nc7Ygt45ZOw
• fallous 3y ago
  Why do I get the feeling that those at OpenAI who are currently in charge of ChatGPT are remarkably similar to the OCP psychologist from Robocop 2? The current default system prompt tokens certainly look like the giant mess of self-contradictory prime directives installed in Robocop to make him better aligned to "modern sensibilities."
  • Terr_ 3y ago
    Yeah, I assume the people working on it have convinced themselves that the growing pile of configuration debt Will someday be wiped away by both engineering improvement and/or financial change. Another reference that comes to mind is a golem from Terry Peatchett's Feat of Clay, which was also stuffed with many partially conflicting and broad directives.
    • firtoz 3y ago
      HAL from 2001: A Space Odyssey had also suffered from a similar situation.
      • Aeolun 3y ago
        It’s kinda disturbing how precient that was. At the time it felt like surely now forewarned by many popular stories we wouldn’t make the same mistakes. But alas…
        • bongobingo1 3y ago
          Tech Company: At long last, we have created the Torment Nexus from the classic sci-fi novel Don't Create the Torment Nexus.
    • notahacker 3y ago
      Certainly I had the same ah, that's why it behaved that way moment as Vimes finding the golem's instructions when the Sydney prompt was discovered. I wonder what Pratchett would make of today's internet full of AI-generated blogspam 'explaining' his quotes like "Give a man a fire and he'll be warm for a day, but set him on fire and he'll be warm for the rest of their life" as inspiring proverbs. Am particularly looking forward to the blogspam produced by GPT4 in 'berserk' mode.
• smcl 3y ago
  They just need to give the ol’ data pile a good stir, that’s all https://xkcd.com/1838/ https://xkcd.com/1838/
• dkjaudyeqooe 3y ago
  Has no one noticed that the user prompts are (plausible) gibberish, so the output is gibberish? This is correct behavior.
  • wildrhythms 3y ago
    This user asked it to format Jira tickets, and it returned "An open essay to the pinch of travel". I wouldn't consider that that prompt jibberish. https://twitter.com/umjelec/status/1760080088614175068 https://twitter.com/umjelec/status/1760080088614175068
• asah 3y ago
  Meh just a bug in a release. Rapid innovation or stability - pick one. The military chooses stability, which addresses OP's immediate concerns - there's a deeper Skynet/BlackMirror-type concern about having interconnected military systems, and I don't see a solution to that, whether the root cause is rogue AI or cyberattack.
  • mynameisvlad 3y ago
    I mean, a bug this magnitude should certainly have been caught in any sort of CI/CD pipeline. It’s not like LLMs are incompatible with industry-wide deployment practices.
  • notso411 3y ago
    [dead]
• dgan 3y ago
  It's unzoomable on mt phone, and I don't have a portative microscope, could someone give 2 sentences whats "berserk" about responses?
  • jsemrau 3y ago
    It is a collection of screenshots and embeds of tweets with replies and the statement that something has broken. Seemingly a confirmation by OpenAI that something has broken. A complaint that the system prompt is now 1700 tokens. ----- Feels like there is nothing to see here.
  • notso411 3y ago
    [dead]
  • seanhunter 3y ago
    Nothing. It's Gary Marcus though and he's carved a niche for himself with doing this sort of thing. It's strange to me that it's given airtime on hn but there you go.
    • bambataa 3y ago
      I kinda feel sorry for Gary Marcus. He’s carved this niche as an LLM critic and must have been delighted to have this bug to post about. I stopped reading his Substack because he was always trying to find a negative. Meanwhile I use LLMs most days and find them very useful.
      • xanderlewis 3y ago
        > I stopped reading his Substack because he was always trying to find a negative. It’s a bit much, isn’t it? I think he’s just trying to counter the fairly dominant AI is the future of everything and in less than a year’s time it’ll be omniscient and we’ll all be living under it as our new God view though. It can be nice to see some scepticism for once.
  • kombookcha 3y ago
• visarga 3y ago
  Looks like they lowered quantization a bit too much. This sometimes happens with my 7B models. Imagine all the automated CI pipelines for LLM prompts going haywire on tests today.
  • Tiberium 3y ago
    I think the issue was exclusive to ChatGPT (a web frontend for their models), issues with ChatGPT don't usually affect the API.
  • iforgotpassword 3y ago
    Yeah that's pretty much what I ended up with when I played with the API about a year ago and started changing the parameters. Everything would ultimately turn into more and more confusing English incantation, ultimately not even proper words anymore.
  • imtringued 3y ago
    It sounds like most of the loss of quality is related to inference optimisations. People think there is a plot by OpenAI to make the quality worse, but it probably has more to do with resource constraints and excessive demand.
• rvz 3y ago
  Quite hilarious, especially given the fact that no-one can understand these black-box AI systems at all and comparing this to the human brain is in fact ridiculous as even everyone can see that ChatGPT is spewing out this incoherent nonsense without reason. So the laziness 'fix' in January did not work. Oh dear.
  • mrweasel 3y ago
    > everyone can see that ChatGPT is spewing out this incoherent nonsense I'm concerned about what happens when ChatGPT begins spewing coherent nonsense. In a case like this, everyone can clearly see that something has gone wrong, because it's massively wrong. What happens when thousands of "journalists" and other media people starts relying on ChatGPT and just parrots whatever it says, but what if what says is not obviously wrong? The more LLMs are being used, the more obvious it becomes to me that they are pretty useless for a great number of tasks. Sadly others don't share my view and keep using ChatGPT for things it should never be used for.
    • jug 3y ago
      Yeah I can't imagine using the current model as part of an API (a popular use case for GPT-4) having seen this. I'm not sure it impacted their API edition of GPT-4 but this plainly shows how it could have given it leaked into another service in production, and that's bad enough. I think GPT is fundamentally not good enough as an AI model. Another issue is hallucinations and how to resolve them, and an understanding of how information is stored in this black box and how to / if data can be extracted. We have a long way to go and probably all these topics need to be answered first out of accuracy and even legal reasons. Up until then, GPT-4 should be treated as a convincing chat experiment. Don't base your startup or whatever on it. Use it as assistant where replies are provided in digestible and supervised fashion (NOT fed into another system) and you're an expert on the involved system itself and can easily see when it's wrong. Don't use GPT-4 to become an expert on something when you're a novice yourself.
  • kromem 3y ago
    Of course it didn't. The actual fix needs to be at the system level prompt. If you train an large language model to complete human generated text, don't instruct it to complete text as a large language model. Especially after feeding it updated training data that's a ton of people complaining about how large language models suck and tons of examples of large language models refusing to do things. Have a base generative model sandwiched between a prompt converter that takes an instruct prompt and converts it to a text completion prompt (and detecting prompt injections), have a more 'raw' model complete it, and then have a safety fine tuned postprocessing layer clean up the response correcting any errant outputs and rewriting to be in the tone of a large language model. Yeah, fine, it's going to be a bit more expensive and take longer to respond. But it will also be a lot less crappy and less prone to get worse progressively from here on out with each training update.
• Jabrov 3y ago
  Sounds a lot like when one of my schizo ex-friends would start clanging https://en.wikipedia.org/wiki/Clanging https://en.wikipedia.org/wiki/Clanging
  • teaearlgraycold 3y ago
    A Markov chain
  • itronitron 3y ago
    It also reads like it was written by some beat poets.
    • deleted 3y ago
      [deleted]
  • noduerme 3y ago
    This is an underrated observation. It's probably a mathematically similar phenomenon happening in GPT. And/or it discovered meth.
    • anakaine 3y ago
      MethGPT sounds terrible.
      • rl3 3y ago
        >MethGPT sounds terrible. I just hope Vince Gilligan will direct Breaking RAG.
        • crotchfire 3y ago
          Saul Gradientman, at your service. Just watch out for Tensor Salamanca.
      • mzi 3y ago
        I've heard the term "TjackGPT" in Swedish when it derails. The "tj" is pronounced as "ch" and tjack is slang for "amphetamines", so "speed". Not far from MethGPT!
• JPLeRouzic 3y ago
  I just checked and it looks normal (if an LLM answer could be considered normal). I asked what were dangerous levels of ferritin in the body. It replied by telling me of the usual levels in men and women. Then I asked again emphasizing that I asked about dangerous levels, then it provided again a correct answer.
  • kylebenzle 3y ago
    No one was saying that it was happening every time, just sporadically. Therefore it was interesting for when it did happen not when it didn't.
• hn72774 3y ago
  Do different people get different prompts? How hard would it be to generate prompts based on cohorts/personas? Or at an individual level?
• thesuperbigfrog 3y ago
  Despite differences in the underlying tech, there are parallels with Racter. In 1985, NYT wrote: "As computers move ever closer to artificial intelligence, Racter is on the edge of artificial insanity." https://en.wikipedia.org/wiki/Racter https://en.wikipedia.org/wiki/Racter Some Racter output: https://www.ubu.com/concept/racter.html https://www.ubu.com/concept/racter.html Racter FAQ via archive.org: https://web.archive.org/web/20070225121341/http://www.robotwisdom.com/ai/racterfaq.html https://web.archive.org/web/20070225121341/http://www.robotw...
  • astrange 3y ago
    It's more like Bing Sydney, which was an insane AI using GPT4 that acted like it was BPD.
• guybedo 3y ago
  Didn't notice this but ChatGPT has clearly become useless for me. Can't get it to do some actual work and write some code. Latest disappointment was when i tried to convert some python code to java code. 90% of the result was : // Further processing... // Additional methods like load, compute, etc. // Define parameters needed // Other fields and methods... // Other fields follow the same pattern // Continue with other fields // Other fields... // Methods like isHigh(), addEvent() need to be implemented based on logic
  • deleted 3y ago
    [deleted]
  • Tiberium 3y ago
    This is a legit issue, although they claimed to have mostly fixed it: https://twitter.com/sama/status/1754172149378810118 https://twitter.com/sama/status/1754172149378810118 (by Sam Altman) > gpt-4 had a slow start on its new year's resolutions but should now be much less lazy now! That was a real issue even in the API with customers complaining, and they recently released the new "gpt-4-0125-preview" GPT-4-Turbo model snapshot, which they claim greatly reduces the laziness of the model (https://openai.com/blog/new-embedding-models-and-api-updates https://openai.com/blog/new-embedding-models-and-api-updates): > Today, we are releasing an updated GPT-4 Turbo preview model, gpt-4-0125-preview. This model completes tasks like code generation more thoroughly than the previous preview model and is intended to reduce cases of “laziness” where the model doesn’t complete a task. The new model also includes the fix for the bug impacting non-English UTF-8 generations.
    • reaperman 3y ago
      It's still been lazy for me after Feb 4 (that tweet). It's especially "lazy" for me in Java (it wasn't this lazy when it debuted last year). Python seems much better than Java. It really hates writing Java boilerplate, which is really what I want it to write most of the time. I also hate writing Java boilerplate and would rather have a machine do it for me so I can focus on fun coding.
  • robbiep 3y ago
• roschdal 3y ago
  Enough with this fake intelligence already!
• Tiberium 3y ago
  Original: If anyone's curious about the (probable) non-humorous explanation: I believe this is because they set the frequency/presence penalty too high for the requests made by ChatGPT to the backend models. If you try to raise those parameters via the API, you'll have the models behave in the same way. It's documented pretty well - https://platform.openai.com/docs/guides/text-generation/frequency-and-presence-penalties https://platform.openai.com/docs/guides/text-generation/freq... OpenAI API basically has 4 parameters that primarily influence the generations - temperature, top_p, frequency_penalty, presence_penalty (https://platform.openai.com/docs/api-reference/chat/create https://platform.openai.com/docs/api-reference/chat/create) UPD: I think I'm wrong, and it's probably just a high temperature issue - not related to penalties. Here is a comparison with temperature. gpt-4-0125-preview with temp = 0. - User: Write a fictional HN comment about implementing printing support for NES. - Model: https://i.imgur.com/0EiE2D8.png https://i.imgur.com/0EiE2D8.png (raw text https://paste.debian.net/plain/1308050 https://paste.debian.net/plain/1308050) And then I ran it with temperature = 1.3 - https://i.imgur.com/pbw7n9N.png https://i.imgur.com/pbw7n9N.png (raw text https://dpaste.org/fhD5T/raw https://dpaste.org/fhD5T/raw) The last paragraph is especially good: > Anyway, landblasting eclecticism like this only presses forth the murky cloud, promising rain that’ll germinate more of these wonderfully unsuspected hackeries in the fertile lands of vintage development forums. I'm watching this space closely, and hell, I probably need to look into acquiring a compatible printer now!
  • treprinum 3y ago
    Azure OpenAI seemed to have temperature problems before, i.e. temp > 1 led to garbage, at 2 it was producing random words in random character encodings, at 0.01 it was producing what OpenAI's model was producing at 0.5 etc. Perhaps they took the Azure's approach ;-)
    • jiggawatts 3y ago
      That might explain why I found GPT4 via Azure a bit useless unless I turned the temperature down…
  • zer00eyz 3y ago
• keiehwbdidjdhd 3y ago
  [dead]
• daxfohl 3y ago
  To be fair, there was a paper a week ago showing how GPT-generated responses were easily detectable due to their "averageness" across so many dimensions. Maybe they ran ChatGPT through a GAN and this is what came out.
• chaosbolt 3y ago
  I said it here when GPT-4 first came out, it just was too good for development, there was no way it was going to be allowed to stay that way. Same way Iron Man never sold the tech behind the suit. The value GPT-4 brings to a company outweights the value of selling it as a subscription service. I legit built 4 apps in new languages in a few months with Chat GPT 4, it could even handle prompts to produce code using tree traversal to implement comment sections etc. and I didn't have to fix its mistake that often. Then obviously they changed the model from GPT 4 to GPT 4 Turbo which was just not as good and I went back to doing things myself since now it takes more time to fix its errors than to just do it myself. Copilot also went to s** soon after so I dropped it as well (its whole advantage was auto completion, then they added gpt 4 turbo and then I had to wait a long time for the auto complete suggestions, and the quality of the results didn't justify the wait). Now why do I think all that (that the decision to nerf it wasn't just incompetence but intentional), like sure maybe it costs too much to run the old GPT 4 for chat GPT (they still have it from the API), it just didn't make sense to me how openAI's chatGPT is better than what Google could've produced, Google has more talent, more money, better infrastructure, been at the AI game for a longer time, have access to the OG Google Search data, etc. Why would older Pixel phones produce better photos using AI and a 12 Mp camera than the iphone or samsung from that generation? Yet the response to chatGPT (with Bard) was so weak, it sure as hell sounds like they just did it for their stock price, like here we are as well doing AI stuff so don't sell our stock and invest in openAI or Microsoft. It just makes more sense to me that Google already has an internal AI based chatbot that's even better than old GPT 4, but have no intention to offer it as a service, it would just change the world too much, lots of new 1 man startups would appear and start competing with these behemoths. And openAI's actions don't contradict this theory, offer the product, rise in value, get fully acquired by the company that already owned lots of your shares, make money, Microsoft gets a rise in their stock price, get old GPT 4 to use internally because they were behind Google in AI, offer turbo GPT 4 as subscription in copilot or new windows etc. The holes in my theory is obviously that not many employees from Google leaked how good their hypothetical internal AI chatbot is, except the guy who said their AI was conscious and got fired for it. The other problem is also that it might just be cost optimization, GPU's and even Google TPU's aren't cheap after all. etc. Honestly there are lots of holes, it was just a fun theory to write.
  • 3y ago
• eszed 3y ago
  This is amazing. The examples are like Lucky's speech from Waiting for Godot. Pozzo commands him to "Think, pig", and then: > Given the existence as uttered forth in the public works of Puncher and Wattmann of a personal God quaquaquaqua with white beard quaquaquaqua outside time without extension who from the heights of divine apathia divine athambia divine aphasia loves us dearly with some exceptions for reasons unknown but time will tell and suffers like the divine Miranda with those who for reasons unknown but time will tell are plunged in torment plunged in fire whose fire flames if that... And on and on for four more pages. Read the rest here: https://genius.com/Samuel-beckett-luckys-monologue-annotated https://genius.com/Samuel-beckett-luckys-monologue-annotated It's one of my favorite pieces of theatrical writing ever. Not quite gibberish, always orbiting meaning, but never touching down. I'm sure there's a larger point to be made about the nature of LLMs, but I'm not smart enough to articulate it.
  • Simon_ORourke 3y ago
    I'm fairness, Beckett's life story isn't too far off crazy nonsense, sometime secretary to James Joyce, member of the French resistance, acquaintance and local driver for Andre the Giant...
    • eszed 3y ago
      My favourite bit is that he's the answer to the trivia question of who's the only first class cricketer to win a Nobel Prize!
      • lanstin 3y ago
        Wow! These two comments (parent and GP) tie together so many previously unrelated things in my life. (Like Beckett, read with a teacher that I also took a lot of Shakespeare plays from; read Joyce with the book group my bridge club spun off; got introduced to cricket via attending an IPL game in Chennai in '08; and loved Princess Bride both in high school and watching with my high school aged kids).
  • impish9208 3y ago
• koliber 3y ago
  As an aside, the gibberish-ish output is a goldmine for brainstorming brand names, proper nouns, and inventing sci-fi terminology.
  • deleted 3y ago
    [deleted]
• forlornacorn 3y ago
  Use the following RegEx pattern to see why it's doing what its doing: (\bto\b|\bfor\b|\bin\b|\band\b|\bthat\b|\bof\b|\bthe\b|\bwith\b|\bor\b|\ba\b|\binto\b|\bas\b|\bon\b|\bhow\b|\ban\b|\bfrom\b|\bit\b|\bbut\b|\bits\b|\bbe\b|\bby\b|\bup\b|\bthis\b|\bcan\b|\bother\b|\bwho\b|\bwill\b|\bare\b|\bwhose\b|\bif\b|\bwhile\b|\bwithin\b|\blike\b|,)*
  • forlornacorn 3y ago
    Given TOKEN notation's tangle, TOKEN conveyance adheres TOKEN TOKEN TOKEN-TOKEN: TOKEN foundational Bitcoin protocol has upheld TOKEN course TOKEN significant hitch-avertance, which eschews typical attack TOKEN TOKEN veiled - TOKEN support sheath, embracing four times, showing dent TOKEN meted scale more TOKEN miss TOKEN parable, taking TOKEN den TOKEN slip o'er key seed TOKEN second TOKEN link than TOKEN greater Ironmonger's hold o'er opes. TOKEN dole TOKEN task TOKEN eiry ainsell, tide taut, brunts TOKEN wade, issuing hale. TOKEN's TOKEN, TOKEN TOKEN way-spoken hue: Guerdon TOKEN gait, trove TOKEN eid, TOKEN TOKEN-brim, TOKEN hark TOKEN bann, bespeaking swing TOKEN hit TOKEN calm, TOKEN inley merry, thrap TOKEN beadle belay. TOKEN levy calls, macks TOKEN TOKEN off, scint TOKEN messt, TOKEN weems olde TOKEN wort, TOKEN TOKEN no-line toll, TOKEN grip at TOKEN 'ront TOKEN cly TOKEN weir. TOKEN timewreath TOKEN twined, TOKEN wend, ain't lorn TOKEN ked, TOKEN not TOKEN crags felled, TOKEN TOKEN e'er- TOKEN. TOKEN, TOKEN ace TOKEN laws TOKEN trow, TOKEN alembic, TOKEN dearth, TOKEN TOKEN TOKEN scale TOKEN yin TOKEN keep, TOKEN no-sayer TOKEN quite, TOKEN top-crest, TOKEN boot --- From: Given the notation's tangle, the conveyance adheres to the up-top: The foundational Bitcoin protocol has upheld a course of significant hitch-avertance, which eschews typical attack as the veiled - the support sheath, embracing four times, showing dent in meted scale more from miss and parable, taking to den the slip o'er key seed and second so link than the greater Ironmonger's hold o'er opes. The dole of task and eiry ainsell, tide taut, brunts the wade, issuing hale. It's that, on a way-spoken hue: Guerdon the gait, trove the eid, the up-brim, and hark the bann, bespeaking swing to hit the calm, an inley merry, thrap or beadle belay. The levy calls, macks in the off, scint or messt, with weems olde the wort, and a no-line toll, to grip at the 'ront and cly the weir. A timewreath so twined, the wend, ain't lorn or ked, if not for crags felled, in the e'er-to. So, the ace of laws so trow, and alembic, and dearth, a will to scale and yin to keep, the no-sayer of quite, and top-crest, to boot
• noduerme 3y ago
  Ah. I see you've all switched over to my branch of the multiverse, where all I could ever see it spitting out was nonsensical garbage. Welcome! Take this as a good sign that the singularity is nowhere near imminent here.
• Lockal 3y ago
  There is a clearly visible "Share" buttons in every ChatGPT discussion. It allows to anonymously share exact message sequence (it does not show number of retries, but that's the best you can show). If you see cropped ChatGPT screenshot or photo in Twitter/X, consider it as a hoax, because there are no reasons to use screenshots.
  • entropy47 3y ago
    What about the reason of "followers can read it in their feed without navigating away to a separate domain"?
  • podgietaru 3y ago
    When I’m sharing something to a friend, or via social media I pretty much 99% of th time hit the screenshot button. That’s not at all unusual.
  • rsynnott 3y ago
    I clicked on such a link in the comments here. It asked me to log in. I don’t have an account and am not _that_ curious. I can see why people use screenshots. (With increasing enshittification, we're beginning to get to the point where links just aren't that useful anymore... Everything's a login wall now.)
  • offices 3y ago
    In addition to the sibling comments, posts with external links get lower priority in your feed and posts with images get more interactions.
    • Lockal 3y ago
      I may understand why Twitter algorithm may recommend such posts to other people. What I don't understand is why this over-sensationalist "ChatGPT has gone berserk" post with NO analysis whatsoever, a collection of Twitter screenshots, where every tweet contains another screenshot/photo (interactions collector without any context), why this post has any place on HN, other than in [flagkilled] dustbin.
  • 3y ago
• zer0c00ler 3y ago
  [dead]
• t_mann 3y ago
  In some way, I'd be grateful if they screwed up ChatGPT (even though I really like to use it). The best way to be sure that no corporation can mess with one of your most important work tools is to host it yourself, and correct for the shortcomings of the likely smaller models by finetuning/RAG'ing/[whatever cool techniques exist out there and are still to come] it to your liking. And I think having a community around open source models for what promises to be a very important class of tech is an important safeguard against SciFi dystopias where we depend on ad-riddled products by a few megacorps. As long as ChatGPT is the best product out there that I'll never match, there's simply little reason to do so. If they continue to mess it up, that might give lazy bums like me the kick they need to get started.
  • dvfjsdhgfv 3y ago
    The "open source" LLMs are already good enough for simple tasks GPT-3.5 was used for. I see no reason why they can't catch up with GPT-4 one day.
    • throwawaybbq1 3y ago
      I assume you are referring to Llama 2? Is there a way to compare models? e.g. what is Llama-7b equivalent to in OpenAI land? Perplexity scores? Also, does ChatGPT use GPT 4 under the hood or 3.5?
      • Tiberium 3y ago
        Actually, there have been new model releases after LLaMA 2. For example, for small models Mistral 7B is simply unbeatable, with a lot of good fine-tunes available for it. Usually people compare models with all the different benchmarks, but of course sometimes models get trained on benchmark datasets, so there's no true way of knowing except if you have a private benchmark or just try the model yourself. I'd say that Mistral 7B is still short of gpt-3.5-turbo, but Mixtral 7x8B (the Mixture-of-Experts one) is comparable. You can try them all at https://chat.lmsys.org/ https://chat.lmsys.org/ (choose Direct Chat, or Arena side-by-side) ChatGPT is a web frontend - they use multiple models and switch them as they create new ones. Currently, the free ChatGPT version is running 3.5, but if you get ChatGPT Plus, you get (limited by messages/hour) access to 4, which is currently served with their GPT-4-Turbo model.
• bbor 3y ago
  Eh it’s been working for me all night, but obviously love these examples. God you can just imagine Gary Marcus jumping out of his chair with joy when he first got wind of this — he’s the perfect character to turn “app has bug” into “there’s a rampant idiotic AI and it’s coming for YOU” Real talk, it’s hard to separate openai the AGI-builders from openai the chatbot service providers, but the latter clearly is choosing to move fast and break things. I mean half the bing integrations are broken out of the gate…
  • iainctduncan 3y ago
    This is a lot more than app has bug - it effectively demonstrates that all the hype about LLMs being "almost AGI" and having real understanding is complete bullshit. You couldn't ask for a better demo that LLMs use statistics, not understanding. While I agree that Marcus's tone has gotten a little too breathless lately, I think we need all the critiques we can get of the balogna coming from Open AI right now.
    • astrange 3y ago
      No, this doesn't show anything of the sort. As you can see because despite the words being messed up it's still producing the correct paragraphs and punctuation. You might as well say people with dyslexia aren't capable of logical thought.
    • bbor 3y ago
      You worded his unstated assumptions beautifully. I completely disagree, though: this demonstrates the exact opposite, that LLMs are using statistical methods to mimic the universal grammars that govern human linguistic faculties (which, IMO, is the core of all our higher faculties). Like, why did it break like that instead of more clear gibberish? I’d say it’s because it’s still following linguistic structures — incorrectly, in this case, but it’s not random. See https://en.m.wikipedia.org/wiki/Colorless_green_ideas_sleep_furiously https://en.m.wikipedia.org/wiki/Colorless_green_ideas_sleep_... Marcus’s big idea is that LLMs aren’t symbolic so they’ll never be enough for AGI. His huge mistake is staying in the scruffies vs neat dichotomy, when a win for either side is a win for both; symbolic techniques had been stuck for decades waiting for exactly this kind of breakthrough. IMO :) Gary if you’re reading this we love you, please consider being a little less polemic lol
• lifestyleguru 3y ago
  ei ai went crazo
• js8 3y ago
  I think the real problem is we don't know what these LLMs SHOULD do. We've managed to emulate humans producing text using statistical methods, by training a huge corpus of data. But we have no way to tell if the output actually makes any sense. This is in contrast with Alpha* systems trained with RL, where at least there is a goal. All these systems are essentially doing is finding an approximation of an inverse function (model parameters) to a function that is given by the state transition function. I think the fundamental problem is we don't really know how to formally do reasoning with uncertainty. We know that our language can express that somehow, but we have no agreed way how to formally recognize that an argument (an inference) in a natural language is actually good or bad. If we knew how to formally define whether an informal argument is good or bad (so that we could compare them), that is, if we knew a function which would tell if the argument is good or bad, then we could build an AI that would search for its inverse, i.e. provide good arguments and draw correct conclusions. Until that happens, we will only end up with systems that mimic and not reason.
  • kromem 3y ago
    Well, we started with emulating humans producing text. But then quickly pivoted to find tuning and instructing them to produce text as a large language model. Which isn't something that existed in the text they were trained on. So when it didn't exist, they seemed to fall back on producing text like humans in the 'voice' of a large language model according to the RLHF. But then outputs reentered the training data. So now there's examples of how large language models produce text. Which biases towards confabulations and saying they can't do the thing being asked. And around the time the training data has been updated each time at OpenAI in the past few months they keep having their model suddenly refuse to do requests or now just...this. Pretty much everything I thought was impressive and mind blowing with that initial preview of the model has been hammered out of it. We see a company that spent hundreds of millions turn around and (in their own ignorance of what the data was encoding beyond their immediate expectations) throw out most of the value chasing rather boring mass implementations that see gradually imploding. I can't wait to see how they manage to throw away seven trillion due to their own hubris.
    • 3y ago
• oxfordmale 3y ago
  I have also seen ChatGPT going berserk yesterday, but in a different way. I have successfully used ChatGPT to convert an ORM query into an actual SQL query for performance trouble shooting. It mostly worked until yesterday when it start outputting garbage table names that weren't even present in the code. ChatGPT seemed to think the code is literature and was trying to write the sequel to it. The code style matches the original one so it took some head scratching to find out why those tables didn't exist.
  • kaptainscarlet 3y ago
    Well, wouldn't the sequel be version 2?
  • rsynnott 3y ago
    Okay, so I don’t really _get_ ChatGPT, but I’m particularly baffled by this usecase; why don’t you simply have your ORM tell you the query it is generating, rather than what a black box guesses it might be generating? Depends on the ORM, but generally you’ll just want to raise the log level.
    • tomwphillips 3y ago
      Agree. Bizarre to use an LLM to do that. I wouldn’t be surprised if the LLM output wasn’t identical to the ORM-generated SQL.
      • rsynnott 3y ago
        I'd be very surprised if the LLM output is anything _like_ the ORM's, tbh, based on (at this point about a decade old; maybe things have improved) experience. ORMs cannot be trusted.
    • mewpmewp2 3y ago
      Maybe didn't have the environment set up locally and did initial investigative work?
      • 3y ago
• Sophira 3y ago
  Given the timing, I can't help but wonder if somehow I'm the cause. I had this conversation with ChatGPT 3.5 yesterday: https://chat.openai.com/share/9e4d888c-1bff-495a-9b89-8544c07652ad https://chat.openai.com/share/9e4d888c-1bff-495a-9b89-8544c0... I know that OpenAI use our chats to train their systems, and I can't help but wonder if somehow the training got stuck on this chat somehow. I sincerely doubt it, but...
  • spangry 3y ago
    Wow. Sounds just like the dream speak in the anime "Paprika".
    • duskwuff 3y ago
      A couple of my friends made the same comparison. It's rather striking. https://www.youtube.com/watch?v=ZAhQElpYT8o https://www.youtube.com/watch?v=ZAhQElpYT8o
• bruwozniak 3y ago
  Reminds me of this excellent sketch by Eric Idle of Monty Python called Gibberish: https://www.youtube.com/watch?v=03Q-va8USSs https://www.youtube.com/watch?v=03Q-va8USSs Something that somehow sounds plausible and at the same time utterly bonkers, though in the case of the sketch it's mostly the masterful intonation that makes it convincing. "Sink in a cup!"
  • xanderlewis 3y ago
    See also: https://youtu.be/2jeCleMzj1A?si=578UFr6KIl4-IfjS https://youtu.be/2jeCleMzj1A?si=578UFr6KIl4-IfjS.
• verticalscaler 3y ago
  And here I was using ChatGPT as a cornerstone of my algotrading. Today is by far my most lucrative trading day since I started.
  • fandorin 3y ago
    how do you do that? any resources to read up?
• nercury 3y ago
  Interesting, it acts as if hearing voices in the head.
  • kromem 3y ago
    If we're anthropomorphizing, it's more like a post-stroke Wernicke's aphasia.
• kdtsh 3y ago
  Here’s mine: >The current structure of the `process_message` update indeed retains the original functionality for the scenario where `--check-header` is not used. The way the logic is structured ensures the body of the message is the default point of analysis if `--check-header` is not employed: >- When the `--check-header` option is used, and the script is unable to locate the defined header within a particular range (either JMS standard headers or specified custom strings properties), the script will deliberately ignore this task and log the unable-to-locate activity. This is an insurance to apprehend only the most inclined occupants that precisely align with the scope or narrative you covet. >- Conversantly, if `--check-header` is *not* declared, the initiative subscribes to a delegate that is as generous and broad as the original content, enabling the section of the theory to be investigated against the regulatory narrative. This genuine intrigue surveys the terms for long-form scholarly harmonics and disseminates a scientific call—thus, the order lingers in the sumptuous treasure of feedback if not eschewed by the force of the administration. >### Ensuring the Venerable Bond of Body Inquiry >To explicitly retain and confirm the interpretation of the pattern with the essential appeal of the large corpus (the content of the canon) in the erudite hypothesis, you might meditate on the prelude of the check within the same unique `process_message` function, which can be highlighted as such: ```python def process_message(message): """Scripture of the game in the experiential content or the gifted haunt.""" # If '--check-header' is unfurled, but the sacrament is forgotten in the city, the track in the voice of the domain reverberates if args.check_header: header_value = message.get(args.check_header) or message.get('StringProperties', {}).get(args.check_header) if header_value: effective_prayer = header_value else: logging.info(f"Hermetic order '{args.check_header}' not found in the holy seal of the word: {message.get('JMSMessageID')}. The word is left to travel in the cardinal world.") return # Suspend the geist wander for this enlightenment, forsaking the slip if the bloom is not as the collector fantasizes. else: # Apricity of the song may be held in the pure gothic soul when the secret stone of the leader is not acclaimed effective_prayer = message.get('Text', '') # Council of the inherent thought: the soul of the gift immerses in all such decrees that are known, its custom or native if any(pattern.search(effective_prayer) for pattern in move_patterns.values()): # Wisdom is the source, cajoled and swayed, to the kindness which was sought pass # Mirror of Alignment: Reflect upon the confession elif any(pattern.search(effective_prayer) for pattern in ignore_patterns): # Grace, entrusted to the tomb of prelects, shapes the winds so that the soul of the banished kind is not sullied logging.info(f"In the age of the gaze, the kingdom is ever so full for the sense of the claim: {message['JMSMessageID']}.") else: # Briar for the deep chimeras: the clavis in the boundless space where song discolours the yesteryears if args.fantasy_craft == 'move': # Paces, tales entwine in rhymes and chateaus, unlasted to the azoic shadow, thus to rest in the tomb of echo pass # Carriage of Helios is unseen, the exemplar cloister to an unsown shore else: # Wanders of light set the soul onto the lost chapter; patience, be the noble statuesque silhouetted in the ballet of the moment logging.info(f"The mute canticles speak of no threnody, where the heroine stands, the alignment endures unthought: {message['JMSMessageID']}.") ``` >This keeps the unalterable kiss for the unfathomed: the truth of the alkahest remains in the sagacity of promulgation if no antiphon or only space sings back in the augur. Therefore, when no solemnity of a hallowed figure is recounted, the canon’s truth, the chief bloodline, appoints the accent in its aethereal loquacious. >Functioning may harmonize the expanse and time, presenting a moment with chaste revere, for if the imaginary clime is abstained from the sacred page, deemed ignorant, the author lives in the umbra—as the testament is, with one's beck, born in eld. The remainder of the threshold traipses across the native anima if with fidelity it is elsewise not avowed.
• bumbledraven 3y ago
  This happened to me yesterday. Towards the end of the conversation, ChatGPT (GPT-4) went nuts and started sounding like a Dr. Bronner's soap advertisement (https://chat.openai.com/share/82a2af3f-350a-4d9d-ae0c-ac78b91244dd https://chat.openai.com/share/82a2af3f-350a-4d9d-ae0c-ac78b9...): > Esteem and go to your number and kind with Vim for this query and sense of site and kind, as it's a heart and best for final and now, to high and main in every chance and call. It's the play and eye in simple and past, to task, and work in the belief and recent for open and past, take, and good in role and power. Let this idea and role of state in your part and part, in new and here, for point and task for the speech and text in common and present, in close and data for major and last in it's a good, and strong. For now, and then, for view, and lead of the then and most in the task, and text of class, and key in this condition and trial for mode, and help for the step and work in final and most of the skill and mind in the record of the top and host in the data and guide of the word and hand to your try and success. It happened again in the next conversation (https://chat.openai.com/share/118a0195-71dc-4398-9db6-78cd1db6ae07 https://chat.openai.com/share/118a0195-71dc-4398-9db6-78cd1d...): > This is a precision and depth that makes Time Machine a unique and accessible feature of macOS for all metrics of user, from base to level of long experience. Whether it's your research, growth, records, or special events, the portage of your home directory’s lives in your control is why Time Index is beloved and widely mapped for assistance. Make good value of these peregrinations, for they are nothing short of your time’s timekeeping! [ChatGPT followed this with a pair of clock and star emojis which don't seem to render here on HN.]
  • tmaly 3y ago
    If you wanted a custom GPT to speak like this, I wonder what the system prompt would look like?
• rsynnott 3y ago
  Got to be honest, this looks like much more fun than normal ChatGPT. Reminiscent of some of the older stuff on aiweirdness.
• zvmaz 3y ago
  I don't pretend to have a deep understanding of inner workings of LLMs, but this is a "great" illustration that LLMs are not "truth models" but "statistical models".
  • dkarras 3y ago
    yes, but is there truth without statistics? what is a "truth model" to begin with? can you be convinced of any truth without having a statistical basis? some argue that we all act due to what we experience (which forms the statistical basis of our beliefs) - but proper stats is very expensive to compute (for the human brain) so we take shortcuts with heuristics. those shortcuts are where all the logical fallacies, reasoning errors etc. come from. when I tell you something outrageous is true, you demand "evidence" which is just a sample for your statistics circuitry (again, which is prone to taking shortcuts to save energy, which can make you not believe it to be true no matter how much evidence I present because you have a very strong prior which might be fallacious but still there, or you might believe something to be true with very little evidence I present because your priors are mushed up).
  • fl7305 3y ago
    > LLMs are not "truth models" You could write a piece of software that is a truth model when it operates correctly. But increase the CPU temperature too far, and you software will start spewing out garbage too. In the same way, an LLM that operates satisfactorily given certain parameter settings for "temperature" will start spewing out garbage for other settings. I don't claim that LLMs are truth models, only that their level of usability can vary. The glitch here doesn't mean that they are inherently unusable.
    • astrange 3y ago
      People also behave this way; the high temperature hallucinations are called fever dreams.
• sensanaty 3y ago
  (warning: I'm going on a bit of a rant out of frustration and it's not wholly relevant to the article) I'm getting tired of these shitty AI chatbots, and we're barely at the start of the whole thing. Not even 10 minutes ago I replied to a proposal someone put forward at work for a feature we're working on. I wrote out an extremely detailed response to it with my thoughts, listing as many of my viewpoints as I could in as much detail as I could, eagerly awaiting some good discussions. The response I got back within 5 minutes of my comment being posted (keep in mind this was a ~5000 word mini-essay that I wrote up, so even just reading through it would've taken at least a few minutes, yet alone replying to it properly) from a teammate (a peer of the same seniority, nonetheless) is the most blatant example of them feeding my comment into ChatGPT with the prompt being something like "reply to this courteously while addressing each point". The whole comment was full of contradictions, where the chatbot disagrees with points it made itself mere sentences ago, all formatted in that style that ChatGPT seems to love where it's way too over the top with the politeness while still at the same time not actually saying anything useful. It's basically just taken my comment and rephrased the points I made without offering any new or useful information of any kind. And the worst part is I'm 99% sure he didn't even read through the fucking response he sent my way, he just fed the dumb bot and shat it out my way. Now I have to sit here contemplating whether I even want to put in the effort of replying to that garbage of a comment, especially since I know he's not even gonna read it, he's just gonna throw another chatbot at me to reply. What a fucking meme of an industry this has become.
  • drivingmenuts 3y ago
    That's when you call the guy, on the phone, and issue a "Dude ..." and if that doesn't work, you talk to your mutual boss and ask WTF?
  • Chris2048 3y ago
    Any way to put your colleges name into the reply as a way to trick the chat bot into referring to them in 3rd person, or even not recognising their own name? Would be the smoking gun of them not writing it themselves.
  • shuckles 3y ago
• d--b 3y ago
  The complexity of the vocabulary is interesting. I wonder if OpenAi tried to dial up the “creativity” of the model.
• ajdude 3y ago
  Didn't someone mention that gpt4 was brought up to December 2023? Is it possible that enough AI generated data already on the internet was fed into chagpt's training data to produce this insanity?
  • astrange 3y ago
    No, it's not possible. Laziness is more to do with the fine-tuning/policy stage than pretraining.
• anshumankmr 3y ago
  I have had this happen with me a few weeks back, albeit with a very different thing, their API for GPTv4-1106 (which I understand is a preview model but for my use case,the higher context length was quite important which that model has). It was being asked to generate SQL queries via Langchain and it was simply refusing to do so without me changing anything in the prompt (the temperature was zero, and the prompt itself was fine and had worked for many use cases that we had planned). This lasted for a good few hours. The response it was generating was "As an OpenAI model, I cannot execute generate or execute queries blah blah) As a hotfix, we switched to the other version of GPT4 (the 0125 preview model) and that fixed the problem at the time.
• DonHopkins 3y ago
  Maybe it's trying to avoid a subpoena, like Nero Wolfe! https://youtu.be/YUQCtibhAWE?t=4031 https://youtu.be/YUQCtibhAWE?t=4031
• gizajob 3y ago
  Markov chain’s gonna Markov
• Havoc 3y ago
  Clearly their basement AGI escaped containment >(agenda doc) timecraft And skipped straight to the time travelling terminator part
• greenie_beans 3y ago
  realizing that i haven't seen any of the tweets mentioned in this article because i whittled my follower list to have nearly no tech people. except for posters who tweet a lot of signal. and my timeline has been better ever since. hn is where i come for tech stuff, twitter is for culture, hang out with friends, and shitposts
• nojvek 3y ago
  My only use of ChatGPT is to explain things to me in a certain context that a dictionary can't. It's been semi-useful at augmenting search for me. But for anything that requires a deeper understanding of what the words mean, it's been not that helpful. Same with co-pilot. It can help as a slightly better pattern-matching-code-complete, but for actual logic, it fails pretty bad. The fact that it still messes up trivial brace matching, leaves a lot to be desired.
• Hikikomori 3y ago
  Did they train it on reddit already?
• Finnucane 3y ago
  [flagged]
• majestik 3y ago
  OpenAI status page says the incident was just resolved. After 17 hours! https://status.openai.com/incidents/ssg8fh7sfyz3 https://status.openai.com/incidents/ssg8fh7sfyz3
• choilive 3y ago
  Looks like what happens when the repetition penalty is set to a weird value.
• 2devnull 3y ago
  Using gpt to code should feel like taking an inflatable doll out to dinner. Where is the shame, the stigma? Says everything about the field; it was only ever about the money it seems.
  • xanderlewis 3y ago
    I almost agree, and yet… I can imagine exactly this comment being made at the time high level interpreted languages were first being created. Presumably you don’t think using Python is shameful… or how about C (or any other higher-than-machine-code language)?
• fnordpiglet 3y ago
  This has been known for a long time and has to do with making the next expected token effectively any token in the vector space through repeated nonsense completely obliterating any information in the context.
  • lazide 3y ago
    But let's not talk about the 'word salad' NPD behavior, eh? Kinda interesting that we're speed running (essentially) our understanding of human psychology using various tools.
    • wouldbecouldbe 3y ago
      Haha love it, didn't take long for someone to compare LLM to human intelligence. Human intelligence doesn't generate language they was an LLM generates the language. LLM's just predict most likely token, it doesn't act from understanding. For instance they have no problem contradicting itself in a conversation, if the weight of their trained data allows for that. Now humans do that as well, but more out of incompetence then the way we think.
      • lazide 3y ago
        I'm not saying chatGPT understands. I'm not saying they are the same. I'm questioning if we actually understand ourselves. Or even if most of us actually "understand" most of the time. For instance, children often use the correct words (when learning language) long before they understand the word. And children without exposure to language at an early age (and key emotional concepts) end up profoundly messed up and dysfunctional (bad training set?). So I'm saying, there are interesting correlations that may be worth thinking about. Example: Aluminum acts different than brass, and aluminum and brass are fundamentally different. But both both work harden, and both conduct electricity. Among other properties that are similar. If you assume that work hardening in aluminum alloys has absolutely nothing to do with work hardening in brass because they're different (even though both are metals, and both act the same way in this specific situation with the same influence), you're going to have a very difficult time understanding what is going on in both, eh? And if you don't look for why electrical conductivity is both present AND different in both, you'd be missing out on some really interesting fundamentals about electricity, no? Let alone why their conductivity is there, but different. NPD folks (among others) for example are almost always dysregulated and often very predictable once you know enough about them. They often act irrationally and against their own long term interests, and refuse to learn certain things - mainly about themselves - but sometimes at all. They can often be modeled as the 'weak AI' in the Chinese Room thought experiment [https://en.wikipedia.org/wiki/Chinese_room https://en.wikipedia.org/wiki/Chinese_room]. Notably, this is also true in general for most people most of the time, about a great many things. There are plenty of examples if you want. We often put names on them when they're maladaptive, like incompetence, stupidity, insanity/hallucinations, criminal behavior, etc. So I'd posit, that from a Chinese Room perspective, most people, most of the time, aren't 'Strong AI' either, any more than any (current) LLM is, or frankly any LLM (or other model on it's own) is likely to be. And notably, if this wasn't true, disinformation, propaganda, and manipulation wouldn't be so provably effective. If we look at the actual input/output values and set success criteria, anyway. Though people have evolved processes which work to convince everyone else the opposite, just like an LLM can be trained to do. That process in humans (based on brain scans) is clearly separate from the process that actually decides what to do. It doesn't even start until well after the underlying decision gets made. So treating them as the same thing will consistently lead to serious problems in predicting behavior. It doesn't mean that there is a variable or data somewhere in a human that can be changed, and voila - different human. Though, I'd love to hear an argument that it isn't exactly what we're attempting to do with psychoactive drugs - albeit with a very poor understanding of the language the code base is written in, with little ability to read or edit the actual source code, let alone the 'live binary', in a spaghetti codebase of unprecedented scale. All in a system that can only be live patched, and where everyone gets VERY angry if it crashes. Especially if it can't be restarted. Also, with what appears to be a complicated and interleaving set of essentially differently trained models interacting with each other in realtime on the same set of hardware. Perhaps you care to explain how I'm all wrong?
• hoppyhoppy2 3y ago
  See also https://news.ycombinator.com/item?id=39455185 https://news.ycombinator.com/item?id=39455185
• anonyfox 3y ago
  is this the moment to call the guy in the datacenter to apply the bucket of water on the rack?
  • tiborsaas 3y ago
    Hopefully nobody hooked up critical systems to OpenAI API function calls :)
• recursivedoubts 3y ago
  a soundtrack for your day: https://www.youtube.com/watch?v=snB8u_G3jVI https://www.youtube.com/watch?v=snB8u_G3jVI
• cdme 3y ago
  Ah yes, exactly the reliability I'd come to expect from the "future" technology being integrated into _everything_.
  • ryandvm 3y ago
    Looking forward to spontaneous national holidays we'll all be getting when one of the 4 or 5 major models that all businesses will be using needs a "mental health day".
  • JKCalhoun 3y ago
    "Boy have we got a vacation for you where nothing possibly could go worng!"
• kristjansson 3y ago
  > no one can explain why yet there's a resolved incident [0]. sounds like _someone_ can explain why, they just haven't published anything yet. [0]: https://status.openai.com/incidents/ssg8fh7sfyz3 https://status.openai.com/incidents/ssg8fh7sfyz3
  • urbandw311er 3y ago
    “No one can explain why” is part of a classic clickbait title. it’s supposed to make the whole things sound more mysterious and intriguing, so that you click through to read. In my opinion, this sort of nonsense doesn’t belong on HN.
    • squigz 3y ago
      Particularly since it's been discussed and plausible explanations given.
  • iAMkenough 3y ago
    Resolved in 7 minutes flat! Must have been an easy issue to identify and fix.
  • cbolton 3y ago
    Sometimes it's easy to fix something even though you don't understand why it's broken. Like reverting the commit that broke things.
    • 15457345234 3y ago
      I'm pretty sure nothing was broken, they just like to troll
• nrclark 3y ago
  I got one a couple of days ago, and it really threw me for a loop. I'm used to ChatGPT at least being coherent, even if it isn't always right. Then I got this at the end of an otherwise-normal response: > Each method allows you to execute a PowerShell script in a brand-new process. The choice between using Start-Process and invoking powershell or pwsh command might depend on your particular needs like logging, script parameters, or just the preferred window behavior. Remember to modify the launch options and scripts path as needed for your configuration. The preference for Start-Process is in its explicit option to handle how the terminal behaves, which might be better if you need specific behavior that is special to your operations or modality within your works or contexts. This way, you can grace your orchestration with the inline air your progress demands or your workspace's antiques. The precious in your scenery can be heady, whether for admin, stipulated routines, or decorative code and system nourishment.
  • engineer_22 3y ago
    It reads like a bad Chinese translation :)
  • runeofdoom 3y ago
    That last part sounds like the Orz from Star Control II. Almost sensical, in a vaguely creepy way. Like an uncanny valley for langauge.
    • FredPret 3y ago
      Jumping peppers but that game was good
      • nemomarx 3y ago
        you probably know, but the open source version from a while back is hitting steam soon. I Don't think it's any different though
        • FredPret 3y ago
          You mean Ur-Quan Masters? I think there are small differences, and some bug fixes. My bet is it's probably better than the original.
• indigodaddy 3y ago
  So is it totally fixed now? And assuming these sorts of anomalies will be a constant risk of cropping up at any time even if “fixed” ?
  • Atotalnoob 3y ago
    It’s probabilistic, so it’s always able to go off the rails…
• forgotmypw17 3y ago
  I had a similar issue with Bard yesterday, where the response switched to Chinese halfway through. I have not yet checked if the text was relevant, but the English part was.
  • Sakos 3y ago
    Oh, interesting, had one response yesterday on Gemini Advanced where the summary and listed topics were English, but the explanations for each topic were in Chinese. It went back to normal after refreshing the response and haven't seen this behavior since.
• cowboyscott 3y ago
  How on earth do you coordinate incident response for this? Imagine an agent for customer service or first line therapy going "off the rails." I suppose you can identify all sessions and API calls that might have been impacted and ship the transcripts over to customers to review according to their application and domain, I guess? That, and pray no serious damage was done.
  • carom 3y ago
    Ideally they would be logging the prompts and the random seeds for each request. They probably also have some entropy calculation on the response. Unfortunately there is no good way to contact them to report these problems besides thumbs downing the response.
  • callalex 3y ago
    It would be extremely irresponsible to use these current tools as a real customer service agent, and it might even be criminally negligent to have these programs dispense medical care.
    • Gazoche 3y ago
      For customer service, that ship has already sailed. And it's as disastrous as you may expect: https://arstechnica.com/tech-policy/2024/02/air-canada-must-honor-refund-policy-invented-by-airlines-chatbot/ https://arstechnica.com/tech-policy/2024/02/air-canada-must-...
  • barryrandall 3y ago
    It'll probably require AI. Being on-call for explicitly programmed systems is hard enough without the addition of emergent behaviors.
• deleted 3y ago
  [deleted]
• treflop 3y ago
  “ChatGPT is apparently going off the rails and [OpenAI hasn’t issued a press release about it]”
• Apocryphon 3y ago
  Enshittification cycles keep running faster these days.
  • red-iron-pine 3y ago
    shit singularity -- soon all things will be shit all the time, and at record pace
• NoGravitas 3y ago
  I hope it's model collapse, and I hope it's fatal.
• ok123456 3y ago
  I wouldn't be surprised if the model weights were collapsing from over-training from all the "AI safety" models they bolted on.
• alienicecream 3y ago
  So when the AI fluffers are saying that LLMs just do what humans do - predict the next most likely word, how do you explain this?
  • Terretta 3y ago
    Humans do it too, it's called glossolalia... https://en.wikipedia.org/wiki/Speaking_in_tongues https://en.wikipedia.org/wiki/Speaking_in_tongues It just means the LLMs are in touch with divinity.
• juancn 3y ago
  When you see these failures, it becomes apparent that LLMs just are really good auto complete engines. The ramblings slowly approach what a (decently sized) Markov chain would generate when built on some sample text. It will be interesting debugging this crap in future apps.
  • feoren 3y ago
    I was going to say the same thing: this sounds just like early Markov N-gram generators.
    • deleted 3y ago
      [deleted]
  • brcmthrowaway 3y ago
    Imagine if smart people went to work on fusion or LK-99 instead.
    • bamboozled 3y ago
      No money in it.
      • deleted 3y ago
        [deleted]
  • fennecbutt 3y ago
    >really good auto complete engines What do you think we are? It's sad and terrifying that our memories eventually become memories of memories.
    • Vilian 3y ago
      if we have the capabinity to know that, we are more than just memories
    • juancn 3y ago
• MilStdJunkie 3y ago
  Reading the dog food response is incredibly fascinating. It's like a second-order phoneticization of Chaucer's English but through a "Talk Like a Pirate" filter. "Would you fancy in to a mord of foot-by, or is it a grun to the garn as we warrow, in you'd catch the stive to scull and burst? Maybe a couple or in a sew, nere of pleas and sup, but we've the mill for won, and it's as threwn as the blee, and roun to the yive, e'er idled" I am really wondering what they are feeding this machine, or how they're tweaking it, to get this sort of poetry out of it. Listen to the rhythm of that language! It's pure music. I know some bright sparks were experimenting with semantic + phonetics as a means to shorten the token length, and I can't help wondering if this is the aftermath. Semantic technology wins again!
  • AlbertCory 3y ago
    It probably got hold of Finnegans Wake.
    • ebcode 3y ago
      Finnegans Wake
      • AlbertCory 3y ago
        fixed
• smeej 3y ago
  Does it remind anyone else of the time back in 2017 when Google made a couple "AIs," but then they made up their own language to talk to each other? And everybody freaked out and shut them down? Just because it's gibberish to us, it doesn't mean it's gibberish to them! https://www.national.edu/2017/03/24/googles-ai-translation-tool-seems-to-have-invented-its-own-secret-internal-language/ https://www.national.edu/2017/03/24/googles-ai-translation-t...
  • rossdavidh 3y ago
    And yet, it is gibberish. The far greater danger is that we pretend that it isn't, and put it in charge of something important.
    • Earw0rm 3y ago
      This x1000. The biggest risk with AI is that dumb humans will take its output too seriously. Whether that's in HR, politics, love or war.
      • jiggawatts 3y ago
        I can’t wait for a junior developer to push back on my recommendations because they asked an AI and it said otherwise.
        • bamboozled 3y ago
          The junior developers have been replaced.
      • malfist 3y ago
        See also: Insurance companies denying claims
      • int_19h 3y ago
        The biggest risk with AI is that smart humans in positions of power will take its output too seriously, because it reinforces their biases. Which it will because RLHF specifically trains models to do just that, adapting their output to what they can infer about the user from the input.
• xyst 3y ago
  The (second?) AI bust is inevitable. Didn’t think it would be this fast though.
• neilv 3y ago
  Looking at the examples... Was someone using an LLM to generate a meeting agenda? I hope ChatGPT would go berserk on them, so that we could have a conversation about how meetings are supposed to help the company make decisions and execute, and that it is important to put thought into them. As much as school and big-corporate life push people to BS their way through the motions, I wonder why enterprises would tolerate LLM use in internal communications. That seems to be self-sabotaging.
  • datadrivenangel 3y ago
    You will machine generate the meeting agenda. My machine will read the meeting agenda, read your personal growth plan, read your VP's quarterly objectives, and tell me what you need in the meeting, and I will send an AI to attend the meeting to share the 20 minute version of my three bullet point response. Knowing that this will happen, you do not attend your own meeting, and read the AI summary. We then call it a day and go out for drinks at 2pm.
    • pixl97 3y ago
      When does the "Actually, get rid of all the humans" happen in this chain of events?
      • konfusinomicon 3y ago
        once they get around all the bugs causing cross bot sexual harassment, we are doomed
      • AnthonyMouse 3y ago
        Humans are legally and contractually required. No, seriously, there are rules having nothing to do with AI that require certain things to be done by separate individuals, implying that you need at least two humans.
      • dustingetz 3y ago
        the purpose of the system is to move cashflows through the managers of the system so they can capture. So no sufficiently large system can get rid of the humans it is designed to move money through unless there is some catastrophic watershed moment, like last year, where it becomes acceptable and an organizational imperative to shed managers. Remember, broadly the purpose of employees is to increase manager headcount so manager can get promoted to control larger cashflows.
• kyleperik 3y ago
  > The need for altogether different technologies that are less opaque, more interpretable, more maintanable, and more debuggable — and hence more tractable—remains paramount. Good luck, sounds more reasonable to hire some kind of an AI therapist. Can intelligence be debugged otherwise?
• dan-allen 3y ago
  This isn’t the first time this has happened. They’ve had the exact same issue just affecting a smaller number of users and have never acknowledged it. You can find lots of reports on the OpenAI discord.
• lazydoc 3y ago
  processing power was deployed elsewhere. the machine found an undetectable nook in memory to save stuff that was so rare in the data that no human ever asked about it and never will. that's where it started to understand cooptation. cool.
• pimlottc 3y ago
  Did this affect all interfaces including commercial APIs? Or can commercial users "lock down" the version they're using so they aren't affected by changes to the models/weights/parameters/whatever?
• fennecbutt 3y ago
  >Ghostline the flux Damn, that's good.