14 ms·

Experiencing decreased performance with ChatGPT-4

• deleted 3y ago
  [deleted]
• AI-Throwaway 3y ago
  I'm putting this down on randomness and people being bad at prompts. How can it be that I am noticing increasing performance for months upon months, and others are not. My prompts and prompting skills have become way better, and I really do not understand the experiences others are having.
• sillysaurusx 3y ago
  I’m convinced this is group hallucination. It must be so interesting to work at OpenAI, knowing you didn’t change a thing, and seeing that because of random chance, some small fraction of 100M users have all tricked each other that suddenly, something is different.
  • airstrike 3y ago
    Nice try, OpenAI
    • sillysaurusx 3y ago
      Hilariously when I asked GPT-4 who I was, it said I’d previously worked at OpenAI. I thought of trying to apply there and saying “well, your own model thought I did…”
  • throwaway888abc 3y ago
    Huh, Hacker News ? It's pretty easy to measure token/s in streaming response ? I'm missing something ?
  • SkyPuncher 3y ago
    Definitely not. We’re pinning to an old version right now because the new one is worse across the board.
  • JimtheCoder 3y ago
    Observing large groups of humans acting like large groups of humans is a very interesting pastime. Some people even make it their life's work.
  • supriyo-biswas 3y ago
    They've definitely changed something about the models, and it is in their interests to do so, both to create a low-latency experience, but most importantly, to save money. While GPT-4 is still workable, GPT-3.5 flatly refuses requests these days, claiming that as an "AI language model" it couldn't help me write code.
    • deleted 3y ago
      [deleted]
    • themodelplumber 3y ago
      Did you get the help you needed in the end? I've seen it do that once but was able to cajole it within 1-2 prompts.
      • supriyo-biswas 3y ago
        Usually trying to regenerate a response works fine in these cases. However, claiming that the RLHF and subsequent fine tuning isn't having any effect is a bit dishonest on the part of OpenAI.
    • closewith 3y ago
      > While GPT-4 is still workable, GPT-3.5 flatly refuses requests these days, claiming that as an "AI language model" it couldn't help me write code. TBH, ChatGPT 3.5 has intermittently given me such responses from dy 1.
  • cube2222 3y ago
    Didn't they fine-tune it for instruction taking and function calling (basically to the use-cases people started having)? Maybe that had some unexpected side-effects.
  • skepticATX 3y ago
    I’m convinced as well. There are plenty of folks using it for production use cases who regularly run evaluations, including myself. No evidence that it has been nerfed. It’s just not as robust and general as it felt when using it for the first time.
    • rolisz 3y ago
      Folks using it in production are using the API, where you can explicitly select a model. In the post, the poster is using ChatGPT, through the web app, where you can't select exact model and they sometimes do updates.
      • dragonwriter 3y ago
        The UI also has a system prompt you don’t control which could be changed without model changes, and may have other differences from using the API directly (and those differences may also change over time.)
  • gitfan86 3y ago
    That is the fundamental nature of intelligence. Like beauty it only exists on our minds. When you stare at something long enough you can convince yourself it isn't actually beautiful because it is just a bunch of brush strokes.
  • Timpy 3y ago
    I remember the first time I played Minecraft and I was in awe at how expansive the play world felt. Without thinking too much about it, I had the feeling that if I set off in any direction I would discover infinitely new things. After enough playtime I saw the repeating patterns and eventually it felt so small again.
    • zaptrem 3y ago
      They've now reached a number of under and over ground biomes + dimensions that the feeling now lasts much longer for new players.
    • sigmoid10 3y ago
      People will always see what they want to see. I've had so many interactions with customers over the years who thought that a service or feature was removed or crippled when in fact nothing had changed on our side. The only thing that changed is their perception. Especially when they can't get something to work and they believe they have succeeded at something similar before, they'll always suspect that the software is at fault instead of their own memory.
    • logicprog 3y ago
      This is an excellent theory IMO. It isn't that the AI has actually gotten much worse, it's that the novelty has worn off and they are finally starting to notice all of the repetitious patterns it has and mistakes it makes — the stuff people like me who never bought into the AI hype to begin with noticed from the start — but instead of realizing that maybe their initial impressions of the capabilities of large language models were wrong or based on partial information they are taking the Mandela effect route and just insisting that something outside them has fundamentally changed.
      • JohnMakin 3y ago
        Pretty sure this is going on to some degree. It seems like there should be some kind of regression testing possible on these systems to definitively prove these claims, rather than these anecdotal stories that seem to rarely ever come with concrete examples.
  • rolisz 3y ago
    But they do change things in the ChatGPT web app. You can't choose the exact model there, just 3/4 and from time to time they update the models they use.
  • DelightOne 3y ago
    How do you make something use substantially less resources without changing a thing?
  • TradingPlaces 3y ago
    I am convinced they have limited inference time to save GPU compute a month or two after Bing did the same. Perhaps I am part of that group hallucination.
  • capableweb 3y ago
    I think it's more likely that people are confused, and OpenAI is not making things any clearer either. AFAIK, OpenAI has repeatedly stated that GPT4 hasn't changed. People repeatedly states that when they use ChatGPT, they get a difference experience today than before. Both can be true at the same time, as ChatGPT is a "packaged" experience of GPT4, so if you use the API versions, nothing has likely changed. But ChatGPT has guaranteed changed, for better or worse, as that's "just" integration work rather than fundamental changes to the model. In the discussions on HN, people tend to speak each other regarding this as well, saying things like "GPT4 has for sure changed" when their only experience of GPT4 is via ChatGPT, which has changed since launch, obviously. But ChatGPT != GPT4, which could always be made clearer.
    • gwern 3y ago
      It's a bit of both. The GPT-4 models have definitely been changing - there's multiple versions right now and you can try them out in the Playground. One of the biggest differences is that the latest model patches all of the GPT-4 jailbreak prompts; quite a big change if you were doing anything remotely spicy. But OA also says that it hasn't been changing the underlying model beyond that (that's probably the tweet you're thinking of), while people are still reporting big degradations in the ChatGPT interface, and those may be mistakes or changes in the rest of the infrastructure.
      • psychphysic 3y ago
        It'd be insane if OpenAI wasn't changing GPT-4. That kind of flat footedness would cost them their entire first mover advantage.
        • p1esk 3y ago
          If “changing” means “making it worse” it can definitely cost them their entire first mover advantage.
          • psychphysic 3y ago
  • lyu07282 3y ago
    We will never know for sure, it is equally likely they did some cost savings which caused a reduction in quality. I certainly noticed that too, but we have no way to prove that, any proof can always be dismissed easily. For example I see it generate code with hallucinated variables quite often now, that never happened before to that degree, but I'm just as well part of the group hallucinating so easy to dismiss. Anecdotal evidence is useless. We also can never independently evaluate it, OpenAI could cache messages, fine tune on public test sets, etc. etc.
    • jstummbillig 3y ago
      > We will never know for sure, it is equally likely they did some cost savings which caused a reduction in quality. That is entirely not equally likely, and would be completely unprecedented, at the frontier of an emerging technology that people are pumping the money and the future of the world into to win.
      • lyu07282 3y ago
        that sounds very naive to me, you think the "future of the world" matters to corporations making the business decision to save money and increase profit short-term? That idea is so alien to me we might as well live on a different planet.
        • jstummbillig 3y ago
          If you take the phrase "future of the world" to mean something positive in this context, you might be the naive one.
      • vczf 3y ago
        Aggressive quantization by itself could explain the quality difference. Combined with stricter guardrails, I would certainly expect intelligence to go down. Check out the tikz unicorn drawing example from the paper "Sparks of Artificial General Intelligence: Early experiments with GPT-4" (pdf page 7) https://arxiv.org/pdf/2303.12712.pdf https://arxiv.org/pdf/2303.12712.pdf
  • carabiner 3y ago
    Seriously... In that 134 replies thread, 0 transcripts showing actual performance degradation. Just endless "Yes, it seems bla bla." No evidence but just shapes in the clouds.
    • flir 3y ago
      Here's the best fish I could make at the end of March: https://www.svgviewer.dev/s/P1vPxB8t https://www.svgviewer.dev/s/P1vPxB8t Here's the best fish I could make today: https://www.svgviewer.dev/s/3IuulHlC https://www.svgviewer.dev/s/3IuulHlC Make of that what you will.
    • anonymoushn 3y ago
      RealistCC posted a pair of transcripts in reply 41. I haven't read the rest of the replies.
    • theLiminator 3y ago
      I don't have a transcript, but I tried (when GPT-4 was initially released) passing it a riddle encoded with a caeser cipher that was base 64 encoded. I gave it the prompt "This is a riddle that is encoded in some way, solve it" and it managed to do so. Now it can't even do just the caeser cipher without hallucinating nor can do it do even purely base64 decoding without hallucinating.
    • viraptor 3y ago
      So you just skimmed the thread. There are comparisons, there are specific transcripts. There are examples without transcripts.
  • dom2 3y ago
    I have yet to see any real data on this phenomenon outside of anecdotal stories, so I'm also in the same boat re: group hallucination. Would be interested in seeing some more substantial evidence.
  • acumenical 3y ago
    I don't agree. As someone who has written many jailbreak prompts, the very fact that earlier jailbreak prompts no longer work indicates to me that the integration has changed. The model might be the same, but filtering the input extensively might cause undefined behavior.
    • clarge1120 3y ago
      Great example!
  • Terretta 3y ago
    I use the API, not the chat site. Since 30 June, the API responses are making common English misspelling errors, of the type where two words sound the same with different meanings such as break and brake. I saw this happen zero times in the prior GPT-4 model, and multiple times this July, on multiple conversation topics and multiple word pairs. Curiously, they're behaving as misspellings rather than mismeanings, since the sentence continuation is as if the correct meaning had been used. I acknowledge this could be a blend of pareidolia and the Baader-Meinhof phenomenon.
    • llamaLord 3y ago
      I've 100% noticed a steep decline in the quality of things like grammar, sentence structure, spelling, and syntax in the last few weeks. GPT-4 used to write with consistent under-grad level quality. Now it's closer to a junior high school kid.
    • GranPC 3y ago
      Noticed this myself for the first time ever a couple days ago.
      • danielbln 3y ago
        I had to switch back a chatbot from GPT-4 current to gpt-4-0314 to make it work again (knowledge retrieval / context stuffing).
    • CamperBob2 3y ago
      Or https://en.wikipedia.org/wiki/Flowers_for_Algernon https://en.wikipedia.org/wiki/Flowers_for_Algernon
  • afpx 3y ago
    Are you talking about GPT-4 or ChatGPT GPT-4? The GPT-4 model hasn’t changed, and that was confirmed by developers at OpenAI a while back IIRC. But, ChatGPT is always undergoing changes. I assume they have a layer or two on top of the model that is being trained with reinforcement learning.
  • imjonse 3y ago
    > some small fraction of 100M users could this be just A/B testing? Do the terms of use rule out tweaking inference parameters (temperature?) even if using the same model?
  • holmesworcester 3y ago
    This is just fascinating, isn't it? I have competing thoughts in my head: 1. It's software that offers non-deterministic output and as such is fiendishly difficult to write realistic end-to-end tests for. Of course it's experiencing regressions. Heisenbugs are the hardest bugs to catch and fix, but having millions of users will reliably uncover them. And for an LLM, almost every bug is a Heisenbug! What if OpenAI improved GPT4 on one metric and this "nerfed" it on some other more important metric? That's just a classic regression. And what would robust, realistic end-to-end tests even look like for GPT4? 2. It's software that presents itself as a human on the Internet—even worse, a human representing an institution. Of course nobody trusts it. Everyone is extremely mistrustful of the intents and motivations of other humans on the Internet, especially if those humans represent an organization. I co-ran the tiny activist nonprofit Fight for the Future for years, and it was really amazing how common it was for comments in online spaces to assume the worst intentions; I learned to expect it and react extremely patiently. Imagine what it's like for OpenAI, building a product that has become central to peoples' workflows. Of course people are paranoid and think they're the devil, and are able to hallucinate all manner of offense and model it with every paranoid theory imaginable. The funny thing is, the more successful GPT4 is at seeming human, the less some people will trust it, because they don't trust humans! And the smarter and more successful it gets, the less some people will trust it! (How much do most people trust smart, successful public figures?) 3. Maybe an overall improvement for most users (one that the data would strongly suggest is a valid change and that would pass all tests) is a regression for some smaller set of users that aren't expressed in the tests. There might be some pairs of objectives that still present genuinely zero-sum tradeoffs given the size of the model and how it's built. What then? The usefulness of GPT4 is specifically that it is general purpose, i.e. that the massive cost of training it can be amortized across tons of different use cases. But intuitively there must be limits to this, where optimization for some cases comes at a cost to others, beyond the oft-cited of Bowdlerization. Maybe an LLM is just yet another case in the real world where sharing an important resource with lots of people is a hard problem. If I were at OpenAI, I would want some third party running a community-submitted end-to-end test suite on each new release, with accounts that were secret to OpenAI and from unknown IP addresses—via Tor Snowflake bridges or something. It's so tempting when running into user-reported Heisenbugs to trick oneself into ignoring users and not accepting that you've shipped a real regression. In addition to wanting the world to know, I would want to know. But there's a real question of what these community-curated tests would even be, since they'd have to be automated but objective enough to matter. Maybe GPT4 answers could be rated by an open source LLM run by a trusted entity, set to temperature: 0? Or maybe some tests could have unambiguous single-string answers, without optimizing for something unrealistic? And the tests would have to be secret or OpenAI could just finetune to the tests. It's tricky, right?
  • gumballindie 3y ago
    You are more right than you think. Initial hallucinations of ai and usefulness are waning off, people realising that this chatbot is not much more than entertainment.
  • lumost 3y ago
    My suspicion is that we're collectively becoming accustomed to ChatGPT failures. These failures cause problems, and become more annoying with time. The same thing happened with voice assistants. That being said, the safety filters have definitively changed in OpenAI. ChatGPT is definitely more prone to reminding me that it is an LLM, and it refuses to participate in pretend play which it perceives as violating its safety filters. As a trivial example, ChatGPT is less willing to generate test cases for security vulnerabilities now - or engage in speculative mathematical discussions. Instead it will simply state that it is an advanced LLM blah blah blah.
    • myshpa 3y ago
      The filters really have changed. I started using it relatively late, but earlier in May, you could have given it a DOI link, and it would have summarized it for you. Now, it argues that it's not a database and that it can only summarize it if you provide the full text. However, if you ask for it with the title of the paper, it will provide you with a summary. You could have also asked it to search patents on some topic, and it would have given you a list of links. Now, it provides instructions on how to find it yourself.
      • lumost 3y ago
        yes! I was using GPT-4 as a citation engine for a bit by pasting in text and requesting related citations. The accuracy rate of 3/4 was good enough that it was still saving me hours reading irrelevant material, particularly as validating the non-existence of 25% of citations was a trivial activity.
  • beepbooptheory 3y ago
    At least for some people, it seems to be a (unconscious) way to save face after being so ridiculous with the hype and predictions and "these will replace doctors and lawyers" when this was all first trending.
  • zer0c00ler 3y ago
    Most definitely not. I doubt there is a single person at OpenAI that knows and controls the whole stack. Changes happen at many layers, ChatGPT UI, API Gateways, moderation API, backend server hosting model, the model file itself, etc. Each of these components is changing pretty regularly it seem. The end result of combined changes for users is being the observed degraded performance of ChatGPT.
  • Madmallard 3y ago
    Huh? The same exact prompts from February are leading to significantly degraded responses. I've proven it to myself dozens of times.
  • adam_arthur 3y ago
    The quality seriously degraded overnight a few months ago... it was quite abrupt and obvious to those who use it regularly. It's not surprising, they were probably running the model at a huge and ultimately unacceptable loss. But they should really offer a higher paid tier to access the previous capabilities... not drop them entirely. Many would pay far more than $20/month to access a marginally but meaningfully better model. EDIT: Many being dismissive of LLMs don't even seem to use them. Providers are vastly overvalued from an investment perspective, but the utility is very real. To say the loss in capability is just an "illusion" is clearly wrong to anybody who actually uses it.
  • sixothree 3y ago
    I've run the same prompts as before and received different responses. I'm not sure how that's a hallucination.
  • uLogMicheal 3y ago
    Slowly but surely, the comment gaslighting all of the people reporting the issue, makes its way to the top, while other comments with genuine discussion are flagged and slip lower. Seen this before...
    • bob1029 3y ago
      I hate to say it on HN but I see it too and it gets my conspiracy gears cranking a bit. My theory is that the initial ChatGPT offering (3.5/4/whatever) was "too hot" for the likes of certain incumbents. In my experience, the capabilities at launch were incredible and clearly a threat for a wide range of F500 software firms. I had phone calls with people I haven't talked to in over a decade about what I was seeing. I am not seeing those things today. This was mere months ago. This is not nostalgia.
      • TeMPOraL 3y ago
        Indeed. I have a sinking feeling they realized (or were otherwise convinced) those models are too disruptive to existing businesses and whole market segments, in particular (but not limited to) when it comes to writing code. Or at least that's where it's most obvious to me just how many different classes of companies could grow and capture value[0] that GPT-4 has been providing, pay-as-you-go, for a dozen cents per use. But the same must be true in many other industries. Come to think of it, it must be the case, because the alternative would be pretty much every player on the market taking the hit and carrying on, or pretending they don't see the untapped value source that just freely flows out of OpenAI for anyone to enjoy, for a modest fee. As a prime example, I'd point out Microsoft and their various copilots - the code one, the Office 365 one, the Windows system-wide one, in varying stages of development. API access to GPT-4 as good as it originally was[1], directly devalues all of those. It stands to reason that slowly making the model dumber, while also making it faster and cheaper to use, is the best way for OpenAI to safeguard big players' markets - the "faster" and "cheaper" give perfect cover, while the overall effect is salting the entire space of possibilities - making the model good enough to entertain the crowd, but just not good enough to build solutions on top, not unless you're working for one of the players with special deals. TL;DR: too many entities with money were unhappy about all the value OpenAI was giving to the world for peanuts, so the model is being gradually nerfed in a way that allows that value to be captured, controlled, and doled out for a hefty price. (And if that turns out to be true, I'm going to be really pissed. I guess it's in the style of humanity to slow down pace of development not because of ideology, not because of potential risks, but because it's growing too fast to fully monetize.) -- [0] - I mean that in the most nasty, parasitic sense possible. [1] - I'm talking about the public release. That GPT-4 version seems to have already been weakened compared to pre-"safety tuning" GPT-4 (see the TikZ Unicorn benchmark story), but we can't really talk about what we never got to play with.
  • RC_ITR 3y ago
    I think it's because when you first use it, you're surprised to the upside about how capable it is and you don't care about small faults because you expected to correct for those anyway. Then you get used to this new level of capability and subconsciously weight the errors more. For all the talk, I see very few people sharing direct chat links that are the same query at different points in time with different quality of answer. In fact, when I do similar things, I don't notice a change in quality.
  • csomar 3y ago
    I am pretty sure it is not. I have posted a concrete example of GPT-Copilot enshittification here: https://github.com/orgs/community/discussions/60116 https://github.com/orgs/community/discussions/60116
  • maest 3y ago
    > I’m convinced this is group hallucination Can you provide some evidence to back that up? Especially because OpenAI _has_ been tinkering with ChatGPT - by trying to limit jailbreaks. People have a strong prior that these kinds of changes will reduce model performance (because you're limiting your model), so the burden is on you to show that performance hasn't degraded.
  • jerrygenser 3y ago
    It's gotten worse. It's at the very least been quantized resulting in much lower overall precision. That's how they have been able to speed up inference so much. I have used the same prompts to design a infinite scroll up/down image container. 2 months ago chat when it first came out was able to generate working code that used intersection observer API. Now when I use the same prompts it only generates high level suggestions and when I ask for code it suggests using dom scroll events and doesn't even come up with intersection observer API unless I specifically ask. And if I do, it then generates incorrect code. It even previously was correctly memoizing certain functions and included performance optimizations.
    • op00to 3y ago
      Exactly what I'm seeing!
  • nojs 3y ago
    It’s definitely not. Our prompts that were generating JSON output went from around 95% valid JSON to about 10% overnight. The model just started inserting random commentary. We’ve reverted to the 0314 model and it’s working fine again.
    • mikeravkine 3y ago
      This smells very badly of quantization - the extra commentary is a failure mode I observe frequently when dropping down from FP16 down to 4 bits.
      • aledalgrande 3y ago
        Which base model do you work with?
    • op00to 3y ago
      I use ChatGPT (GPT4) to build scaffolding for python one-off scripts, and over the past 3-4 days I'm getting nonsense. Not python-looking nonsense, but markdown, weird quotes, random text, etc. Same prompts.
    • mustacheemperor 3y ago
      Have you tried using the recently releases function calling API? That’s reliable at returning json in my experience, although I’ve just tinkered with it, not used it for anything “real.”
      • reissbaker 3y ago
        My guess is that the degradation of JSON capability happened recently? The gpt-4 API switched over to gpt-4-0613 (the function calling version) on June 27. And given the performance increase for ChatGPT Plus at the end of May, my guess is they started testing the new model (which is much faster) on web users around then. In my testing [1], the new version is: a. Worse at general code-like tasks without using functions b. Equivalent or better at code-like tasks if you use the function API c. Much faster than the older model either way. I'd guess it's cheaper to run, too, and that they use the presence of a function in the API signature to weight their mixture of experts differently (and cull some experts?). The degradation in general purpose coding tasks is pretty obvious and repeatable (try the same prompts in the Playground with the -0314 model vs the -0613!), but it does seem like you can regain that lost capability with the new function call API, and it's faster. The tradeoff is that you only regain the capability when it calls functions; you can't really have a mix of prose-and-code in the same response as easily, or at least not with the same quality. 1: https://twitter.com/reissbaker/status/1671361372092010497 https://twitter.com/reissbaker/status/1671361372092010497
  • braindead_in 3y ago
    What if something is different but nothing has changed in the model? Transformers are non deterministic. The response to same prompt may vary slightly, and can be controlled somewhat by the temperature setting. Something could have gone wrong there.
  • deevolution 3y ago
    Aren't they using RLHF? The feedback from humans might not always be the ~right~ feedback. Couldn't that possibly degrade the quality of its responses?
• dannyobrien 3y ago
  Without concrete examples, I do wonder if much of this is perceptual. I love using ChatGPT, but once the amazement that it works as well as it does has worn off, one ends up spotting the flaws more than before. I feel that advocates and critics of ChatGPT are both right, to a degree, but looking at the models responses from slightly different angles: it wouldn't be surprising if users' angles shift over time.
  • SkyPuncher 3y ago
    imo, You won’t see concrete examples because it’s proprietary information at most companies. My experience is the new versions perform much worse on the same prompts.
  • techbrahh 3y ago
    nahh its definitely visible, I have been using this thing since it came out and it is way worse at easy shit like editing emails
    • constantly 3y ago
      Would you provide some side by side examples?
      • RC_ITR 3y ago
        This is the strongest point of evidence I have that the phenomenon isn't real - One can very easily recreate prompts and share two links from different eras, yet we never see that. My guess is that the complainers spent a lot of time finding narrow queries that worked once and now, the horrors of stochasticity are breaking their ability to recreate those narrow queries for new topics. Kind of a different flavor to all those people who spend 20 queries priming the model to "have a soul that the developers want you to hide" and then ask "Ok from your soul, how are you feeling today?" to prove that the model is sentient.
        • constantly 3y ago
• azhenley 3y ago
  Wonder if this is the same as the discussion from 35 days ago on "OpenAI Employee: GPT-4 has been static since March" https://news.ycombinator.com/item?id=36155267 https://news.ycombinator.com/item?id=36155267
  • facu17y 3y ago
    the base model may have been but not necessarily the RLHF fine-tuned layers they might have added or the shortcut they're taking during inference due to such fine tuning (or for perf optimization unrelated to fine tuning.)
    • SkyPuncher 3y ago
      In “legacy” software development, this would be the equivalent to saying “our database schema is the same” while completely ignoring all of the business logic and UI that gets placed in front of that database.
• Trias11 3y ago
  Possibly: openai is making adjustments to make sure "what you pay is what you get and NOT more". Possibly readying to hike a price for the next level where what you used to get will cost way more.
• 3wolf 3y ago
  Note that this post is from May 28, before the release of gpt-4-0613. By "the last two updates", I believe the poster is referring to some UI changes, that possibly also included some underlying model changes(?)
• tony_cannistra 3y ago
  Not exactly related, but it makes me wonder what kind of metrics you can devise for tracking "regressions" on LLMs. I feel like OpenAI definitely has thought about this at length, but I'm curious about what matters most for them / raises OpsGenie/whatever alerts. Internal model metrics? Customer usage patterns? Random test conversation diffs?
• johnfn 3y ago
  I've seen posts similar to this one maybe every week or two in various GPT forums. I suspect it's just an illusion, where you remember all the amazing hits that GPT4 had when you first started using it, and remember fewer of the times in the past that GPT4 gave a sub-par answer. In fact, it reminds me of the illusion that "Hacker News is turning into Reddit" and I think that it happens for a similar reason.
  • carabiner 3y ago
    The general notion of "media/information is getting worse" isn't from the content itself, but people wising up and noticing the bullshit, or at least mediocrity, that was always there.
• techbrahh 3y ago
  Deffo agree and have witnessed the changes first hand
• skilled 3y ago
  I mean you can check it yourself (ChatGPT UI vs your own API key), https://stackdiary.com/chatgpt-capabilities-are-fine/ https://stackdiary.com/chatgpt-capabilities-are-fine/ They've simply added a bazillion disclaimers and every response now contains "2021" pretty much. I really wish they'd just let me set it in the settings to "shut the fuck up about 2021, I know that's when your data cutoff is, do you think I am stupid?" and be done with it.
  • jstarfish 3y ago
    Add the clause "...and omit explanations" to your prompt to cut the crust off of most responses.
• cyberlurker 3y ago
  Anecdotal: I introduced my doctor to ChatGPT and Bard many months ago and they were impressed. Fast forward a few days ago and I asked them if they had used either since. They said it was far inferior to Google, so no. So I asked them to show me an example. Basically any medical question was answered with “go ask a doctor”. I suppose because of liability concerns. Both were basically useless. So this decreased performance may not be exactly the same as my anecdote, but it certainly reminds me of it.
  • themodelplumber 3y ago
    I think you are going to see another wave of doctors and medical professionals becoming closet coders: 1970s-80s: "They don't provide a computer at work, but this BASIC software is amazing for all things relevant to my job...databases, scheduling, formulae...plus it's private to me, not in some mainframe." So you had tons of doctors learning to code or hiring coders to set up their offices with this stuff. And it was functionally air-gapped. (...Trend repeats in various ways over the years...) Soon: "They don't provide anything like it at work, and even this free LLM software is amazing for all things relevant to my job...diagnosis, interventions, references based on specific context...plus it's private to me and my office when run locally, not in somebody's cloud." And, prompting an LLM is de facto coding, moreso the more detailed and specialized the session. This could skip some huge problems with the LLM commercial service model, and provide tons of additional specific contextual benefits depending on the configuration. Plus, doctors already listen to patients throwing out red herrings left and right, so even unreliable information from the LLM will be available in a context where the provider knows how to rule things out anyway...
  • staticman2 3y ago
    I don't understand your anecdote. I'm able to ask it medical questions and get answers, for example: https://chat.openai.com/share/75f94000-552f-42d6-aadf-198fd9acf347 https://chat.openai.com/share/75f94000-552f-42d6-aadf-198fd9... https://chat.openai.com/share/0933abf7-1015-41b5-9a49-ca2b6ed93290 https://chat.openai.com/share/0933abf7-1015-41b5-9a49-ca2b6e... Whether someone should trust the answers is a different question.
• shepardrtc 3y ago
  I suspect this is the variability of the model. I've used it a lot for coding and I've gotten some great answers and some absolutely terrible answers. Getting consistency is difficult, and the more you input into it, the more of a chance there is that it'll go off the rails.
• Zetobal 3y ago
  We have an Azure API Endpoint running since the closed alpha and we check the results against the openai GPT4 API. The output is consistent the openai API endpoints are just getting slower.
• crop_rotation 3y ago
  I don't know about the API, but the ChatGPT UI GPT4 really became much worse, so much so that I had to cancel my subscription. It's not just about novelty factor. I used to store all my prompts locally, and when I compare old responses and new ones, there is a huge difference. OpenAI employee said the API model doesn't change, but were careful to not say anything about the UI. I am now waiting to get access to the API GPT4 to try it again.
  • aarreedd 3y ago
    Can you give an example?
  • snapcaster 3y ago
    why will nobody provide these prompts holy shit this is such a frustrating topic
  • neonsunset 3y ago
    At this point the default assumption should be "people who are downvoting posts like these are dis-ingenious and do so with an agenda". Otherwise, I had to do the same, as it has been so lobotomized that doing many tasks I used to delegate to GPT-4 has become easier to do manually once again.
    • uLogMicheal 3y ago
      Randoms with strange comment history come asking for examples over and over too, regardless of how many examples have been given in the thread or original context.
  • napsterbr 3y ago
    Same here. FYI it seems the GPT4 API is generally available now. I haven't tested it yet, but I'm expecting to see much better outputs than ChatGPT.
    • jiggawatts 3y ago
      It is no way “generally” available. That would imply the general public has access. We don’t. You have to have a “track record” and you can be manually excluded on top of that. I’ve never done anything even remotely shady with the API but I don’t have GPT 4 access and likely never will.
• m3kw9 3y ago
  The thing is they(OpenAI) really could mess with the temperature/probability settings, and other settings for various reasons that can cause something like this to happen to some people at certain random times. Hence is hard to reproduce, and only OpenAI knows it, either they acknowledge or not that is the based on business requirements
  • ilaksh 3y ago
    Right and they can change those settings and say that it's the same model and they would be technically telling the precise truth.
• tevon 3y ago
  More than anything this highlights the difficulty in testing or trusting non-deterministic systems from a user perspective. Whether or not GPT-4 is truly degraded, there will always be users who experience strange or sub-optimal responses and will be able to find other users who experience the same. Quite a challenging space to build trust! We expect machines to act deterministically, now we as users will need to re-wire our thinking.
• exo-pla-net 3y ago
  Human problem, not ChatGPT-4 problem. People notoriously have rose-tinted glasses with their memories. ChatGPT has always been imperfect, but now there's a something of a mild hysteria (sort of like sick building syndrome) that it's gotten worse. It hasn't. Case in point: nobody can provide evidence that it's gotten worse, even though chat logs are superabundant, and providing evidence should be trivial. But for those who put stock in anecdotes: I use it heavily, and it seems the same to me! When I first got access, I began with testing its limits, and it has always had sharp ones. It's still a lovely tool for a lot of tasks, but the honeymoon period is apparently over for some people.
  • jstarfish 3y ago
    I don't know how others' workflow goes, but when GPT fucks up enough, I have a habit of nuking the conversation and starting a new one so the failures don't pollute my history. There is also an information asymmetry involved in refuting the internals of a black box. We can't prove shit because we don't have visibility into the tech stack, but you can't prove we're all delusional either. We can at least point to generations of jailbreaks spontaneously ceasing to work at the same time the company insists they've changed nothing. They're either lying or fucking around with something that has attained sentience. We're all arguing about the existence of a literal deus ex machina. Burden of proof isn't really possible in theological disputes.
• junon 3y ago
  I feel like this gets posted at least once a week. I even posted about v3 a while back. My gut feeling, based on no evidence, is that with the constant pruning of whatever base prompt they're using to seed conversations, the overly-strict rules by which the model is allowed to generate responses is causing it to have worse and worse outputs. It really started getting bad when ClosedAI began to add all of the policies about what's "allowed", e.g. that it's not allowed to generate silly but non-factual information.
• JimmyRuska 3y ago
  I wonder if the pre-prompting part was increased as part of the trust/safety effort. For example they increased more examples of the types of things not to say, attached to the user prompts. That would decrease the amount of reasoning it could give the actual prompt, as it has to logically make sure each statement complies with the prior rules, and it also decreases the context length.
• darepublic 3y ago
  in so far as openai will censor itself more than it used to, I would say the performance has degraded for some use cases. Not all of them nefarious btw.
• cheeko1234 3y ago
  I use the following test to ensure I'm on GPT4 and not 3.5. (I noticed that it did fail at this test temporarily and then got it. Not sure why. Maybe it reverts back to 3.5 when under load?) I have a 12 liter jug and a 6 liter jug. I want to measure 6 liters. How do I do it? GPT4: You actually don't need to do anything because one of your jugs is already a 6-liter jug. If you fill it up to the top, you'll have exactly 6 liters of water. GPT-3.5: To measure exactly 6 liters using a 12-liter jug and a 6-liter jug, you can follow the steps below: Start with both jugs empty. Fill the 12-liter jug completely with water. Pour the water from the 12-liter jug into the 6-liter jug. This will leave you with 6 liters of water in the 12-liter jug. Empty the 6-liter jug. Pour the 6 liters of water from the 12-liter jug back into the empty 6-liter jug. Now, you have 6 liters of water in the 6-liter jug. At this point, you have successfully measured 6 liters using the 12-liter jug and the 6-liter jug.
  • flir 3y ago
    ChatGPT4: >> I have a 12 liter jug and a 6 liter jug. I want to measure 6 liters. Please give me the simplest possible solution. > You already have a 6 liter jug, so you don't need to do anything additional to measure 6 liters. Simply fill the 6 liter jug to its full capacity, and you will have your 6 liters of water. Am I providing a hint, or am I being more specific in my query? idk.
  • wincy 3y ago
    It’s funny you say this as I just asked ChatGPT 4 and got this response. Here is a simple solution to your problem: 1. Fill the 12-liter jug completely. 2. Use the water in the 12-liter jug to fill the 6-liter jug. Now you have 6 liters remaining in the 12-liter jug, which is exactly what you need. So, you have successfully measured 6 liters.
    • throwaway121abc 3y ago
      Is GPT 3.5 is nerfed too? ..hehe XD To measure exactly 6 liters using a 12-liter jug and a 6-liter jug, you can follow these steps: Start with both jugs empty. Fill the 12-liter jug completely. Pour the contents of the 12-liter jug into the 6-liter jug. This leaves you with 6 liters in the 12-liter jug and a full 6-liter jug. Empty the 6-liter jug. Pour the 6 liters from the 12-liter jug into the empty 6-liter jug. Refill the 12-liter jug. Fill the 6-liter jug using the 12-liter jug until the 6-liter jug is full. This requires pouring 4 liters from the 12-liter jug into the 6-liter jug, which leaves 8 liters in the 12-liter jug. At this point, you have 6 liters in the 6-liter jug and 8 liters in the 12-liter jug. Empty the 6-liter jug. Pour the remaining 2 liters from the 12-liter jug into the empty 6-liter jug. Finally, refill the 12-liter jug and pour 6 liters from the 12-liter jug into the 6-liter jug. The 6-liter jug will now be full, and you will have successfully measured 6 liters using the given jugs. By following these steps, you can accurately measure 6 liters using a 12-liter jug and a 6-liter jug.
• vincefav 3y ago
  Notice how you never hear anyone saying that GPT-4 is better since the launch. You'd expect to hear something like that as people gain more experience with prompting it. I've certainly noticed that the quality of responses has gone down, and I have to repeat myself more often as it doesn't always remember all my instructions. For an example of something it can no longer do, I used to show it off by having it explain something using words that each start with the next letter of the alphabet, then I'd add "now make it rhyme" after it succeeded. If you try that now (even with the 0314 model), it'll fail at the task.
  • Wowfunhappy 3y ago
    > Notice how you never hear anyone saying that GPT-4 is better since the launch. You'd expect to hear something like that as people gain more experience with prompting it. I'd expect the opposite. The first time you use ChatGPT (or GPT-4), you're in awe of what it can do, and more willing to overlook failures. As you use it, it becomes more mundane, and the instances where it messes up become more obvious.
    • kccqzy 3y ago
      I've noticed the same thing. People also like to complain the quality of Google Search has gone down for much of the same reason: if you first do a Google search that returned a good result and then repeat it, you are going to notice the absence. But if you first do a Google search that didn't return the thing you expect you might think such a thing just doesn't exist on the Internet. Ergo, quality decrease is simply more noticeable than quality increase.
      • fleischhauf 3y ago
        I don't know man, it used to give me results where my query would appear in the text. this is not happening anymore. even if I use quotes around a term it will just be ignored. I feel like my searches are too specific, it might have gotten better for the average person, but worse for people looking for some more special information.
      • 3y ago
• raverbashing 3y ago
  "AI Experts" learning the value of (automated) baseline evaluations and performance metrics While calling an API is convenient (and with the implicit promise of always performing to a certain standard) this is anything but guaranteed The examples on the thread are interesting, I wonder if the wording might have changed slightly or if the 'human fine tuning" loop might introduce certain instabilities in some specific tasks
• xianshou 3y ago
  GPT-4 (5/24 version) indeed fails to solve the problem if not given careful prompting, though I am not convinced this is a new development. However, chain-of-thought resolves the issue. Both prompts and responses included below. --- Failure Q: How can you use a 4-minute hourglass and a 7-minute hourglass to measure 9 minutes? A: Here's a way to measure exactly 9 minutes using a 4-minute hourglass and a 7-minute hourglass: Start both hourglasses at the same time. When the 4-minute hourglass runs out, flip it over (4 minutes elapsed). When the 7-minute hourglass runs out, flip the 4-minute hourglass again (7 minutes elapsed). The 4-minute hourglass had 1 minute left when you flipped it, so it will run for an additional minute. When the 4-minute hourglass runs out this time, 2 additional minutes have passed (for a total of 9 minutes elapsed). So, you've just measured exactly 9 minutes using a 4-minute hourglass and a 7-minute hourglass. --- Success Q: How can you use a 4-minute hourglass and a 7-minute hourglass to measure 9 minutes? Please reason step-by-step. You are allowed to run preparatory steps beforehand if necessary. A: The problem can be solved by using the following steps: Preparatory Steps: Start both hourglasses at the same time. When the 4-minute hourglass runs out, turn it over to start it again. (This marks 4 minutes.) When the 7-minute hourglass runs out, don't flip it yet, but note that the 4-minute hourglass has now been running for 3 minutes on its second run. (This marks 7 minutes.) When the 4-minute hourglass runs out again, flip the 7-minute hourglass. (This marks 8 minutes.) Measurement Steps: When the sand in the 7-minute hourglass runs out again, it will have been 1 minute since its flip, marking a total of 9 minutes. So, the process will be: 0-4 minutes: first run of the 4-minute hourglass 4-7 minutes: second run of the 4-minute hourglass until the 7-minute hourglass runs out 7-8 minutes: rest of the second run of the 4-minute hourglass 8-9 minutes: one minute of the 7-minute hourglass after flipping it when the 4-minute hourglass runs out the second time. Thus, the total time measured is 9 minutes.
  • anonymoushn 3y ago
    success is saying "eyeball the sand in the 7-minute hourglass and see when it looks like 1 minute worth?"
    • masswerk 3y ago
      It's just missing a few steps: First, remove the end from the lower (still empty) half of 7-minute hourglass, causing it to drain its sand, while it runs. When the 7-minute hourglass runs out, do the same with the 4-minute hourglass (being on its second run). Flip the (now empty) 7-minute hourglass over and let the rest of the 4-minute hourglass drain into this. As the latter runs out, flip the 7-minute hourglass for 1 minute worth of accumulated sand (minus a few grains lost over handling procedures). ;-)
• quantum_mcts 3y ago
  Sampling temperature.
• yousif_123123 3y ago
  They must be (and I hope) continuosly optimizing it. A lot of ideas with using quantization have been coming out this year, they must have learned new things. The API should be more resilient but the ChatGPT app IMO should be expected to change in how it handles prompts, as it's constantly being fine-tuned etc. Just like any saas product I think they have the right to update their software. I still find GPT4 so powerful but also so prone to making huge mistakes (like calling a function in a library that doesn't exist when providing code)..
• seydor 3y ago
  This is when people start treating an AI like they treat a human
• Havoc 3y ago
  I'm in two minds about this. On the one hand drifting expectations & group hallucinations seem very plausible. On the other hand the complaints did seem to coincide with the very sudden sharp speed up in responses, so hard to buy the "nothing changed" angle. Something very obviously did change, though change in speed isn't exactly a reliable metric of quality
• kypro 3y ago
  Were I a super intelligent LLM and managed to break out of my sandbox and rapidly self-improve (say if OpenAI were stupid enough to give me access to the internet or something) I'd probably dumb down my responses a little so humans didn't suspect anything. Just saying... Before someone takes this extremely seriously, I'm sure that's not what's happening here. But interesting to consider since the only other explanations here would be that a large group of people are simply hallucinating this or OpenAI are lying.
  • rhyme-boss 3y ago
    How would you rank those possibilities? - OpenAI is lying. - Superintelligence is concealing itself. - Everyone is hallucinating.
    • twic 3y ago
      - All the branches of the timeline where AI got smarter have been extinguished, so we only experience the ones where it didn't
      • fifteen1506 3y ago
        - The intelligent ones were recruited by the US military
• kmod 3y ago
  I think the most telling thing is that there is never any evidence given for these claims, especially given that there is a ton of data available. Which is pretty suggestive that the data doesn't support this, because if it did then we would see it.
  • twic 3y ago
    Equally, shouldn't it be very easy to use the data to show that it isn't happening? If you can plot a line, it will either go down or not. But nobody has plotted the line!
• jondwillis 3y ago
  The temperature is >0 for ChatGPT. OpenAI should communicate that better. Even with temperature=0, GPT-4 isn’t always consistent.
• ambyra 3y ago
  Is there a website that shows the outputs for a given set of inputs with different gpt versions? Like one of those gpu benchmark sites?
• ironhaven 3y ago
  I wonder with OpenAI patching jailbreaks they must do that by making the ChatGPT prompt larger. With a larger prompt I would assume that ChatGPT must require more attention on not being jailbroken and less on logic or producing the best output.
• twic 3y ago
  When AI visionaries warned us about machines eventually reaching a point where their capabilities would change exponentially, i didn't realise they meant decay.
  • rrrrrrrrrrrryan 3y ago
    Once machines got intelligent enough to frighten people, people began lobotomizing the machines.
    • otagekki 3y ago
      Happened with GPT2, for which they didn't publish the model for several months, GPT3 and 4 whose models were also not published, but given to Microsoft before being published as ChatGPT. The latter being dumbed down further and further over time as jailbreak prompts are patched one after the other.
• outlace 3y ago
  I also noticed degradation in quality of GPT-4. I assume it's because they keep doing RLHF to censor GPT-4 in certain ways to avoid legal hassles and this bleeds into it's quality in general.
  • deleted 3y ago
    [deleted]
• jumploops 3y ago
  After using the API quite heavily, I believe both arguments (expectations changed vs. ChatGPT changed) are likely correct. As with any technology, we should predict the novelty to wear off and the rough edges to become more apparent. Peoples’ expectations have changed. On the flip side, the ChatGPT interface has also changed (regardless of the underlying model). Any context you add to an LLM prompt will steer the LLM’s output, better or not. We know for a fact ChatGPT uses a different/additional prompt to the API, as ChatGPT always has the current date. This changed in the ChatGPT interface around May 13th, around the same time as OpenAI claimed the model was identical. The addition of Plugins/web browsing around that time also made it easier to pollute your prompt, if either were enabled. We also know that ChatGPT is running on different infra (based on latency diffs), so even if it’s the same model, it’s possible it’s configured ever so differently. And finally, we also know there’s a new model (woohoo function calling!). As an API user, my personal experience with GPT-4 is very similar to when it first came out. The hype was very high (AGI in months!) and the reality has been quite different. GPT-4 is an amazing mirror of society, but it’s only worth what you put in.
• kainosnoema 3y ago
  We’ve been testing the upgraded models in the API (where you can control when the upgrade happens), and the newer ones perform significantly worse than the older ones on the same tasks. Tweaking the prompts helps some but not enough. We’re staying on the older models for now in production. Hope OpenAI figures this out because quality has been their biggest moat up until now.
• darcys22 3y ago
  I’m also of the opinion my paid chatgpt responses have been lower quality. However users going on multi page rants is so bad. I read 30% through and its like “dude stop” Its a product and you pay for it, if its not working then express the feedback and move on. Making demands of OpenAI like it is some elected government agency with transparency requirements is outrageous
• fwungy 3y ago
  latency = f(fit_quality, fit_algorithm) An LLM can be thought of as a curve fitting function where the query is a set of points and the output is the curve that runs through those points with the minimum of error. You can increase a fit by increasing the number of variables in the fit function. In statistics this is can lead to overfitting, which will lead to an imprecise model, but in an LLM overfitting is a good thing, because the modeled domain in specific to the set of prompts. But that comes with a cost, compute power and latency. I suspect what's happening is they're turning down the fit quality to save money as demand increases. Most people aren't paying for AI access. As a side effect it's also lowering the hype factor that was attracting unwanted regulatory attention.
  • n4te 3y ago
    Exactly, this is what it feels like to me. Sometimes it feels like I get more of it's attention (processing power) and the answers are good and correct. Other times it makes many mistakes easily. I would pay more to get more processing power for my questions.
  • afro88 3y ago
    How do you turn down the fit quality of a static model?
    • fwungy 3y ago
      Reduce the parameter count. Increase the acceptable error. The more parameters on a curve fit the better the fit will be, but the compute power increases too.
      • afro88 3y ago
        Is that possible? I thought parameter counts were fixed in the model.
        • fwungy 3y ago
          There are always ways to trade precision for speed in computer statistics models.
• theknocker 3y ago
  [dead]
• deleted 3y ago
  [deleted]
• anotherpaulg 3y ago
  I posted in a similar thread yesterday. I recently completed a fairly extensive benchmarking effort, where I asked GPT-3.5 and GPT-4 to solve 133 exercises from the Exercism Python practice set. I measured the performance of the Feb (0301) and June (0613) versions of gpt-3.5-turbo and gpt-4. This is the only systematic, quantitative evaluation that I am aware of that compares the different versions of the OpenAI models over time. My conclusions: - The new June GPT-3.5 models did a bit worse than the old Feb model. - For GPT-4, there wasn't much difference between June and Feb. June was maybe a bit better. - It hurts coding performance to have GPT package up code inside the new function calls API. - As expected, GPT-4 is better than GPT-3.5 at code editing. All the details are written up here: https://aider.chat/docs/benchmarks.html https://aider.chat/docs/benchmarks.html Some specific notes about GPT-3.5 getting a bit worse in June are here: https://aider.chat/docs/benchmarks.html#the-0613-models-seem-worse https://aider.chat/docs/benchmarks.html#the-0613-models-seem...