16 ms·

OpenAI Employee: GPT-4 has been static since March

• amelius 3y ago
  Instead of complaining, why not show the benchmarks? Like: first it scored 83, now it scores only 42 (or whatever).
  • ShamelessC 3y ago
    the commenters of HN are simply so smart that their hunch clearly holds more weight than scientific rigor.
  • sebzim4500 3y ago
    What benchmarks? We have no way to go back in time and run the old ChatGPT accessble models though benchmarks. And to my knowledge, no one has copy pasted an entire benchmark into the ui in order to run it, they just use the API.
• PaulHoule 3y ago
  People have just fallen out of love with it and are realizing how it really isn't that good. Sorry "prompt engineers" but papers on arXiv show that when you give it fairly sampled problems it struggles to get the right answer more than 70-80% of the time. When you are under its spell you will keep making excuses but when you are looking at it objectively you'll realize the emperor is naked. If you give it very conventional problems it seems to do better than that because it is a mass of biases and shortcuts and of course it will sentence Tyrone to life in prison because it's a running gag that "Tyrone is a thug"... That's how neurotypicals think and no wonder why many of them think ChatGPT is so smart... It mirrors them perfectly.
  • ThrowawayTestr 3y ago
    I used it to help me write an email begging for my job back and it was pretty helpful
    • BulgarianIdiot 3y ago
      Just ignore this fella, GPT-4 is massively useful. But it takes intelligence to get intelligence out of it.
      • visarga 3y ago
        I think this is our path - become good at working with AI, for jobs or if there are no jobs, to directly support ourselves. Supposing we will be able to use tools and AI to build things and make our own means.
        • BulgarianIdiot 3y ago
          This is our path yes. All paths end, eventually, but oh well.
    • deleted 3y ago
      [deleted]
  • BulgarianIdiot 3y ago
    I very much doubt the tweet in question is of someone who loved GPT-4 yesterday, and has "fallen out of love" with it starting today. For someone so dismissive of "neurotypicals" you did the neurotypical thing and drop a hot take before clicking the link.
    • wouldbecouldbe 3y ago
      It's more that the balance in usefulness vs waste of time shifts into the other direction after a few times spending long time getting it right and just doing it yourself anyway.
      • BulgarianIdiot 3y ago
        I've spent 20 years explaining my code to a rubber duck and getting great use out of that duck. No one will ever convince me that it's less useful now when the duck actually has read the Internet and has useful suggestions and ideas back to share. Even if it makes shit up on an occasion.
        • JohnFen 3y ago
          I don't know. The value of rubber-ducking is that you have to break the problem down into very simple terms to explain it. It's the explaining it part that is magic. That value goes away (or is greatly diminished) if the rubber duck responds with anything other than a request for further explanation.
          • BulgarianIdiot 3y ago
            You have to break down a problem to GPT just like you would to a human, and if it misunderstands you or asks you, you'd have to elaborate just the same. It's modeled after us, after all. The rubber duck is a stand-in for a human. GPT also is. But just a vastly better one.
            • JohnFen 3y ago
  • mrbombastic 3y ago
    I don’t know, I think there is certainly a lot of hype but it is still pretty damn good. I used it every day for a wide variety of coding tasks and more often than not it is correct. As in compiles, produces correct output for inputs, and mostly writes reasonable code. Has it made software dev obsolete like some maximalists said it would? definitely not, but it seems equally hyperbolic to say the emperor has no clothes, it is undeniably a useful tool to me.
    • PaulHoule 3y ago
      "More often than not" is compatible with "correct 70% to 80% of the time".
      • swores 3y ago
        Your previous comment was a bit confusingly worded, I, and presumably the commenter you just replied to, read it as "more than 70-80% of the time" it struggles; rather than your intended meaning of it struggling to do better than correct 70-80% of the time.
  • vsareto 3y ago
    >When you are under its spell you will keep making excuses but when you are looking at it objectively you'll realize the emperor is naked. It's not perfect, but you have to admit the emperor is at least wearing a thong. It is the second most intelligent thing on the planet at creating text, even with its flaws. Putting that accomplishment in league with naked emperors is astoundingly biased.
    • PaulHoule 3y ago
      Yeah, maybe it is "wearing a thong" since it's developers were so concerned about social acceptability. I would say though that it is no mean feat for the Emperor to get away with being naked in public, it takes power and the ability to wield it. Similarly ChatGPT has a few competences that add up to people perceiving it is able, one of which is the ability to come up with plausible and satisfying answers whether they are right or wrong (often using shortcuts) and another one is getting people to engage with these answers.
    • michaelt 3y ago
      By this, do you mean 'the second most intelligent thing' after humans?
      • shrimpx 3y ago
        A dog understands body language and subtle cues, participates in social hierarchy and community, has self-reflection, dreams, etc.
    • shrimpx 3y ago
      > you have to admit the emperor is at least wearing a thong And a lot of people are getting turned on by it.
    • krger 3y ago
      >It is the second most intelligent thing on the planet at creating text, even with its flaws. This also puts it in last place.
  • guraf 3y ago
    [flagged]
  • throwuwu 3y ago
    [flagged]
• furyofantares 3y ago
  I think we don't notice our expectations have gone up, and we don't notice that we remember the hits and then expect all hits. We didn't notice the misses at first, because it's what we expected to begin with, and we very strongly noticed the hits because they were unexpected. Now we notice the misses and expect the hits.
  • huevosabio 3y ago
    I think this is the right way to look at it: we are very quick at updating expectations. The first flight is magic, the nth one is a chore.
    • williamcotton 3y ago
      Whenever I’m flying over an ocean on a clear moonlit night it still very much feels like magic!
      • myshpa 3y ago
        And now it's even gonna smell like bacon ... The fat of dead pigs, cattle and chickens is being used to make greener jet fuel https://www.bbc.co.uk/news/science-environment-65727664 https://www.bbc.co.uk/news/science-environment-65727664
    • pmontra 3y ago
      You're generally right but one of my nth ones was Brisbane to Cairns over the reef during the day. I'd fly it again and again.
      • huevosabio 3y ago
        One of my favorites is landing in Rio, the landscape is spectacular and then you get Christ the Redeemer as the cherry on top of the cake.
    • camgunz 3y ago
      I feel this for a lot of things, but I love flying and I've probably done it dozens of times. I know people who travel for business and that sounds like it'd kill the magic, but I still love it. Earbuds + laptop! Isolated for hours! They bring food _to you_! You're miles up in the sky!
• deleted 3y ago
  [deleted]
• rat9988 3y ago
  I thought the API was versioned, though it doesn't mean the new versions aren't worse. And he doesn't talk about the model used in chat gpt. I'm a bit skeptical that this answers exactly what we were worried about. Or maybe we were not all worrying exactly about the same thing.
• dubcanada 3y ago
  Is it static? If I ask it the same question I asked it a few months ago it gave me a completely different answer. Is that because of some additional context above? Should we be starting fresh chats sooner?
  • carabiner 3y ago
    It's never been static and there's always been randomness when you regenerate an answer. There is no control of a random seed, so you can get a different answers for the same prompt.
    • guraf 3y ago
      You're right of course that some randomness is inherent, but you can adjust the "temperature". With a low enough temperature you get essentially the same output every time, with a at most just minor words swapped.
  • acomjean 3y ago
    It’s not stactic. That’s the thing about ai, you ask it the same thing it gives you a different answer/different image every time.
    • drexlspivey 3y ago
      That’s not really an issue with AI in general, you can make it deterministic by tuning some parameters if you want (temperature in chatGPT, passing a seed in midjourney etc)
  • schrodinger 3y ago
    There’s _is_ a degree of randomness in its content generation, you can see it in action by hitting regenerate. It’s a statistical text predictor that iteratively selects the most like my word after the ones it’s selected so far, but when there are a few good candidates, it’ll choose one at random. However, the model is static (it’ll present the same candidate word list until retrained), which is what you may have heard. But the way a response is generated introduces the randomness. Note: I forget the parameter, but if you get the direct API access you can turn off this and get consistent answers.
• geraldwhen 3y ago
  Whatever happens to chat gpt, ai image generation is incredible. It’s so incredibly powerful that I don’t think it’s ever going away
  • saiya-jin 3y ago
    If you are doing some indie gaming, it can save tons of money on asset generation, you either ramp up and finalize it yourself or hire somebody but for fraction of the time. Ditto for web design, why would anybody but big, calcified megacorps with their ridiculous processes go and buy some stock images if you can get exactly what you want, in any imaginable style?
    • textninja 3y ago
      It still takes a surprising amount of labour and artistry to get an AI to give you exactly what you want (see also: Pareto principle). Consumer expectations scale proportionally to technological progress, so demand for premium assets and brand differentiators won’t be going away anytime soon; the state of the art has advanced but stock photo business will advance right along with it.
      • geraldwhen 3y ago
        I’d say ai art lets me get to a point where I can justify paying someone. It’s hard to understand a game without decent visuals, even if they’re filled with ai turds.
• skilled 3y ago
  ChatGPT was a massive dopamine hit, particularly for people in areas like development. It was a tremendous release that has definitely laid out some new tracks for the future, especially on the web. I myself have found to be using ChatGPT a lot less recently. I got the GPT-4 API access and then I realized that I can't really use it for anything super major because I can't afford it, it is ridiculously expensive if you consider that you have to pay for all the failed requests, the wrong information or the wrong context also. Instead, I have written a bunch of Python scripts that do a select few tasks for me and I have my terminal open 24/7 anyway. As for the topic at hand, I have _definitely_ noticed a lot more disclaimers in the UI. I don't get it from the API at all, in 6 months that I have been using the API - I've gotten one disclaimer. In the ChatGPT UI - I get them a lot. "Remember this", "Remember that", "Always look up the information" and things like this. I mean if it wasn't happening I would know because I have been a power-user pretty much all this time...
  • caeril 3y ago
    > it is ridiculously expensive What? It's 3p/6c cents per 1k tokens. I use GPT4 for all kinds of things, but a very basic example: Automatic API client/stub/endpoint generation with GPT4 for reasonable/typical data structure sizes, 4k tokens buys me GET/POST/DELETE/PUT for all of the above. The completion request finishes in about 90 seconds. A human junior developer would take nearly an hour. In the best case, maybe 20 minutes using something like Swagger. So that's ~20 cents versus $60 or so, to say nothing of the time. And this extends to everything else: writing tests, refactoring, writing configuration files, UI generation, database schema and SQL query generation, etc. What the hell are you even talking about?
    • skilled 3y ago
      Maybe more pay attention to my comment, > super major Also, it sounds like you have no idea that feeding the API back what it has written improves it tenfold for complex tasks. Enjoy your 90 second requests though.
• jimsimmons 3y ago
  He’s talking about the API. Not the web client
• bugglebeetle 3y ago
  Company who has repeatedly lied in public: there’s nothing to see here!
• greatpostman 3y ago
  Recent research showed that RLHF/censoring the model hurts the performance of the model. This is intuitively obvious, censorship isn’t real, the data is (moral issues aside). So it hurts the integrity of the weights. The future is open sourced uncensored models, capitalism will demand the high performance. There’s a huge discussion on it on Reddit right now: https://www.reddit.com/r/MachineLearning/comments/13tqvdn/uncensored_models_finetuned_without_artificial/?utm_source=share&utm_medium=ios_app&utm_name=iossmf https://www.reddit.com/r/MachineLearning/comments/13tqvdn/un...
  • haolez 3y ago
    Maybe humans are like that as well? :)
    • greatpostman 3y ago
      Lots of evidence pointing in that direction. Suppression of logic (censorship) leaks into other areas of cognition
  • mschuster91 3y ago
    > The future is open sourced uncensored models, capitalism will demand the high performance. No. The future is ML built on more than just Wikipedia, Github code and Reddit hot takes. Or, sarcasm aside, GIGO: garbage in, garbage out - when you don't take care what datasets you train any kind of model on, you're bound to get some surprises if something unexpected comes along. Be it the infamous "racist soap dispenser" or the Google (?) image classifier that made the rounds here just a day or two ago which had a safeguard because it kept confusing Black people with gorillas. Preventing this kind of harmful content isn't censorship - it's after-the-fact compensation for bad training (and the tendency of 4chan and other trolls to use discriminatory content generation as a weapon, just remember what they did to Microsoft Tay). That is where the money is: curated training datasets. Everyone and their dog can train a ML model from scratch, all you need is money and cloning a few more-or-less-broken Github repositories for that. But acquiring a high-quality dataset as a foundation? That costs real money to create.
• chatmasta 3y ago
  My experience using GPT-4 for coding is that it's got the knowledge and skill of a senior engineer, with the high maintenance of a junior engineer. You can get it to output small sections of quality code, but the amount of prodding it takes to piece it all together means you may as well have spent the time writing it yourself. But the future of GPT as a coding assistant is definitely bright. It just needs more chaining, so I feel less like its assistant, asking it to come up with prompts and then pasting them back to it after iterating on the code.
  • Our_Benefactors 3y ago
    It’s still way faster than I could hope to be. It’s a true refactoring machine with 0 mental effort. “Here is this code block, make it do this other thing, add this feature and argument and error condition” etc. It augments the amount of output that’s possible to achieve at any seniority level.
  • moffkalast 3y ago
    > so I feel less like its assistant Ah so I'm not the only one who feels this way lol. It's as brilliant as it is dumb.
  • naiv 3y ago
    For me it is not even the code per se but how it helps me with naming database tables and properties when I describe the broader scope. The future is more than bright, especially with a new model that is more current. Offtopic but just today I was aksing it for the most current versions it knows of right now: Python: 3.9 JavaScript: ECMAScript 2021 (ES12) Java: 16 C++: C++20 C#: 9.0 (.NET 5.0) Ruby: 3.0.0 Swift: 5.4 Go: 1.16 Rust: 1.51.0 TypeScript: 4.3 PHP: 8.0 Kotlin: 1.5.0 Scala: 3.0 R: 4.0.5 Perl: 5.32 And frameworks: Python: Django: Version 3.2 Flask: Version 2.0.1 Pyramid: Version 2.0 TensorFlow: Version 2.6.0 PyTorch: Version 1.9.0 JavaScript: Node.js: Version 16.9.1 Express.js: Version 4.17.1 React.js: Version 17.0.2 Angular: Version 12.2.4 Vue.js: Version 3.2.6 Java: Spring Framework: Version 5.3.9 Hibernate: Version 5.4.32.Final Struts: Version 2.5.26 PHP: Laravel: Version 8.54.0 Symfony: Version 5.3.6 CodeIgniter: Version 4.1.4 Ruby: Ruby on Rails: Version 6.1.4 Sinatra: Version 2.1.0 C#: .NET Core: Version 5.0 ASP.NET: Version 5.0 Entity Framework Core: Version 5.0
• mustacheemperor 3y ago
  In the thread yesterday it was brought up that if you use the API it feels considerably less hamstrung than the ChatGPT client, and this tweet seems to fit the assumption that the ChatGPT product is being tuned or governed differently from the API. >The API does not just change without us telling you. The models are static there. This reads to me as specifically indicating the models are not static elsewhere, ie, in ChatGPT.
  • jonplackett 3y ago
    It feels less hamstrung on output, but is instead hamstrung by being incredibly slow. GPT-4 via API will sometimes take 30 seconds + to respond to simple questions without any chat history, where through Chat GPT it will give you near enough instant replies only slightly slower than 3.5 turbo.
    • danielbln 3y ago
      Maybe the reverse is true? To provide faster performance they dumb down gpt4 (quantization?) that serves the chatgpt UI but the API is full power albeit slow.
      • jonplackett 3y ago
        I think it’s just prioritising ChatGPT. The reason that’s my guess is that the API is occasionally much faster and then goes slow again. It’s a bit all over the place which leads me to suspect it’s based on demand
• pierat 3y ago
  "GPT-4 hasn't gotten worse since March" can be 100% true at the same time OpenAI puts more rules and limiters keeping more interesting answers from being said. Ive noticed it quit giving as detailed answers and as thorough. It's also refused to do more complex programming where it used to accept those questions. Being artificially limited by OpenAI can still be done without it getting "worse". But it effectively is worse for us users.
  • H8crilA 3y ago
    His point is that it literally didn't change, that includes the safeguards (which are a part of the model).
    • jeremyjh 3y ago
      He said the API hasn't changed. But what about the Chat website?
      • jacquesm 3y ago
        Exactly my question... I wonder if the pre and post processing of the chat interaction isn't what is driving the perceived differences.
        • danielbln 3y ago
          I could not figure out what everyone was talking about yesterday as my experience with gpt4 has not degraded at all, but I'm using a third party client via API.
          • flir 3y ago
            Which 3rd party client, if you don't mind me asking? I'm looking to move, and the space isn't mature enough yet for there to be a clear leader.
            • 3y ago
• textninja 3y ago
  Must be in the trough of disillusionment. The tech is truly useful though so we’ll reach the plateau soon enough.
• josecyc 3y ago
  I agree I've noticed I need to be quite specific now, it won't realize bugs in the code unless I tell it so. A head to head comparison of the different versions is needed to validate this.
• karmasimida 3y ago
  The model didn’t change but doesn’t mean the inference didn’t change. Without going specifics it is meaningless for the discussion
  • heliophobicdude 3y ago
    The models for the api's could have not changed but the web app's might have too
• armchairhacker 3y ago
  I doubt the March 14 model is any different, since it's versioned and I don't see why OpenAI would want to change it behind the scenes. Also, companies are evaluating GPT-4 to determine whether they want to pay for it, so OpenAI has an strong incentive to not downgrade at least the API. I believe the May 5 model is different, at least in the chat interface, because it's fine-tuned to detect jailbreaks and the temperature/other hyper-parameters may have changed. And I can imagine this fine-tuning making the model less creative and worse at solving analytical tasks. Personally I haven't noticed any change, except in my own awareness. Sometimes GPT4 gets very hard prompts right, and sometimes it gets simple problems wrong. So it's not hard to see how people can form biased opinions from selective attention or just luck.
• haolez 3y ago
  I use GPT-4 a lot and something has definitely changed in my recent experience. It's simply worse.
• phillipcarter 3y ago
  Posted this in yesterday's thread, but once again I think this is just people feeling the magic wear off. People have poked around a lot more and found the flaws while also trying to get it to do real-world tasks. That wasn't true when it first came out. It's fine, this tech has never been magic anyways, won't be replacing all our jobs, won't take over the world, etc. It's still awesome for what it is.
  • bazmattaz 3y ago
    it’s a very neat parlour trick
• jacquesm 3y ago
  To what degree is the pre and post processing of the chat client the source of the confusion rather than GPT-4 itself?
• RecycledEle 3y ago
  If ChatGPT has been static since March (of 2023) the. Why does my version always change? Right now I'm running May 24. FWIW, I think it has improved.
• neonsunset 3y ago
  There is No War in Ba Sing Se.
• Mizoguchi 3y ago
  Unrelated to the model itself but infrastructure, yesterday was unusable for me, getting random "too many requests sorry bout that" errors. I think 1/4 of the requests during a 3 hr period didn't make it through. Imposible to build anything beyond experimental stuff on top of a service so unreliable. Haven't tried it through Azure yet, I wonder if it is any better?
  • lee101 3y ago
    [dead]
• dangerlibrary 3y ago
  A couple observations from attempting to use both GPT-3.5 and GPT-4 via the web interface for coding tasks: - The model's ability to respond accurately drops drastically when asked questions of the form "is there a different way to accomplish X, using Y?" or "is there a way to accomplish X that runs in O(log(n)) time instead?" Example: I wanted to upsert an integer value using a SQLite db using "INSERT ... RETURNING..." ChatGPT repeatedly told me that sqlite doesn't support "RETURNING" (it does, since March 2021). It insisted I would need two DB round trips from my application to accomplish this. When asked "can this be done in one round trip, instead?" it repeatedly wrote code that would return the number of rows modified instead of the integer column value. - ChatGPT's limited standard library knowledge means that the solutions it produces, even when correct, are often lower-level and less idiomatic. Problems that would be trivially solved with e.g. a Java.String.replaceAll or .codePointCount will instead loop over each character, often splitting the string into an intermediate array and implementing special cases for first/last character edge cases. The code winds up being mostly correct, but also (for lack of a better word) weird. No human I've ever worked with would do things the way ChatGPT sometimes does, which means the code will likely be much harder to maintain and debug over time.
  • ch4s3 3y ago
    > are often lower-level and less idiomatic I would go as far as to say it's adept at producing technically working spaghetti.
  • jstarfish 3y ago
    > ChatGPT repeatedly told me that sqlite doesn't support "RETURNING" (it does, since March 2021). Its dataset cuts off around 2021. There's a little footer message warning you not to expect knowledge of recent events.
    • dangerlibrary 3y ago
      It cuts off in September 2021, supposedly - that's why I pointed out the month of the relevant sqlite release.
• pleb_nz 3y ago
  I've found it's gotb eyes and have given up using it for tasks more often than I used to. For copilot, I no longer get multi line complete suggestions and it's really slow to deliver single line suggestions and they're more often incorrect. I need to dig into it, but it's definitely degraded further and I don't know if it's just my environment or a wider issue. I need to dig in and figure out - is anyone else experiencing these things?
• nashashmi 3y ago
  Is there a way to use GPT-4 directly? Instead of the muzzled ChatGPT that is supposed to give answers authorities and opinionators deem appropriate?
  • heliophobicdude 3y ago
    Assuming you mean the base model, no not right now. GPT-4 base model is not public.
• AtNightWeCode 3y ago
  Are we chatting directly with the model? Maybe the interface has changed. With long term use the likelihood of hitting edge cases is higher and maybe that is a cause as well for what users are seeing. People probably ask more vague questions over time. I might have done that. I have never experienced the amnesia problem in v3.5 though, that v4 clearly has. Just repeating incorrect answers that you ask it not to give. I did not have access to v4 in march so I can't do that comparison.
• yodsanklai 3y ago
  It's really fascinating to see all the irrational thinking triggered by ChatGPT. "risks of human race extinction", "soon no more need for developers, doctors or lawyers", "possibility of consciousness emerging", so called experts in "prompt engineering". It's a dumb tool, if you're lucky you can get it to spit something useful (but you need other tools to check the correctness of what it returned). There are certainly many useful applications, but the technology is inherently limited.
  • nerdbert 3y ago
    Sure, but you fail to understand how rapidly this technology is changing. Yes, today it's a machine for spewing out glibly rephrased reddit posts. But think back to the 1970s when jetpacks were first demonstrated. Today we're all flying around on them; the automobile, aviation, and flame-retardant headgear industries have been radically altered, and humanity will never be the same again.
• andrewstuart 3y ago
  GPT 3.5 is a much better coding assistant.
  • textninja 3y ago
    The quality is noticeably worse than 4 but the tighter feedback loops and lower expectations make up for it most of the time. So I’m inclined to agree; speed matters.
• elboru 3y ago
  I don’t get why is this being discussed, isn’t it easy to go check old conversations and try to replicate them today?
  • dgellow 3y ago
    GPT isn’t deterministic
  • renewiltord 3y ago
    You need to have used the API and set temperature zero, but if you have that historical data you can reasonably test and see.
  • ShamelessC 3y ago
    You would think so, but not a single person complaining has provided convincing proof of the degradation.
    • sebzim4500 3y ago
      I went through a bunch of old prompts and the response was definitely worse now than it was then. Since it's not determinstic though, it's hard to draw conclusions. Especially since the sample size was 10ish.
• emptyfile 3y ago
  [dead]
• rkapsoro 3y ago
  Perhaps the evolution of this story is an interesting example of confirmation bias?
• TradingPlaces 3y ago
  The model may be the same, but the chatbot is not. They have made the responses shorter to save inference expenses, which are huge in a model the size of GPT-4.
• lee101 3y ago
  [dead]
• dang 3y ago
  Recent and related: Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately? - https://news.ycombinator.com/item?id=36134249 https://news.ycombinator.com/item?id=36134249 - May 2023 (711 comments)
• madrox 3y ago
  This doesn’t pass the smell test for me. Something has changed. Maybe the model hasn’t changed, but something somewhere is making output worse. I’m honestly more concerned if OpenAI doesn’t even realize it. Nothing is more infuriating as a user than convincing the developer your bug actually does exist. It speaks to poor monitoring, testing, and tooling.
• petabite 3y ago
  The amount of times per day I have to click "Stop generating" and then say "No!" has definitely increased
• wg0 3y ago
  This was predictable from the get go and many had had pointed that out already that soon people will start noticing that LLMs aren't as magical as they thought once the initial awe wanes away. Don't think OpenAPI is doing anything here, it's not in their interest to reduce the "quality" even there's no objective and repeatable way to measure the quality either. It's all probabilities all the way down. Who knows what the model will do. I mean, you can dry run by hand but even on quad core processors, it's damn slow so imagine the inference by hand.
  • sebzim4500 3y ago
    Was anyone in that thread claiming that the API had gotten worse? A lot of people were suggesting to use the API rather than the ChatGPT interface in order to avoid the degradation.
    • wg0 3y ago
      Has OpenAPI themselves announced or someone has objectively and verifiably proved that the UI and API invoke two different models?
      • sebzim4500 3y ago
        Not directly, but the model version displayed in the UI changes every week or so while the api version has apparently been stable.
  • uw_rob 3y ago
    > This was predictable from the get go and many had had pointed that out already that soon people will start noticing that LLMs aren't as magical as they thought once the initial awe wanes away. Familiarity makes something novel appear trivial. I don't think the magic going away has anything to do with the magic of the underlying technology. Airplanes are amazing technology, but they become as boring as a car to regular fliers.
• two_in_one 3y ago
  ChatGPT Plus shows the version. It was May 12, if I remember correctly, then it changed to May 24. Which probably means there were some changes. If not in GPT-4 itself, then in pre or post processing. They should have some safety filters at the end, I think.
  • doubleunplussed 3y ago
    I mean, there were UI changes. Could be just that.
• jiggawatts 3y ago
  I look at everything someone from OpenAI says as if a politician is saying it. Sam Altman especially is fond of statements that are deceptive but technically true. His employees appear to be following his lead. GPT 4 isn’t ChatGPT 4, which is what most people use. There is also the “system prompt”, which is also likely to be changing but not part of GPT 4. Etc…
  • sebzim4500 3y ago
    I don't see this as being political doublespeak. He is very clearly just talking about the API.
    • CSMastermind 3y ago
      Right but people are complaining about their experiences with ChatGPT. If you say, "ChatGPT has gotten noticeably worse" and they respond "nothing about our APIs have changed" then it would be reasonable to interpret that as them saying that nothing about ChatGPT has changed. When in reality many things might have changed about ChatGPT.
      • stavros 3y ago
        He says "the models are static", which means that, if you keep using the same model, you will keep getting the same results.
        • Capricorn2481 3y ago
          That is not what that means
      • furyofantares 3y ago
        The tweet is a response to an API user saying GPT-4 got worse, and the tweet is clearly talking about the API and the model - the context of the HN megathread mostly talking about ChatGPT isn't present in the tweet that's being replied to. There's a lot of confusion around this, but nothing appears to be caused by doublespeak to me. Confusion about which GPT/ChatGPT anyone is reporting about has been pretty ubiquitous for a while.
• Ozzie_osman 3y ago
  He said "the API". He didn't say anything about the Chat version, which seems to have more protections that may be on top of the model and not embedded into it.
• sebzim4500 3y ago
  I understand why people don't read 10,000 word articles, but this is a 15 word tweet. Would it really kill people to read it before commenting? He is very explicitly talking about the API, which uses a different model than the UI. The recent discussion was about the degradation in the UI model.
• dontupvoteme 3y ago
  I just discovered when playing around with translations that there is some hidden filter/killswitch that immediately stops the generation of the opening of some books. It doesn't matter if the invoking prompt is to recite the book opening paragraph, or to translate it from a foreign language to English, or what. It's not RLHF induced because it works via API and it only triggers in English, but sure enough try to get it to output >Call me Ishmael. Some years ago— >"It was the best of times, I guess this might get you flagged (there is no alert to the user that this filter kicked in, and it will output it in any other language, and it works in the API) so I'm hesitant to play around with it more, but it's very strange - especially as these are long since in the public domain.
  • jiggawatts 3y ago
    They're trying to stop students using ChatGPT to cheat on homework assignments.
    • dontupvoteme 3y ago
      by preventing it from outputting one of the most famous sentences in English literature? well done lads hilariously if you throw s/worst/blurst/ on to your request it can output the forbidden dickens n-gram. does this constitute a jailbreak? or is the entire thing utterly emblematic of the modern technolegal mess of things since dickens is squarely and quintessentially in the Public Domain
• boppo1 3y ago
  I read somewhere there was a paper they released that as they tuned for alignment, overall quality dropped proportionally. Anyone know the name of it?
  • dr_dshiv 3y ago
    Search for alignment tax
    • boppo1 3y ago
      Thanks!
  • bazmattaz 3y ago
    What is “alignment” in this context?
• s1mon 3y ago
  I can believe that the core GPT-4 model itself has not changed, but clearly they've changed some features. They've added plug-in access, which can change GPT-4's capabilities greatly with some chats. The over all UI of the web site has changed several times (dropdown for GPT-3.0/3.5/4.0 turned into a GPT 3.5|GPT 4.0 button, they added the ability to share chats, and I'm sure there are other small details).
• moneywoes 3y ago
  Yea that is very tough to believe
• geoduck14 3y ago
  Gpt is starting to suck? I'll take the blame for this one. I've been submitting crazy prompts since the beginning in the hopes of confusing it!
• m3kw9 3y ago
  Who knows? Maybe this engineer doesn’t know what’s going on behind the scenes. I would think the models storage is so tightly guarded only few would have access to even know if something has changed. The ones that know would have nda
  • heliophobicdude 3y ago
    His role is Developer Relations fwiw
• deleted 3y ago
  [deleted]
• oldgregg 3y ago
  OpenAI turned on the "full featured" GPT4 just long enough to learn what people could use it for. Now they turn off those features and use all of those ideas to spawn new companies on a now-private GPT4 Original api and cripple everyone else. When they say "don't worry about building on our API we won't go up the value-chain" -- no shit, but only because they don't want regulatory scrutiny. So they will just trickle-out API access to companies that they own and control through extensive personal networks. Google, Facebook, Twitter, Amazon... Build on someone else's API and they will jam you 100% of the time.
• agentcoops 3y ago
  Many comments miss the mark as they fail to make the crucial distinction between ChatGPT and GPT-4. GPT-4 is the underlying model one can indeed have direct access to on a pay-per-request pricing scheme. ChatGPT is an application built on top of GPT-4 which manages how the 'context' of your interaction is passed in on a per-request basis. I don't doubt the spokesperson for a minute: from my own experience, the underlying GPT-4 models have not changed and I sincerely believe that OpenAI will be careful on this front, given that they are aiming to build a once-in-a-generation company that provides a stable platform for other firms to build products on top of. The ChatGPT application, on the other hand, and how it manages context etc has certainly changed in the intervening time. That is completely expected as even and perhaps especially OpenAI is figuring out how to build applications on top of LLMs, which means balancing how one can get the best quality results out of the model while making ChatGPT in particular a profitable business. Stratechery has analyzed this problem for OpenAI in the most detail I've seen. I imagine the company is in something of a bind figuring out how to invest between the APIs themselves and ChatGPT. On the one hand, the latter is incredibly successful as a consumer app with a lead it will be difficult for rivals to catchup with and it is likely plugins will provide a good revenue basis. On the other hand, there is certainly a greater business opportunity in being the foundation for an entire generation of AI products and taking BPs off of revenues -- if and only if GPT4 indeed has a significant moat over the opensource alternatives. For the moment, it would seem they will have to hedge both bets as we see how the consumer space and the competition between models heats-up.
• denverllc 3y ago
  The actual statement is "The API does not just change without us telling you. The models are static there." But that isn't ChatGPT. Here's how I feel ChatGPT answers my coding questions now: Me: Write a Python script to sum two numbers. ChatGPT: Python is a programming language that was invented in 1991 and can be used to solve a variety of programs. Here is an example of how to sum two numbers: def sum(a, b): # note: the actual code has been left out as it depends on the actual specifications of how you want to add\* Note that this code is merely an example and writing a Python script to sum two numbers is a complex problem that requires careful attention to whether the numbers you are trying to sum can be summed. Also, as my knowledge has a cutoff date of 2021, there may be other ways to perform this summation. Please check with the documentation or ask someone who knows how to code. * note: ChatGPT has actually done this to me
• huksley 3y ago
  Model has not changed. Prompt transformation code have changed. So the same prompt you do are delivered to the model with a different "wrapper" prompt, significantly changing the answer model is producing.
• Madmallard 3y ago
  What can we do to upheave this organization and their blatantly anticompetitive tactics?
• renonce 3y ago
  Does the author refer to GPT-4 API or the ChatGPT version of GPT-4? The recent discussions on GPT-4 quality deterioration seem to focus on the ChatGPT version rather than the API. Also, since ChatGPT version of GPT-4 is now supporting web browsing and plugins, I would assume it has to have been updated.
• kvetching 3y ago
  This is why I unsubscribed from OpenAI. They seem to be virtue signaling about their lack of progress now. Months later, GPT-4 still slow, still not multi-modal as they advertised, still significantly limited, you need to sign up for a waitlist for almost every feature, no sense of privacy, no understanding of their plan for improvements. Google is full steam ahead and consistently improving their free LLMs. They actually had a genius strategy. Put out Bard with a very stupid LLM, so people aren't blown away and it doesn't get the doomsayers on their case. Now they can continue to quietly upgrade Bard. Eventually it will be so obvious that they have surpassed OpenAI. OpenAI must enjoy watching their unsubscriber count go down. After all, Sam did say at the congressional hearing multiple times "We would prefer if people used it less".
• theknocker 3y ago
  [dead]
• mensetmanusman 3y ago
  If they are overwhelmed with usage requests and can't buy GPUs fast enough, the model could be the same but the amount of compute spent on the answer could be ramping down impacting the quality. It would be really cool to quantify how much compute spend per interaction. If this was visible, users could spend an arbitrary amount and modify how much they're willing to pay for 'better' responses. This is probably a better business model.
• deleted 3y ago
  [deleted]
• boringuser2 3y ago
  This is basically a tacit admission that ChatGPT4 *has* gotten worse.