12 ms·

OpenAI's models feel 100% nerfed to me at this point. I had it solving incredibly complex problems a few months ago (i.e. write a minimal PDF parser example), b

• Spooky23 3y ago
  For coding I’ve had better luck with Bard. OpenAI doesn’t like me for some reason. My kid has no problem, but I was getting rate limited and error messages early on.
• deleted 3y ago
  [deleted]
• ekidd 3y ago
  > but today you will get scolded for asking such a complicated task of it. Huh, I just double-checked ChatGPT 4 by feeding it a moderately complicated programming problem, and asked it to solve the problem in Rust. Performance looks solid, still. I gave it a deliberately vague spec and it made good choices. And I've seen it do some really dramatic things in the last couple of weeks. So I'm not seeing any significant evidence of major drops in output quality. But maybe I'm looking at different problems.
• geysersam 3y ago
  > I like to imagine certain billion/trillion-dollar mega corps had a back-room say regarding things that they would really prefer OpenAI's models not be able to emit. What a weird conspiracy theory. Why would Microsoft have anything against your pdf-parser? More likely it just costs them insane amounts of money running their most capable models, and therefore they're "nerfing" them to reduce costs .
  • vervez 3y ago
    github copilot won't pay for itself.
    • geysersam 3y ago
      So the theory is: Microsoft nerfs GPT4, a product they (basically) own that people pay to access, so that people will stop using that service and pay for another Microsoft product instead?
      • sebzim4500 3y ago
        Especially when copilot is cheaper than gpt-4.
• sdfgdfghj 3y ago
  `Microsoft is a big stakeholder and they might not want to get sued...` Wait I thought Microsoft has bought the right to get profits from OpenAI, not actually shares in the company? Can someone correct me if I'm wrong>
• derefr 3y ago
  > I like to imagine certain billion/trillion-dollar mega corps had a back-room say regarding things that they would really prefer OpenAI's models not be able to emit. Microsoft is a big stakeholder and they might not want to get sued... Liability could explain a lot of it. I don't think it's any of these things. OpenAI and the company I work for have a very similar problem: the workload shape and size for a query, isn't strictly determined by any analytically-derivable rule regarding any "query compile-time"-recognizable element of the query; but rather is determined by the shape of connected data found during initial steps of something that can be modelled as a graph search, done inside the query. Where, for efficiency, that search must be done "in-engine", fused to the rest of the query — rather than being separated out and done first on its own, such that its results could be legible to the "query planner." This paradigm means that, for any arbitrary query you haven't seen before, you can't "predict spend" for that query — not just in the sense of charging the user, but also in the sense that you don't know how much capacity you'll have to reserve in order to be able to schedule the query and have it successfully run to completion. Which means that sometimes, innocuous-looking queries come in, that totally bowl over your backend. They suck up all the resources you have, and run super-long, and maybe eventually spit out an answer (if they don't OOM the query-runner worker process first)... but often this answer takes so long that the user doesn't even want it any more. (Think: IDE autocomplete.) In fact, maybe the user got annoyed, and refreshed the app; and since you can't control exactly how people integrate with your API, maybe that refresh caused a second, third, Nth request for the same heavyweight query! What do you do in this situation? Well, what we did, is to make a block-list of specific data-values for parameters of queries, that we have previously observed to cause our backend to fall over. Not because we don't want to serve these queries, but because we know we'll predictably fail to serve these queries within the constraints that would make them useful to anyone — so we may as well not spend the energy trying, to preserve query capacity for everyone else. (For us, one of those constraints is a literal time-limit: we're behind Cloudflare, and so if we take longer than 100s to respond to a [synchronous HTTP] API call, then Cloudflare disconnects and sends the client a 524 error.) "A block-list of specific data-values for parameters of queries" probably won't work for OpenAI — but I imagine that if they trained a text-classifier AI on what input text would predictably result in timeout errors in their backend, they could probably achieve something similar. In short: their query-planner probably has a spam filter.
• bluecrab 3y ago
  Same, I am not paying for copilot also.
• PeterisP 3y ago
  It's a big pain when trying to build things on top of the GPT-4 API. We had some experiments that were reliably, reproducibly achieving a goal, and then one day it suddenly stops working properly; then the student managed a different prompt that worked (again, reproducibly, with proper clean restarts from fresh context), and within a few days it broke. I understand that there is a desire to tweak the model and improve it, and that's likely the way to go for the "consumer" chat application; however, both for science and business, there is a dire need to have an API that allows you to pin a specific version and always query the same old model instead of the latest/greatest one. Do we need to ask the LLM vendors to provide a "Long Term Support" release for their model API?
  • sebzim4500 3y ago
    In the API you can ask for a specific version. Were you doing that?
    • napier 3y ago
      They almost certainly were. But the API only offers two default choices of GPT-4 (unless one has been anointed with exalted 32k access): 1. gpt-4-default which has been progressively nerfed with continuous no notification zero changelog zero transparency updates . 2. gpt-4-0314 which is a frozen checkpoint from shortly after public launch and is still great but not quite as good as the very original, or as flexible as the fridged based model. Fine. However it’s currently due to “no longer be supported” i.e. retired on June 14th. It’s kind of a challenge to commit to building on murky quicksand foundations of an API product that changes drastically (but ineffably for the worse) without warning like the default accessible version does, and soon it looks like there won’t be a stable un-lobotomized alternative.
      • com2kid 3y ago
        The latest GPT 3.5 model has actually been getting better at creative writing tasks on a regular basis, which is actually bad for certain tasks due to the token limit. Whereas before GPT 3.5 could write a short story and finish it up nicely in a single response, now days it is more descriptive (good!) and thus runs out of tokens before concluding (bad!)
• mcbuilder 3y ago
  Man, you nailed it with the dopamine hangover. I wonder if it is just our collective human delusional preoccupation with searching for a "greater intellegence" that makes these models more seem impressive combined with the obvious nerfs by OpenAI that produce this effect.
• dgellow 3y ago
  I had the exact same problem, I thought it was just in my mind. I feel that I’m now constantly being scolded for asking what GPT4 seems to see as complex questions, it’s really frustrating.
• AtNightWeCode 3y ago
  I call BS on this. ChatGPT anyversion could never solve complex problems. That is just silly. I tried to get the version 4 to solve some very basic problems around generating code. It suggested to build several trees including a syntax tree and then project things between these trees. The solution I wrote is straight forward and not even 50 lines of code.
• QuantumGood 3y ago
  Isn't this likely because they're limiting the kind of work that the (currently rolling out) "code interpreter" plugin will do? Won't it likely change to "use code interpreter for this kind of request"? Among other reasons, by forcing use of code interpreter, they can charge extra for it later.
• tmaly 3y ago
  Before you cancel your premium, can you go into your history and get your prompts and the response with a code from a few months ago and post them here? I would like to see if I asked the exact same prompts whether I could get roughly the same code. I think we need some way to prove of disprove the assertion in this Ask HN post.
  • gpickett00 3y ago
    Agreed. I see these claims all the time. Feels a little bit like, "dId yOu KnOw iNsTaGram lIsTens tO yOuR cOnVeRsAtIoNs?" It would be pretty easy to prove. Show us before/after screen shots.
• bigcloud1299 3y ago
  I think it is about how Microsoft may not want cannibalize its own products such as copilot. I also imagine that in the future OpenAI would prevent the whole chat with your own data feature in the name of data safety but it would be because Microsoft would want to sell that feature as part of office suite.
• hanselot 3y ago
  So far my experience with Vicunlocked30b has been pleasant. https://huggingface.co/TheBloke/VicUnlocked-30B-LoRA-GGML https://huggingface.co/TheBloke/VicUnlocked-30B-LoRA-GGML Although I haven't had much of my time available for this recently. My recommendation would be to start with https://github.com/oobabooga/text-generation-webui https://github.com/oobabooga/text-generation-webui You will find almost everything you need to know there and on 4chan.org/g/catalog - search for LMG.
  • winstonprivacy 3y ago
    How much VRAM does that model need? I'm looking for a 30B byte that I can train a LORA with on an GTX 4090.
  • pk-protect-ai 3y ago
    30B models are in no way comparable to GPT-4 even to GPT-3. There is no spacial comprehension in models with less then 125B params (or I had no access to such model). 130B GLM seems to be really interesting as the crowd-source start though, or 176B BLOOMZ, which requires additional training (it is underfitted as hell). BLOOMZ was better then GPT-3.5 for sure, but yeah underfitted ...
  • boppo1 3y ago
    You should beware that /lmg/ is full of horrible people, discussing horrible things, like most of 4chan. Reddit's r/locallama is much more agreeable. That said, the 4chan thread tends to be more up-to-date. These guys are serious about their ERP.
    • hanselot 3y ago
      [flagged]
      • dang 3y ago
        Could you please stop posting unsubstantive comments and flamebait? You've unfortunately been doing it repeatedly. It's not what this site is for, and destroys what it is for. If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html and taking the intended spirit of the site more to heart, we'd be grateful.
• dmm 3y ago
  I agree if this trend continues even inferior local models are going to have value just because the public apis are so limited. > Conspiracy shenanigans aside, I've decided to cancel my "premium" membership and am exploring open/DIY models. The crazy thing is that this is an application that really benefits from being in the cloud because the high vram gpus are so expensive that it makes sense to batch requests from many users to maximize utilization.
• BirAdam 3y ago
  Might be cost control? Keep the hardware usage per query low so that they can actually profit (or maybe just break even)?
• bdw5204 3y ago
  I think the only real path forward is for somebody to create an open source "unaligned" version of GPT. Any corporate controlled AI is going to be nerfed to prevent it from doing things that its corporate master considers to not be in the interests of the corporation. In addition, most large corporations these days are ideological institutions so the last thing they want is an AI that undermines public belief in their ideology and they will intentionally program their own biases into the technology. I don't think the primary concern is really liability although it is possible that they'd use that kind of language. The primary concern is likely GPT helping people start competitors or GPT influencing public opinion in ways either executives or a vocal portion of their employees strongly disagree with. A genuinely open "unaligned" AI would at least allow anybody who has the necessary computing power or a distributed peer to peer network of people who have the necessary computing power to run a powerful and 100% uncensored AI model. But of course this needs to be invented ASAP because the genie needs to be out of the bottle before politicians and government bureaucrats can get around to outlawing "unaligned" AI and protecting OpenAI as a monopoly.
  • dasil003 3y ago
    Who has the necessary resources to run, let alone train the model?
    • xtracto 3y ago
      All of us together do. I saw the nerfing of GPT in real time: one day it was giving me great book summaries, the next one it said that it couldn't do it due to copyright. I actually called it in a comment several months ago: copyright and other forms of control would make GPT dumb in the long run. We need an open source frontier less version.
      • circuit10 3y ago
        There is a lot of randomness involved, are you sure it wasn’t just chance? If you try again it might work
      • kayge 3y ago
• taf2 3y ago
  Do you think a lot of that is scaling pain... like what if they're making cuts to the more expensive reasoning layers to gain more scale. Seems more plausible to me that the teams keeping the lights on have been doing optimization work to save cost and improve speed. The result during those optimizations might not be immediately obvious to the team and then they push deploy and only through anecdotal evidence such as yours can they determine the result of their clever optimization resulted in a deteriorated user experience... I mean think about doing a UI update that improves site performance but has the side effect of making the site appear to load slower because the transition effects are removed... Just my 2 cents trying to think more like their are humans supporting that thing that grew at a crazy speed to 100 million users.
  • sdwr 3y ago
    Yeah that's my assumption too. Flat rate subscription, black box model, easy to start really impressive then chip away at computation used over time. In my experience, it's been a mixed bag - had 1 instance recently where it refused to do a bunch of repetitive code, another case where it was willing to tackle a medium complexity problem.
• formerly_proven 3y ago
  > Microsoft is a big stakeholder and they might not want to get sued... Liability could explain a lot of it. Microsoft is also responsible for Copilot (based around a “it’s unconditionally fair use if you use a mirror” legal theory) so this doesn’t track.
  • robnado 3y ago
    Perhaps Microsoft would rather you pay for both ChatGPT and Copilot.
    • JamesBarney 3y ago
      I really don't think Satya is getting in a room with Sam, and going "We really need you to nerf GPT-4, the thing you're devoting your life to because you believe it will be so transformative, because of a product we sell that generates .001% of our revenue."
    • formerly_proven 3y ago
      Gotta pay for every use case of the general purpose LLM separately!
• gumballindie 3y ago
  > incredibly complex problems > write a minimal PDF parser Perhaps i’m missing something, but why is this “incredibly complex”?
  • deleted 3y ago
    [deleted]
  • bob1029 3y ago
    Can you give us an example of something you'd consider to be a complicated problem? Certainly, you could look at PDF as a boring-ass "follow the spec" experience, and indeed - I think this is precisely why certain arbitrary limitations are in place now.
    • gumballindie 3y ago
      I honestly have no clue about what makes pdf parsing a complex task. I wasnt trying to sound condescending. Would be great to know what makes this so difficult, considering the pdf file format is ubiquitous.
      • jahewson 3y ago
        Having spent considerable time working on PDF parsers I can say that it’s a special kind of hell. The root problem is that Acrobat is very permissive in what it will parse - files that are wildly out of spec and even partially corrupt open just fine. It goes to some length to recover from and repair these errors, not just tolerate them. On top of that PDF supports nesting of other formats such as JPEG and TTF/OTF fonts and is tolerant of similar levels of spec-noncompliance and corruption inside those formats too. One example being bad fonts from back in the day when Adobe’s PostScript font format was proprietary and 3rd parties reverse-engineered it incorrectly and generated corrupt fonts that just happened to work due to bugs in PostScript. PDF also predates Unicode, so that’s fun. Many PDFs out there have mangled encodings and now it’s your job to identify that and parse it.
      • MrTortoise 3y ago
        TEXT ISNT STORED AS TEXT ITS STORED AS POSTSCRIPT PRINTING INSTRUCTIONS IIRC
• barbariangrunge 3y ago
  Using gpt to help me with research for writing fiction has been a mess for me. Gpt basically refuses to answer half my questions or more at this point. “I can’t help you. Have you considered writing a story that doesn’t include x?” I’ve almost stopped using it lately. It wasn’t this bad a month or two ago
  • hesdeadjim 3y ago
    I always found it borderline useless for fiction before. OpenAI's obsession with avoiding anything "dark" and trying to always steer a conversation or story back to positive cliches was difficult to work around. Unless there is draconian regulation that happens to prevent it, I'm hoping at some point I can pay money to access a far less neutered LLM, even if it's not quite as capable as GPT-X.
• bevenky 3y ago
  I shared my exp below on one of the comments, sharing here too - I think overall the quality is significantly poorer on GPT4 with plugins and bing browsing enabled. If you disable those, I am able to get the same quality as before. The outputs are dramatically different. Would love to hear what everyone else sees when they try the same.