4 ms·
They certainly need the money. The Pro service has been running in limited mode all week due to being over capacity. It defaults to “concise” mode during high c
by cainxinth 2y ago
They certainly need the money. The Pro service has been running in limited mode all week due to being over capacity. It defaults to “concise” mode during high capacity but Pro users can select to put it back into “Full Response.” But I can tell the quality drops even when you do that, and it fails and brings up error messages more commonly. They don’t have enough compute to go around.
- sbuttgereit 2y agoHmmm... I wonder if this is why some of the results I've gotten over the past few days have been pretty bad. It's easy to dismiss poor results on LLM quality variance from prompt to prompt vs. something like this where the quality is actively degraded without notification. I can't say this is in fact what I'm experience, but it was noticeable enough I'm going to check.
- 55555 2y agoSame experience here.
- jmathai 2y agoNever occurred to me that the response changes based on load. I’ve definitely noticed it seems smarter at times. Makes evaluating results nearly impossible.
- kridsdale1 2y agoMy human responses degrade when I’m heavily loaded and low on resources, too.
- TeMPOraL 2y agoUnrelated. Inference doesn't run in sync with the wall clock; it takes whatever it takes. The issue is more like telling a room of support workers they are free to half-ass the work if there's too many calls, so they don't reject any until even half-assing doesn't lighten the load enough.
- baxtr 2y agoRecently I started wondering about the quality of ChatGPT. A couple of instances I was like: "hmm, I’m not impressed at all by this answer, I better google it myself!" Maybe it’s the same effect over there as well.
- dave84 2y agoRecently I asked 4o to ‘try again’ when it failed to respond fully, it started telling me about some song called Try Again. It seems to lose context a lot in the conversations now.
- Seattle3503 2y agoThis is one reason closed models suck. You can't tell if the bad responses are due to something you are doing, or if the company you are paying to generate the responses is cutting corners and looking for efficiencies, eg by reducing the number of bits. It is a black box.
- neya 2y agoI am a paying customer with credits and the API endpoints rate-limited me to the point where it's actually unusable as a coding assistant. I use a VS Code extension and it just bailed out in the middle of a migration. I had to revert everything it changed and that was not a pleasant experience, sadly.
- htrp 2y agoControl your own inference endpoints.
- its_down_again 2y agoCould you explain more on how to do this? e.g if I am using the Claude API in my service, how would you suggest I go about setting up and controlling my own inference endpoint?
- handfuloflight 2y agoYou can't. He means by using the open source models.
- datavirtue 2y agoRuna local LLM tuned for coding on LM Studio. It has a server and provides endpoints.
- square_usual 2y agoWhen working with AI coding tools commit early, commit often becomes essential advice. I like that aider makes every change its own commit. I can always manicure the commit history later, I'd rather not lose anything when the AI can make destructive changes to code.
- webstrand 2y agoI can recommend https://github.com/tkellogg/dura https://github.com/tkellogg/dura for making auto-commits without polluting main branch history, if your tool doesn't support it natively
- nowahe 2y agoI've had it refuse to generate a long text response (I was trying to concise a 300kb documentation to 20-30kb to be able to put it in the project's context), and every time I asked it replied "How should structure the results ?", "Shall I go ahead with writing the artifacts now ?", etc. It wasn't even during the over-capacity event I don't think, and I'm a pro user.
- Filligree 2y agoHate to be that guy, but did you tell it up front not to ask? And, of course, in a long-running conversation it's important not to leave such questions in the context.
- nowahe 2y agoThe weird thing is that when I tried to tell it to distill it to a much smaller message it had no problem outputting it without any followup questions. But when I edited my message to ask it to generate a larger response, then I got stuck in the loop of it asking if I was really sure or telling me that `I apologize, but I noticed this request would result in a very large response.` It sparks me as odd, because I've had quite a few times where it would generate me a response over multiple messages (since it was hitting its max message length) without any second-guessing or issue.
- el_benhameen 2y agoInteresting. I also find it frustrating to be rate limited/have responses fail when I’m paying for the product, but I’ve actually found that the “concise” mode answers have less fluff and make for faster back and forth. I’ve once or twice looked for the concise mode selector when the load wasn’t high.
- johnisgood 2y agoAgreed, I was surprised by it after I first have subscribed to Pro and had a not-that-long chat with it.
- rvz 2y agoAll that money and talk of "scale" and yet not only it is slow but costs billions a year to run at normal load and is struggling at high load. This is essentially Google-level load and they can't do it.
- jmathai 2y agoI’ve been using the API for a few weeks and routinely get 529 overloaded messages. I wasn’t sure if that’s always been the case but it certainly makes it unsuitable for production workloads because it will last hours at a time. Hopefully they can add the capacity needed because it’s a lot better than GPT-4o for my intended use case.
- rmbyrro 2y agoSonnet is better than 4o for virtually all use cases. The only reason I still use OpenAI's API and chatbot service is o1-preview. o1 is like magic. Everything Sonnet and 4o do poorly, o1 solves like a piece of cake. Architecting, bug fixing, planning, refactoring, o1 has never let me know on any 'hard' task. A nice combo is have o1 guiding Sonnet. I ask o1 to come up with a solution and explanation, then simply feed its response into Sonnet to execute. That running on Aider really feels like futuristic stuff.
- gcanko 2y agoExactly my experience as well. Like Sonnet can help me in 90% of the cases but there are some specific edge cases where it struggles that o1 can solve in an instant. I kinda hate it because of having to pay for both of them.
- andresgottlieb 2y agoYou should check out Librechat. You can connect different models to it and, instead of paying for both subscriptions, just buy credits for each API.
- joseda-hg 2y agoHow does the cost compare?
- cruffle_duffle 2y ago> just buy credits for each API I’ve always considered doing that but do you come out ahead cost wise?
- moffkalast 2y agoTheir shitty UI is also not doing them any infrastructure favors, during load it'll straight up write 90% of an answer, and then suddenly cancel and delete the whole thing, so you have to start over and waste time generating the entire answer again instead of just continuing for a few more sentences. It's like a DDOS attack where everyone gets preempted and immediately starts refreshing.
- wis 2y agoYes! It's infuriating when Claude stops generating mid response and deletes the whole thread/conversation. Not only you lose what it has generated so far, which would've been at least somewhat useful, but you also lose the prompt you wrote, which could've taken you some effort to write.
- cma 2y ago> But I can tell the quality drops even when you do that Dario said in a recent interview that they never switch to a lower quality model in terms of something with different parameters during times of load. But he left room for interpretation on whether that means they could still use quantization or sparsity. And then additionally, his answer wasn't clear enough to know whether or not they use a lower depth of beam search or other cheaper sampling techniques. He said the only time you might get a different model itself is when they are A-B testing just before a new announced release. And I think he clarified this all applied to the webui and not just the API. (edit: I'm rate limited on hn, here's the source in reply to the below https://www.youtube.com/watch?v=ugvHCXCOmm4&t=42m19s https://www.youtube.com/watch?v=ugvHCXCOmm4&t=42m19s )
- llm_trw 2y agoNeither does OAI. Their service has been struggling for more than a week now. I guess everyone is scrambling after the new qwen models dropped and matched the current state of the art with open weights.
- shmatt 2y agoin the beginning i was agitated by Concise and would move it back manually. But then I actually tried it, I asked for SQL and it gave me back SQL and 1-2 sentences at most Regular mode gives SQL and entire paragraphs before and after it. Not even helpful paragraphs, just rambling about nothing and suggesting what my next prompt should be Now I love concise mode, it doesn't skimp on the meat, just the fluff. Now my problem is, concise only shows up during load. Right now I can't choose it even if i wanted to
- cruffle_duffle 2y agoTotally agree. I wish there was a similar option on ChatGPT. These things are seemingly trained to absolutely love blathering on. And all that blathering eats into their precious context window with tons of repetition and little new information.
- therein 2y agoOh you are asking for a 2 line change? Here is the whole file we have been working on with a preamble and closing remarks, enjoy checking to see if I actually made the change I am referring to in my closing remarks and my condolences if our files have diverged.
- cruffle_duffle 2y agoYou know the craziest thing I’ve seen ChatGPT do is claim to have made a change to my terraform code acting all “ohh here is some changes to reflect all the things you commented on” and all it did was change the comments. It’s very bizarre when it rewrites the exact same code a second or third time and for some reason decides to change the comments. The comments will have the same meaning but will be slightly different wording. I think this behavior is an interesting window into how large language models work. For whatever reason, despite unchanging repetition, the context window changed just enough it output a statistically similar comment at that juncture. Like all the rest of the code it wrote out was statistically pointing the exact same way but there was just enough variance in how to write the comment it went down a different path in its neural network. And then when it was done with that path it went right back down the “straight line” for the code part. Pretty wild, these things are.
- 0xDEAFBEAD 2y agoMore evidence that people should use wrappers like OpenRouter and litellm by default? (Makes it easy to change your choice of LLMs, if one is experiencing problems)
- deleted 2y ago[deleted]