3 ms·
No amount of valuation can fix global supply issues for GPUs for inference unfortunately. I suspect they're highly oversubscribed, thus the reason why we're se
by jjcm 6mo ago
No amount of valuation can fix global supply issues for GPUs for inference unfortunately.
I suspect they're highly oversubscribed, thus the reason why we're seeing them do other things to cut down on inference cost (ie changing their default thinking length).
- natpalmer1776 6mo agoRemember when OpenAI wasn’t allowing new subscriptions to their ChatGPT pro plans because they were oversubscribed? Pepperidge Farms remembers.
- andai 6mo agoWouldn't that be good? I remember back in the day you could only get Gmail thru an invite, it was an awesome strategy. "Currently closed for applications" creates FOMO. They'd just need to actually get the GPUs in relatively short supply. They could do it in bursts though, right? "Now accepting applications for a short time." I'm not an internet marketer but that sounds like a win win to me. People feel special, they get extra hype, and the service isn't broken.
- the_gipsy 6mo agoYes, "Pepperidge farm remembers" is usually about how something used to be good.
- CoastalCoder 6mo agoYeah, but there was a spoof on that (in Family Guy?). It was a tie in to the movie "I Know what you Did last Summer", IIRC.
- hirako2000 6mo agoIn the case of Gmail that was fake scarcity. In the case of Anthropic is fake availability. Sam Altman explained the idea is to scale the thing up, and see what happens. He hadn't claimed to offer a solution to the supply problem that would unfold.
- iainmerrick 6mo agoAre you mixing up Anthropic and OpenAI here?
- hirako2000 6mo agoI didn't. Anthropic and others followed the concept of scaling up models and worry about efficiency and availability later. Sam likely didn't invent the idea but he talked about it.
- bruckie 6mo agoAre you sure it was fake scarcity for Gmail? IIRC they did it because they were worried about systems falling over if it grew too fast, and discovered the marketing benefits as a side effect.
- joquarky 6mo agoGoogle Wave demonstrated that this doesn't always work.
- sobellian 6mo agoTheir issues seem to extend well beyond inference into services like auth.
- ryandrake 6mo agoYes. Whenever these outages happen, it always seems that it's their login system that is broken.
- bostik 6mo agoThat implies that either the auth is too heavy (possible, ish) or their systems don't degrade gracefully enough and many different types of failures propagate up and out all the way to their outermost layer, ie. auth (more plausible). Disclosure: I have scars from a distributed system where errors propagated outwards and took down auth...
- scratchyone 6mo agomaybe, but the response to GPU shortages being increased error rates is the concern imo. they could implement queuing or delayed response times. it's been long enough that they've had plenty of time to implement things like this, at least on their web-ui where they have full control. instead it still just errors with no further information.
- hirako2000 6mo agoCoding is a problem solved. Claud writes the code. I edit it. I code around it. Engineer roles dead in 6 months.
- post-it 6mo ago> I edit it. I code around it. You're never gonna guess what software engineers do.
- deleted 6mo ago[deleted]
- bulbar 6mo agoBecause of the context I would think this is sarcasm, but I am not sure.
- hirako2000 6mo agoIt is.
- skeledrew 6mo agoI've been experiencing a good amount of delays (says it's taking extra time to really think, etc), and I'm using during off-peak time.
- scratchyone 6mo agoi notice that as well. most of the time when i see those it has a retry counter also and i can see it trying and failing multiple requests haha. almost never succeeds in producing a response when i see those though, eventually just errors out completely.
- zachncst 6mo agoSure but we don't need GPUs to log in.
- paulddraper 6mo agoA. These aren’t rate limit errors from the API. B. Everything is down, even auth.
- AlecSchueler 6mo ago> thus the reason why we're seeing them do other things to cut down on inference cost (ie changing their default thinking length). The dynamic thinking and response length is funny enough the best upgrade I've experienced with the service for more than a year. I really appreciate that when I say or ask something simple the answer now just comes back as a single sentence without having to manually toggle "concise" mode on and off again.