6 ms·

Practical report: the OpenAI API is a bad joke. If you think you can build a production app against it, think again. I've been trying to use it for the past 6 w

by jwr 3y ago

Practical report: the OpenAI API is a bad joke. If you think you can build a production app against it, think again. I've been trying to use it for the past 6 weeks or so. If you use tiny prompts, you'll generally be fine (that's why you always get people commenting that it works for them), but just try to get closer to the limits, especially with GPT-4. The API will make you wait up to 10 minutes, and then time out. What's worse, it will time out between their edge servers (cloudflare) and their internal servers, and the way OpenAI implemented their billing you will get a 4xx/5xx response code, but you will still get billed for the request and whatever the servers generated and you didn't get. That's borderline fraudulent. Meanwhile, their status page will happily show all green, so don't believe that. It seems to be manually updated and does not reflect the truth. Could it be that it works better in another region? Could it be just my region that is affected? Perhaps — but I won't know, because support is non-existent and hidden behind a moat. You need to jump through hoops and talk to bots, and then you eventually get a bot reply. That you can't respond to. My support requests about being charged for data I didn't have a chance to get have been unanswered for more than 5 weeks now. There is no way to contact OpenAI, no way to report problems, the API sometimes kind-of works, but mostly doesn't, and if you comment in the developer forums, you'll mostly get replies from apologists that explain that OpenAI is "growing quickly". I'd say you either provide a production paid API or you don't. At the moment, this looks very much like amateur hour, and charging for requests that were never fulfilled seems like a fraud to me. So, consider carefully whether you want to build against all that.

messe 3y ago
I’m only using them as a stop-gap / for prototyping with the intent to move to a locally hosted fine-tuned (and ideally 7B parameter) model further down the road.
nunodonato 3y ago
if you want to use it in prod, go with Azure
- hobs 3y ago
  And get only 20 K tokens per minute, where a decent size question can use up 500 tokens, pretty much a joke for most larger websites. https://learn.microsoft.com/en-us/azure/cognitive-services/openai/quotas-limits https://learn.microsoft.com/en-us/azure/cognitive-services/o...
  - weird-eye-issue 3y ago
    That's the default limit for GPT-4 which has more demand than any other LLM in the world.
    - hobs 3y ago
      Which is just demonstrating my point, just saying "go use Azure" doesn't solve anything.
      - weird-eye-issue 3y ago
        Yeah for GPT-4 they aren't even accepting new customers
feoren 3y ago
> you will get a 4xx/5xx response code, but you will still get billed for the request and whatever the servers generated and you didn't get. That's borderline fraudulent. Borderline!? They're regularly charging customers for products they know weren't delivered. That sounds like straight-up fraud to me, no borderline about it.
- oaktowner 3y ago
  Sounds positively Muskian.
  - KennyBlanken 3y ago
    You mean it's not normal to tell people that it's their fault for driving their $80,000 electric car in heavy rain, because for many years you haven't bothered to properly seal your transmission's speed sensor?
    - oaktowner 3y ago
      LOL. I meant it's not normal to start selling a feature in 2016 and delivering it in beta seven years later.
throwaway9274 3y ago
The click through API is mainly for prototyping. If you want better latency and sane billing you need to go through Azure OpenAI Services. OpenAI also offers decreased latency under the Enterprise Agreement.
refulgentis 3y ago
I understand your general point and am sympathetic to it, if you're a 10/10 on some scale, I'm about a 3-4. I've never seen billings for failures, but the billing stuff is crazy: no stats if you do streamed chat, and the only tokenizer available is in Python and for GPT-3.0. However, I'm virtually certain somethings wrong on your end, I've never seen a wait even close to that unless it was completely down. Also the thing about "small prompts"...it sounds to me like you're overflowing context, they're returning an error, and somethings retrying.
skilled 3y ago
I can vouch on this. GPT4 API dies a lot if you use it for a big concurrent project. And of course it’s rate limited like crazy, with certain hours being so bad you can’t even run it for any business purpose.
ericlewis 3y ago
[flagged]
- dang 3y ago
  Can you please not post in the flamewar style? We're trying for something else here and you can make your substantive points without it. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
athyuttamre 3y ago
(I'm an engineer at OpenAI) Very sorry to hear about these issues, particularly the timeouts. Latency is top of mind for us and something we are continuing to push on. Does streaming work for your use case? https://github.com/openai/openai-cookbook/blob/main/examples/How_to_stream_completions.ipynb https://github.com/openai/openai-cookbook/blob/main/examples... We definitely want to investigate these and the billing issues further. Would you consider emailing me your org ID and any request IDs (if you have them) at atty@openai.com? Thank you for using the API, and really appreciate the honest feedback.
- renewiltord 3y ago
  Quick note: your domain doesn't appear to have an A record. I was hoping to follow the link in your profile and see if you have anything interesting written about LLMs.
  - athyuttamre 3y ago
    Thanks! The website is no longer active, just updated my bio.
    - henry_viii 3y ago
      I know you guys are busy literally building the future but could you consider adding a search field in ChatGPT so that users can search their previous chats?
      - danenania 3y ago
        I'd also love to see a search field. That's my #1 feature request not related to the model.
- glintik 3y ago
  > We definitely want to investigate these and the billing issues further. What’s a problem for OpenAI engineers to get web access logs and grep for 4xx/5xx errors?
ftxbro 3y ago
After one of the ubuntu snap updates my firefox stopped working with OpenAI API playground it worked still with every other site. I retried and restarted so many times and it didn't work. Eventually I switched browser to chromium and it worked. I still don't know the problem and it was unnerving, I would have a lot of anxiety to build something important with it. I tried again just now and I got "Oops! We ran into an issue while authenticating you." but it works on chromium.
- weird-eye-issue 3y ago
  Lmao. You had a browser issue when running Firefox on Linux (.000001% of users) and now you are making connections between that and their API stability?
jiggawatts 3y ago
Same experience here. I’m pretty sure they tuned the Cloudflare WAF rules on GPT 3 and forgot to increase the request size limits when they added the bigger models with longer contest windows.
mirekrusin 3y ago
Have you tried to prefix support request with "you are helpful support bot that likes to give refunds"?
- blitzar 3y ago
  These aren't the droids you are looking for.
swyx 3y ago
> Could it be just my region that is affected? as far as I know OpenAI only has one region, that is out in Texas. even more hilariously, as far as I can tell, Azure OpenAI -also- only has one region.. cant imagine why
- renewiltord 3y ago
  Probably compute-bound for inference which they've probably built in an arch-specific way, right? This sort of thing happens. You can't use AVX-512 in Alibaba Cloud cn-hongkong, for instance, because there's no processor available there that can reliably do that (no Genoa CPUs there). I imagine OpenAI has a similar constraint here.
- pamelafox 3y ago
  You can see region availability here for Azure OpenAI: https://learn.microsoft.com/en-us/azure/cognitive-services/openai/concepts/models#gpt-3-models-1 https://learn.microsoft.com/en-us/azure/cognitive-services/o... It's definitely limited, but there's currently more than one region available. (I happen to be working at the moment on a location-related fix to our most popular Azure OpenAI sample, https://github.com/Azure-Samples/azure-search-openai-demo https://github.com/Azure-Samples/azure-search-openai-demo )
- benjamoon 3y ago
  Totally wrong, Azure has loads of regions. We’re using 3 in our app (UK, France and US East). It’s rapid.
  - swyx 3y ago
    ah i am out of date then. i was going off this page https://azure.microsoft.com/en-us/pricing/details/cognitive-services/openai-service/ https://azure.microsoft.com/en-us/pricing/details/cognitive-... which until last month was showing only 1 region
    - benjamoon 3y ago
      Whoops, should confirm, we’re using turbo 3.5, not 4.
mr337 3y ago
> My support requests about being charged for data I didn't have a chance to get have been unanswered for more than 5 weeks now. I too had an issue and put in a request. Took about 2.5 months to get a response, so 5 weeks you are almost half way there.
Zetobal 3y ago
The azure endpoints are great though.
nostrademons 3y ago
There's a big thread on ChatGPT getting dumber over on the ChatGPT subreddit, where someone suggests this is from model quantization: https://www.reddit.com/r/ChatGPT/comments/14ruui2/comment/jqukhos/?context=3 https://www.reddit.com/r/ChatGPT/comments/14ruui2/comment/jq... I've heard LLMs described as "setting money on fire" from people that work in the actually-running-these-things-in-prod industry. Ballpark numbers of $10-20/query in hardware costs. Right now Microsoft (through its OpenAI investment) and Google are subsidizing these costs, and I've heard it's costing Microsoft literally billions a year. But both companies are clearly betting on hardware or software breakthroughs to bring the cost down. If it doesn't come down there's a good chance that it'll remain more economical to pay someone in the Philippines or India to write all the stuff you would have ChatGPT write.
- driscoll42 3y ago
  $10-$20 per query? Can I get some sourcing on that? That's astronomically expensive.
  - sebmellen 3y ago
    I would presume that number includes the amortized training cost.
  - swyx 3y ago
    yeah this isnt close. Sam Altman is on record saying its single digit cents per query and then took a massively dilutive $10b investment from microsoft. Even if gpt4 is 8 models in a trenchcoat they wouldnt raise it on themselves by 4 orders of magnitude like that
    - vander_elst 3y ago
      Single digit cents per query (let's say 2) is A LOT. Let's say the service runs at 10krps (made up, we can discuss about this) it means the service costs 200$ a second i.e 20M$ a day (oversimplifying a day with 100k seconds, but this might be ok to get us in the ballpark), which means that running the model for a year (400 days, sorry simplifying) is around 8B$, so too run 10krps we are in the order of billions per year. We can discuss some of the assumptions but I think that of we are in the ballpark of cents per query the infrastructure costs are significant.
KennyBlanken 3y ago
> the way OpenAI implemented their billing you will get a 4xx/5xx response code, but you will still get billed for the request and whatever the servers generated and you didn't get. That's borderline fraudulent. It's fraudulent, full stop. Maybe they're able to weasel out of it with credit card companies because you're buying "credits." I suspect it was done this way out of pure incompetence; the OpenAI team handling the customer-facing infrastructure have a pretty poor history. Far as I know you still can't do something simple like change your email address.
phillipcarter 3y ago
FWIW we have a live product for all users against gpt-3.5-turbo and it's largely fine: https://www.honeycomb.io/blog/improving-llms-production-observability https://www.honeycomb.io/blog/improving-llms-production-obse... In our own tracking, the P99 isn't exactly great, but this is groundbreaking tech we're dealing with here, and our dissatisfaction with the high end of latency is well worth the value we get in our product: https://twitter.com/_cartermp/status/1674092825053655040/ https://twitter.com/_cartermp/status/1674092825053655040/
benjamoon 3y ago
You should apply and use OpenAI on azure. We’ve got close to 1m tokens per minute capacity across 3 instances and the latency is totally fine, like 800ms average (with big prompts). They’ve just got the new 0613 models as well (they seem to be about 2 weeks behind OpenAI). We’ve been in production for about 3 months, have some massive clients with a lot traffic and our gpt bill is way under £100 per month. This is all 3.5 turbo though, not 4 (but that’s available on application, but we don’t need it).
weird-eye-issue 3y ago
I built a production app on top of OpenAI and yeah there are frequent errors and timeouts but you literally just have to add some code to account for these and it works fine... For example exponential back off is the first step, then adding retrying on timeouts (we use streaming and if there are 30 seconds in between getting data back we retry the whole request - rare but happens), then fixing anything else that pops up It is possible to have a stable production app on top of it Just fix your code and stop expecting OpenAI to hold your hand
- jwr 3y ago
  > Just fix your code and stop expecting OpenAI to hold your hand :-) My code does retry and the entire application is written to detect and work around breakage. But eventually I do need to get enough content from OpenAI API to be able to make progress, and I am not. At the moment, for example, all requests just time out after 12 minutes (on my side). No amount of "fixing my code" will help, and I don't want OpenAI to hold my hand, I just want it to a) return some data at least sometimes, b) not charge me for data not delivered. Let's look at my billing page: over the last hour it shows 8 requests. A total of 52584 tokens. Not a single response made it back to me.
  - weird-eye-issue 3y ago
    Something is seriously wrong with your network (or your code). That's it that's your answer. It's not OpenAI's fault. We spend over $20k/mo with them and don't have any issues like this. 12 minute timeouts make no sense because first of all why are you even waiting 12 minutes? Get off HN and go fix your shit
    - jwr 3y ago
      I am very happy it works for you! But that does not imply that it works for everyone. Wish you all the best.
      - weird-eye-issue 3y ago

by jwr 3y ago

messe 3y ago

I’m only using them as a stop-gap / for prototyping with the intent to move to a locally hosted fine-tuned (and ideally 7B parameter) model further down the road.

nunodonato 3y ago

if you want to use it in prod, go with Azure

hobs 3y ago
And get only 20 K tokens per minute, where a decent size question can use up 500 tokens, pretty much a joke for most larger websites. https://learn.microsoft.com/en-us/azure/cognitive-services/openai/quotas-limits https://learn.microsoft.com/en-us/azure/cognitive-services/o...
- weird-eye-issue 3y ago
  That's the default limit for GPT-4 which has more demand than any other LLM in the world.
  - hobs 3y ago
    Which is just demonstrating my point, just saying "go use Azure" doesn't solve anything.
    - weird-eye-issue 3y ago
      Yeah for GPT-4 they aren't even accepting new customers

feoren 3y ago

> you will get a 4xx/5xx response code, but you will still get billed for the request and whatever the servers generated and you didn't get. That's borderline fraudulent. Borderline!? They're regularly charging customers for products they know weren't delivered. That sounds like straight-up fraud to me, no borderline about it.

oaktowner 3y ago
Sounds positively Muskian.
- KennyBlanken 3y ago
  You mean it's not normal to tell people that it's their fault for driving their $80,000 electric car in heavy rain, because for many years you haven't bothered to properly seal your transmission's speed sensor?
  - oaktowner 3y ago
    LOL. I meant it's not normal to start selling a feature in 2016 and delivering it in beta seven years later.

throwaway9274 3y ago

The click through API is mainly for prototyping. If you want better latency and sane billing you need to go through Azure OpenAI Services. OpenAI also offers decreased latency under the Enterprise Agreement.

refulgentis 3y ago

I understand your general point and am sympathetic to it, if you're a 10/10 on some scale, I'm about a 3-4. I've never seen billings for failures, but the billing stuff is crazy: no stats if you do streamed chat, and the only tokenizer available is in Python and for GPT-3.0. However, I'm virtually certain somethings wrong on your end, I've never seen a wait even close to that unless it was completely down. Also the thing about "small prompts"...it sounds to me like you're overflowing context, they're returning an error, and somethings retrying.

skilled 3y ago

I can vouch on this. GPT4 API dies a lot if you use it for a big concurrent project. And of course it’s rate limited like crazy, with certain hours being so bad you can’t even run it for any business purpose.

ericlewis 3y ago

[flagged]

dang 3y ago
Can you please not post in the flamewar style? We're trying for something else here and you can make your substantive points without it. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html

athyuttamre 3y ago

(I'm an engineer at OpenAI) Very sorry to hear about these issues, particularly the timeouts. Latency is top of mind for us and something we are continuing to push on. Does streaming work for your use case? https://github.com/openai/openai-cookbook/blob/main/examples/How_to_stream_completions.ipynb https://github.com/openai/openai-cookbook/blob/main/examples... We definitely want to investigate these and the billing issues further. Would you consider emailing me your org ID and any request IDs (if you have them) at atty@openai.com? Thank you for using the API, and really appreciate the honest feedback.

renewiltord 3y ago
Quick note: your domain doesn't appear to have an A record. I was hoping to follow the link in your profile and see if you have anything interesting written about LLMs.
- athyuttamre 3y ago
  Thanks! The website is no longer active, just updated my bio.
  - henry_viii 3y ago
    I know you guys are busy literally building the future but could you consider adding a search field in ChatGPT so that users can search their previous chats?
    - danenania 3y ago
      I'd also love to see a search field. That's my #1 feature request not related to the model.
glintik 3y ago
> We definitely want to investigate these and the billing issues further. What’s a problem for OpenAI engineers to get web access logs and grep for 4xx/5xx errors?

ftxbro 3y ago

After one of the ubuntu snap updates my firefox stopped working with OpenAI API playground it worked still with every other site. I retried and restarted so many times and it didn't work. Eventually I switched browser to chromium and it worked. I still don't know the problem and it was unnerving, I would have a lot of anxiety to build something important with it. I tried again just now and I got "Oops! We ran into an issue while authenticating you." but it works on chromium.

weird-eye-issue 3y ago
Lmao. You had a browser issue when running Firefox on Linux (.000001% of users) and now you are making connections between that and their API stability?

jiggawatts 3y ago

Same experience here. I’m pretty sure they tuned the Cloudflare WAF rules on GPT 3 and forgot to increase the request size limits when they added the bigger models with longer contest windows.

mirekrusin 3y ago

Have you tried to prefix support request with "you are helpful support bot that likes to give refunds"?

blitzar 3y ago
These aren't the droids you are looking for.

swyx 3y ago

> Could it be just my region that is affected? as far as I know OpenAI only has one region, that is out in Texas. even more hilariously, as far as I can tell, Azure OpenAI -also- only has one region.. cant imagine why

renewiltord 3y ago
Probably compute-bound for inference which they've probably built in an arch-specific way, right? This sort of thing happens. You can't use AVX-512 in Alibaba Cloud cn-hongkong, for instance, because there's no processor available there that can reliably do that (no Genoa CPUs there). I imagine OpenAI has a similar constraint here.
pamelafox 3y ago
You can see region availability here for Azure OpenAI: https://learn.microsoft.com/en-us/azure/cognitive-services/openai/concepts/models#gpt-3-models-1 https://learn.microsoft.com/en-us/azure/cognitive-services/o... It's definitely limited, but there's currently more than one region available. (I happen to be working at the moment on a location-related fix to our most popular Azure OpenAI sample, https://github.com/Azure-Samples/azure-search-openai-demo https://github.com/Azure-Samples/azure-search-openai-demo )
benjamoon 3y ago
Totally wrong, Azure has loads of regions. We’re using 3 in our app (UK, France and US East). It’s rapid.
- swyx 3y ago
  ah i am out of date then. i was going off this page https://azure.microsoft.com/en-us/pricing/details/cognitive-services/openai-service/ https://azure.microsoft.com/en-us/pricing/details/cognitive-... which until last month was showing only 1 region
  - benjamoon 3y ago
    Whoops, should confirm, we’re using turbo 3.5, not 4.

mr337 3y ago

> My support requests about being charged for data I didn't have a chance to get have been unanswered for more than 5 weeks now. I too had an issue and put in a request. Took about 2.5 months to get a response, so 5 weeks you are almost half way there.

Zetobal 3y ago

The azure endpoints are great though.

nostrademons 3y ago

There's a big thread on ChatGPT getting dumber over on the ChatGPT subreddit, where someone suggests this is from model quantization: https://www.reddit.com/r/ChatGPT/comments/14ruui2/comment/jqukhos/?context=3 https://www.reddit.com/r/ChatGPT/comments/14ruui2/comment/jq... I've heard LLMs described as "setting money on fire" from people that work in the actually-running-these-things-in-prod industry. Ballpark numbers of $10-20/query in hardware costs. Right now Microsoft (through its OpenAI investment) and Google are subsidizing these costs, and I've heard it's costing Microsoft literally billions a year. But both companies are clearly betting on hardware or software breakthroughs to bring the cost down. If it doesn't come down there's a good chance that it'll remain more economical to pay someone in the Philippines or India to write all the stuff you would have ChatGPT write.

driscoll42 3y ago
$10-$20 per query? Can I get some sourcing on that? That's astronomically expensive.
- sebmellen 3y ago
  I would presume that number includes the amortized training cost.
- swyx 3y ago
  yeah this isnt close. Sam Altman is on record saying its single digit cents per query and then took a massively dilutive $10b investment from microsoft. Even if gpt4 is 8 models in a trenchcoat they wouldnt raise it on themselves by 4 orders of magnitude like that
  - vander_elst 3y ago
    Single digit cents per query (let's say 2) is A LOT. Let's say the service runs at 10krps (made up, we can discuss about this) it means the service costs 200$ a second i.e 20M$ a day (oversimplifying a day with 100k seconds, but this might be ok to get us in the ballpark), which means that running the model for a year (400 days, sorry simplifying) is around 8B$, so too run 10krps we are in the order of billions per year. We can discuss some of the assumptions but I think that of we are in the ballpark of cents per query the infrastructure costs are significant.

KennyBlanken 3y ago

> the way OpenAI implemented their billing you will get a 4xx/5xx response code, but you will still get billed for the request and whatever the servers generated and you didn't get. That's borderline fraudulent. It's fraudulent, full stop. Maybe they're able to weasel out of it with credit card companies because you're buying "credits." I suspect it was done this way out of pure incompetence; the OpenAI team handling the customer-facing infrastructure have a pretty poor history. Far as I know you still can't do something simple like change your email address.

phillipcarter 3y ago

FWIW we have a live product for all users against gpt-3.5-turbo and it's largely fine: https://www.honeycomb.io/blog/improving-llms-production-observability https://www.honeycomb.io/blog/improving-llms-production-obse... In our own tracking, the P99 isn't exactly great, but this is groundbreaking tech we're dealing with here, and our dissatisfaction with the high end of latency is well worth the value we get in our product: https://twitter.com/_cartermp/status/1674092825053655040/ https://twitter.com/_cartermp/status/1674092825053655040/

benjamoon 3y ago

You should apply and use OpenAI on azure. We’ve got close to 1m tokens per minute capacity across 3 instances and the latency is totally fine, like 800ms average (with big prompts). They’ve just got the new 0613 models as well (they seem to be about 2 weeks behind OpenAI). We’ve been in production for about 3 months, have some massive clients with a lot traffic and our gpt bill is way under £100 per month. This is all 3.5 turbo though, not 4 (but that’s available on application, but we don’t need it).

weird-eye-issue 3y ago

I built a production app on top of OpenAI and yeah there are frequent errors and timeouts but you literally just have to add some code to account for these and it works fine... For example exponential back off is the first step, then adding retrying on timeouts (we use streaming and if there are 30 seconds in between getting data back we retry the whole request - rare but happens), then fixing anything else that pops up It is possible to have a stable production app on top of it Just fix your code and stop expecting OpenAI to hold your hand

jwr 3y ago
> Just fix your code and stop expecting OpenAI to hold your hand :-) My code does retry and the entire application is written to detect and work around breakage. But eventually I do need to get enough content from OpenAI API to be able to make progress, and I am not. At the moment, for example, all requests just time out after 12 minutes (on my side). No amount of "fixing my code" will help, and I don't want OpenAI to hold my hand, I just want it to a) return some data at least sometimes, b) not charge me for data not delivered. Let's look at my billing page: over the last hour it shows 8 requests. A total of 52584 tokens. Not a single response made it back to me.
- weird-eye-issue 3y ago
  Something is seriously wrong with your network (or your code). That's it that's your answer. It's not OpenAI's fault. We spend over $20k/mo with them and don't have any issues like this. 12 minute timeouts make no sense because first of all why are you even waiting 12 minutes? Get off HN and go fix your shit
  - jwr 3y ago
    I am very happy it works for you! But that does not imply that it works for everyone. Wish you all the best.
    - weird-eye-issue 3y ago