3 ms·

They're interesting numbers, but the linked article's cite amounts to: > ChatGPT could cost OpenAI up to $700,000 a day to run due to "expensive servers," an a

by zamnos 4y ago

They're interesting numbers, but the linked article's cite amounts to: > ChatGPT could cost OpenAI up to $700,000 a day to run due to "expensive servers," an analyst told The Information. which, pardon me, but no shit. Before I break out my back of the envelope calculator, on how many biggest GPU instances in Azure that is, the real question is what their underlying assumptions are, and where they're getting them from. Especially since OpenAI is definitely not paying list price on those GPU instances. The other question is how close to capacity their cluster is running, and how much free time on it can be reclaimed, either in terms of spinning down servers for diurnal patterns, or in terms of being able to do training runs in the background.

capableweb 4y ago
Considering that Microsoft is a huge investor in OpenAI, I'd be surprised they pay anything at all in reality.
- gabereiser 4y ago
  that's ridiculous, OpenAI is paying. Granted Microsoft invested heavily into OpenAI but those are two separate financial transactions. Sure you can rationalize in your head that IN-OUT=DIFF but that's not how books are kept.
  - capableweb 4y ago
    Why is that ridiculous? Cloud services gives companies "coupons" and free usage for X hours for a bunch of other companies, why wouldn't they do that for a company they invested heavily in?
    - rickdeckard 4y ago
      Because that's not how it works. Even company cars of General Motors employees have to be purchased from General Motors. Such "free usage" coupons are marketing activities to gain new customers, Microsoft already completed the "dating phase" with OpenAI. They surely don't pay list-price for Azure but it's surely also not free. Moreover, as per Microsoft themselves, the 1bn USD investment into OpenAI carried the condition that Azure becomes the exclusive provider for cloud-services: https://blogs.microsoft.com/blog/2023/01/23/microsoftandopenaiextendpartnership/ https://blogs.microsoft.com/blog/2023/01/23/microsoftandopen... It's not exclusive because it's free, it's exclusive because "we paid you 1bn USD to buy it from us"
      - mattpallissard 4y ago
        I've personally worked on a project where Microsoft ate the cloud cost in order partner with us. They might not give unfettered credits, it could be for specific projects. That said, I wouldn't be surprised if it was unfettered either.
  - nerdix 4y ago
    Microsoft invested $1 billion OpenAI in 2019 and half of that amount was in Azure credits. I'm not sure about the most recent $10 billion investment but I wouldn't be surprised if a significant amount of it is in Azure credits as well. While that's not "free" (they exchanged equity for it), it's likely not an expense (or at least not an expense that they have to cover fully).
- dmix 4y ago
  At best OpenAI has negotiated a near 0 profit margin for Microsoft when paying for the services. But even that is unlikely given how much money/resources are involved. There's no scenario where it's free at that scale.
vitus 4y ago
The estimate in the article pins most of the cost on inference, not training, so diurnal patterns are unfortunately not as useful here. > While training ChatGPT's large language models likely costs tens of millions of dollars, operational expenses, or inference costs, "far exceed training costs when deploying a model at any reasonable scale," Patel and Afzal Ahmad, another analyst at SemiAnalysis, told Forbes. "In fact, the costs to inference ChatGPT exceed the training costs on a weekly basis," they said.
- zamnos 4y ago
  Why wouldn't inference follow diurnal patterns?
  - reaperman 4y ago
    Agreed. That seems backwards. Training would not follow circadian rhythms, inference would.
    - vitus 4y ago
      I should clarify: training is not latency sensitive, so you can run your workloads at off-peak hours when people are asleep. Inference means you need to run your workloads at peak when people are using your service. (Looking back, I'm happy that I was careful in my wording in that I didn't say diurnal cycles aren't relevant, just that they aren't as useful in this case) That said, I suppose I misread the specific suggestion about spinning down servers off-peak and was thinking more about spot pricing at peak vs trough.

by zamnos 4y ago

capableweb 4y ago

Considering that Microsoft is a huge investor in OpenAI, I'd be surprised they pay anything at all in reality.

gabereiser 4y ago
that's ridiculous, OpenAI is paying. Granted Microsoft invested heavily into OpenAI but those are two separate financial transactions. Sure you can rationalize in your head that IN-OUT=DIFF but that's not how books are kept.
- capableweb 4y ago
  Why is that ridiculous? Cloud services gives companies "coupons" and free usage for X hours for a bunch of other companies, why wouldn't they do that for a company they invested heavily in?
  - rickdeckard 4y ago
    Because that's not how it works. Even company cars of General Motors employees have to be purchased from General Motors. Such "free usage" coupons are marketing activities to gain new customers, Microsoft already completed the "dating phase" with OpenAI. They surely don't pay list-price for Azure but it's surely also not free. Moreover, as per Microsoft themselves, the 1bn USD investment into OpenAI carried the condition that Azure becomes the exclusive provider for cloud-services: https://blogs.microsoft.com/blog/2023/01/23/microsoftandopenaiextendpartnership/ https://blogs.microsoft.com/blog/2023/01/23/microsoftandopen... It's not exclusive because it's free, it's exclusive because "we paid you 1bn USD to buy it from us"
    - mattpallissard 4y ago
      I've personally worked on a project where Microsoft ate the cloud cost in order partner with us. They might not give unfettered credits, it could be for specific projects. That said, I wouldn't be surprised if it was unfettered either.
- nerdix 4y ago
  Microsoft invested $1 billion OpenAI in 2019 and half of that amount was in Azure credits. I'm not sure about the most recent $10 billion investment but I wouldn't be surprised if a significant amount of it is in Azure credits as well. While that's not "free" (they exchanged equity for it), it's likely not an expense (or at least not an expense that they have to cover fully).
dmix 4y ago
At best OpenAI has negotiated a near 0 profit margin for Microsoft when paying for the services. But even that is unlikely given how much money/resources are involved. There's no scenario where it's free at that scale.

vitus 4y ago

The estimate in the article pins most of the cost on inference, not training, so diurnal patterns are unfortunately not as useful here. > While training ChatGPT's large language models likely costs tens of millions of dollars, operational expenses, or inference costs, "far exceed training costs when deploying a model at any reasonable scale," Patel and Afzal Ahmad, another analyst at SemiAnalysis, told Forbes. "In fact, the costs to inference ChatGPT exceed the training costs on a weekly basis," they said.

zamnos 4y ago
Why wouldn't inference follow diurnal patterns?
- reaperman 4y ago
  Agreed. That seems backwards. Training would not follow circadian rhythms, inference would.
  - vitus 4y ago
    I should clarify: training is not latency sensitive, so you can run your workloads at off-peak hours when people are asleep. Inference means you need to run your workloads at peak when people are using your service. (Looking back, I'm happy that I was careful in my wording in that I didn't say diurnal cycles aren't relevant, just that they aren't as useful in this case) That said, I suppose I misread the specific suggestion about spinning down servers off-peak and was thinking more about spot pricing at peak vs trough.