3 ms·

No way you need $3,800 to run an 8B model. 3090 and a basic rig is enough. That being said, the difference between OpenAI and AWS cost ($1 vs $17) is huge. Is

by causal 2y ago

No way you need $3,800 to run an 8B model. 3090 and a basic rig is enough. That being said, the difference between OpenAI and AWS cost ($1 vs $17) is huge. Is OpenAI just operating at a massive loss? Edit: Turns out AWS is actually cheaper if you don't use the terrible setup in this article, see comments below.

throwup238 2y ago
AWS's pricing is just ridiculous. Their 1-year reserve pricing for an 8x H100 or A100 instance (p4/p5) costs just as much as buying the machine outright with tens of thousands left over for the NVIDIA enterprise license and someone to manage them (per instance!). Their on demand pricing is even more insane - they're charging $3.x/hr for six year old cards.
- readams 2y ago
  What about the cost of the power and cooling to run the machine (a lot!), and the staff to keep it running?
  - throwup238 2y ago
    That's why I said "and someone to manage them". The difference is in the tens of thousands of dollars per instance. The savings from even a dozen instances is enough to pay for someone to manage them full time, and that's just for the first year. Year 2 and 3 you're saving six figures per instance so you'd be able to afford one person per machine to hand massage them like some fancy kobe beef. A100 TDP is 400W so assuming 4kW for the whole machine, that's a little more than $5k/year at $0.15/kWh. Again, the difference is in the tens of thousands per instance. Even at 50% utilization over three years, if you need more than a dozen machines it's much cheaper to buy them outright, especially on credit.
refulgentis 2y ago
I mean, no, I came to scan the comments quick after reading because there's a lot of bad info you can walk away with from the post, it's sort of starting from scratch on hosting LLMs If you keep reading past there, they get it down significantly. The 8 tkn/s number AWS was evaluated on is really funny, that's about what you'd get on last year's iphone and it's not because apples special, it's because theres barely any reasonable optimization being done here. No batching, float32 weights (8 bit is guaranteed indistinguishable from 32 bit, 5 bit tests as definitely indistinguishable in blind tests, 4 bit arguably is indistinguishable)
- causal 2y ago
  You're right. In fact, using EKS at all is silly when AWS offers their Bedrock service with Claude Haiku (rated #19 on Chat Arena vs. ChatGPT3.5-Turbo at #37) for a much lower cost of $0.75/M tokens (averaging input and output like OP does)[0]. So in reality AWS is cheaper for a much better model if you don't go with a wildly suboptimal setup. [0] https://aws.amazon.com/bedrock/pricing/ https://aws.amazon.com/bedrock/pricing/
throwaway240403 2y ago
I thought it was generally known they were operating at a loss? even with the subs and api charges, they still let people use chatGPT for free with no monetization options. Sure they are collecting the data for training, but that's hard to quantify the value of.
thot_experiment 2y ago
yeah what, a GV100 is $1600 on eBay and you can run like a 3.8bpw quant of 70b llama at some decent number of tk/s, if you buy two you can run a 70b 5bpw quant with 32k context no problem

by causal 2y ago

throwup238 2y ago

AWS's pricing is just ridiculous. Their 1-year reserve pricing for an 8x H100 or A100 instance (p4/p5) costs just as much as buying the machine outright with tens of thousands left over for the NVIDIA enterprise license and someone to manage them (per instance!). Their on demand pricing is even more insane - they're charging $3.x/hr for six year old cards.

readams 2y ago
What about the cost of the power and cooling to run the machine (a lot!), and the staff to keep it running?
- throwup238 2y ago
  That's why I said "and someone to manage them". The difference is in the tens of thousands of dollars per instance. The savings from even a dozen instances is enough to pay for someone to manage them full time, and that's just for the first year. Year 2 and 3 you're saving six figures per instance so you'd be able to afford one person per machine to hand massage them like some fancy kobe beef. A100 TDP is 400W so assuming 4kW for the whole machine, that's a little more than $5k/year at $0.15/kWh. Again, the difference is in the tens of thousands per instance. Even at 50% utilization over three years, if you need more than a dozen machines it's much cheaper to buy them outright, especially on credit.

refulgentis 2y ago

I mean, no, I came to scan the comments quick after reading because there's a lot of bad info you can walk away with from the post, it's sort of starting from scratch on hosting LLMs If you keep reading past there, they get it down significantly. The 8 tkn/s number AWS was evaluated on is really funny, that's about what you'd get on last year's iphone and it's not because apples special, it's because theres barely any reasonable optimization being done here. No batching, float32 weights (8 bit is guaranteed indistinguishable from 32 bit, 5 bit tests as definitely indistinguishable in blind tests, 4 bit arguably is indistinguishable)

causal 2y ago
You're right. In fact, using EKS at all is silly when AWS offers their Bedrock service with Claude Haiku (rated #19 on Chat Arena vs. ChatGPT3.5-Turbo at #37) for a much lower cost of $0.75/M tokens (averaging input and output like OP does)[0]. So in reality AWS is cheaper for a much better model if you don't go with a wildly suboptimal setup. [0] https://aws.amazon.com/bedrock/pricing/ https://aws.amazon.com/bedrock/pricing/

throwaway240403 2y ago

I thought it was generally known they were operating at a loss? even with the subs and api charges, they still let people use chatGPT for free with no monetization options. Sure they are collecting the data for training, but that's hard to quantify the value of.

thot_experiment 2y ago

yeah what, a GV100 is $1600 on eBay and you can run like a 3.8bpw quant of 70b llama at some decent number of tk/s, if you buy two you can run a 70b 5bpw quant with 32k context no problem