4 ms·

Today's Cheap AI Services Won't Last

22 points by vincent_s 2y ago

londons_explore 2y ago
> Price increases: As VC funding dries up and companies face pressure to turn a profit, we'll likely see sharp increases in the cost of AI services. I disagree. Yes, the VC funding will dry up, but hardware and algorithmic advances will decrease running costs by equal amounts. Lack of a moat will prevent companies recouping past expenses, since any who try will be outcompeted by new market entrants who don't have those past expenses.
- jsheard 2y ago
  This presumes that the SOTA models won't keep growing to fill any efficiency gains made. The models of today will get cheaper to run, sure, but surely the expectation is that AI will do a lot more than it does today in order to justify the staggering investments going into it?
- sigmoid10 2y ago
  >Lack of a moat will prevent companies recouping past expenses I think we'll see a lot of vendor lock in by introducing service complexity. We are already at a point where the hassle of switching from one big API provider to another is not always worth the hassle for a small reduction in price or a tiny improvement on some benchmarks. AI will go the same route as all SaaS products until we have GAI-level stuff running locally on affordable consumer hardware. And there's a lot of money to be made until then.
- fsiefken 2y ago
  Yes, the new Ryzen APU's might have 128G, great for running LLMs. https://videocardz.com/181601/amd-testing-strix-halo-apu-with-128gb-memory-config https://videocardz.com/181601/amd-testing-strix-halo-apu-wit... Perhaps some other party will come up with Groq like performant LPUs and LLMs will get more precise and efficient. I read that more and more people are running their LLMs locally.
- dscottboggs 2y ago
  Do you have some numbers to back that up or just vibes?
- deleted 2y ago
  [deleted]
- cosmicradiance 2y ago
  > Yes, the VC funding will dry up, but hardware and algorithmic advances will decrease running costs by equal amounts. The author said the same - "The current reliance on expensive GPU clusters may give way to more specialized, efficient AI hardware. This could help mitigate some cost pressures." As with the cloud, prices increase once competition dries out. Even if hardware becomes cheap, the services will cost more because shrewd and greedy boards run most businesses.
- nertirs 2y ago
  I would even argue, that a lot of AI services in the future will be close to free for the public. That is because in a lot of cases, the data received from user interactions is more valuable, than the data generated by the AI service.
- vincent_s 2y ago
  1. These companies have burned through billions in VC funding. Just breaking even won't be enough - they'll need significant profits to justify those investments. 2. We have at least one real-world example suggesting profitability is challenging. Microsoft is reportedly still losing money on GitHub Copilot, even after years of operation and many paying customers.
  - londons_explore 2y ago
    > These companies have burned through billions in VC funding. These companies, yes. But new startups will come, spend only millions, and achieve the same thing. No moat.
pedrocr 2y ago
> The current landscape bears a striking resemblance to the early days of cloud computing. Fifteen years ago, it was commonplace for individuals and businesses to pay less than $10 a month for shared hosting or a VPS. This pricing model seemed set in stone, with costs expected to continue declining. A Digital Ocean VPS starts at $4 a month. It's not so much that we've lost the cheap part of the cloud, it's that the big cloud providers figured out how to do enterprise sales and capture the part of the market that was already overpriced and wasn't sophisticated enough to optimize itself.
leke 2y ago
I think AI services will be generally free because there is too much juicy data that can be mined and a risk someone might just make it easy to make your own AI "server" so to speak.
worstspotgain 2y ago
This was a huge concern in 2022, before open models with useful IQ levels were released. Today, it's hard to imagine prohibitive prices at the end-user level, because it's hard to envision an application that won't eventually have a "good enough" open/free on-device inference implementation. Two caveats: application-specific patents are still possible (and many torpedo patents are undoubtedly en route right now) and this argument might not hold at the middleman level (where wrapper apps might get their margins squeezed out.)
- antupis 2y ago
  I would say RAG + llama3 is already in that good enough level. Claude and gpt4-o are best but when you need steady performance or a cheap interface on a scale is it very hard to compete against llama3.
gnrlst 2y ago
I have pretty much the opposite take: years from now we'll tell our grandchildren that the AI APIs we used to use were paid BY THE TOKEN! We're in the dialup days of AI, where capabilities are in the hands of very few companies because hardware and training costs are prohibitively expensive. Sure, the apps we use are heavily subsidized by investment funding and the competition is very fierce. I'll also concede that 99% of the AI startups today will fail. But that doesn't mean only the 1% will be left: new ones will continuously enter the arena, compete for attention, and to do that they'll need to lower their prices. All the while, hardware costs will decrease, and incumbents like NVIDIA will inevitably grow stagnant and others will come to eat their lunch. It's the circle of (business) life.
deleted 2y ago
[deleted]
merryboxingday 2y ago
You guys are paying for AI services? The models are literally free, just run them yourself
- verdverm 2y ago
  I pay much less by using an API, and I get faster/higher quality results, than if I paid for the graphics card(s), electricity, and cooling myself. The ROI is not there for me Even the $20/month would take a few years to recoup the costs
hexo 2y ago
Of course. At current trend its going to consume ridiculous amount of electricity for no good reason (and make some more CO2). Funny that 60W old school light bulb is bad bad bad, but 400W gfx card is ok and armies of 3kW servers to produce bs responses to queries are also ok. Consuming ridiculous kilowatts to produce even more bs video is also "no problemo". Hm. Poor artists, I can see why they're furious about this.
- Ukv 2y ago
  > At current trend its going to consume ridiculous amount of electricity for no good reason (and make some more CO2). 5 joules per token[0] * 200 tokens per query * 10 queries per day * 365 days per year = 1.014kWh Which, going by the US's energy mix[1], means a year of someone's LLM usage would be about 0.4kg of CO2 - the same as a single cup of coffee[2]. Worth double-checking, since these are just napkin calculations and I could have made a large mistake. If it is correct (within an order of magnitude or two), that seems to me a relatively small amount of energy. Obviously still expensive to provide for free to hundreds of millions of users, as almost anything would be. [0]: https://arxiv.org/pdf/2310.03003 https://arxiv.org/pdf/2310.03003 [1]: https://www.eia.gov/tools/faqs/faq.php?id=74&t=11 https://www.eia.gov/tools/faqs/faq.php?id=74&t=11 [2]: https://www.co2everything.com/co2e-of/coffee https://www.co2everything.com/co2e-of/coffee
  - hexo 2y ago
    I've read somewhere that half of googles datacenter energy consumption is ai. So I think the real numbers are probably somewhere else. Its not just about one query - its about everything that come with it, including production of hw including price manipulation (that means we [better say companies] have to produce more money [ie. make some more CO2] to even afford the hw in the first place) and soon also energy plants construction - and we can only hope thats not a coal one. Also, there's that pesky training that is hugely expensive on electricity. Oh dont get me wrong - there are good uses of neural networks, but the stuff that pushed so much is not that. An example of good one was nvidias sw (rtx voice) to clear up voice during calls, dunno if it exists or not.
cosmicradiance 2y ago
I feel that a specialized AI becomes too niche to be useful for the common folks. A general LLM model, that is "jack of all and master of none" may remain the go-to choice for the masses. These systems leave the last bits of intelligence to be filled by humans.
OhNoNotAgain_99 2y ago
[dead]

22 points by vincent_s 2y ago

> Price increases: As VC funding dries up and companies face pressure to turn a profit, we'll likely see sharp increases in the cost of AI services. I disagree. Yes, the VC funding will dry up, but hardware and algorithmic advances will decrease running costs by equal amounts. Lack of a moat will prevent companies recouping past expenses, since any who try will be outcompeted by new market entrants who don't have those past expenses.

jsheard 2y ago
This presumes that the SOTA models won't keep growing to fill any efficiency gains made. The models of today will get cheaper to run, sure, but surely the expectation is that AI will do a lot more than it does today in order to justify the staggering investments going into it?
sigmoid10 2y ago
>Lack of a moat will prevent companies recouping past expenses I think we'll see a lot of vendor lock in by introducing service complexity. We are already at a point where the hassle of switching from one big API provider to another is not always worth the hassle for a small reduction in price or a tiny improvement on some benchmarks. AI will go the same route as all SaaS products until we have GAI-level stuff running locally on affordable consumer hardware. And there's a lot of money to be made until then.
fsiefken 2y ago
Yes, the new Ryzen APU's might have 128G, great for running LLMs. https://videocardz.com/181601/amd-testing-strix-halo-apu-with-128gb-memory-config https://videocardz.com/181601/amd-testing-strix-halo-apu-wit... Perhaps some other party will come up with Groq like performant LPUs and LLMs will get more precise and efficient. I read that more and more people are running their LLMs locally.
dscottboggs 2y ago
Do you have some numbers to back that up or just vibes?
deleted 2y ago
[deleted]
cosmicradiance 2y ago
> Yes, the VC funding will dry up, but hardware and algorithmic advances will decrease running costs by equal amounts. The author said the same - "The current reliance on expensive GPU clusters may give way to more specialized, efficient AI hardware. This could help mitigate some cost pressures." As with the cloud, prices increase once competition dries out. Even if hardware becomes cheap, the services will cost more because shrewd and greedy boards run most businesses.
nertirs 2y ago
I would even argue, that a lot of AI services in the future will be close to free for the public. That is because in a lot of cases, the data received from user interactions is more valuable, than the data generated by the AI service.
vincent_s 2y ago
1. These companies have burned through billions in VC funding. Just breaking even won't be enough - they'll need significant profits to justify those investments. 2. We have at least one real-world example suggesting profitability is challenging. Microsoft is reportedly still losing money on GitHub Copilot, even after years of operation and many paying customers.
- londons_explore 2y ago
  > These companies have burned through billions in VC funding. These companies, yes. But new startups will come, spend only millions, and achieve the same thing. No moat.

pedrocr 2y ago

> The current landscape bears a striking resemblance to the early days of cloud computing. Fifteen years ago, it was commonplace for individuals and businesses to pay less than $10 a month for shared hosting or a VPS. This pricing model seemed set in stone, with costs expected to continue declining. A Digital Ocean VPS starts at $4 a month. It's not so much that we've lost the cheap part of the cloud, it's that the big cloud providers figured out how to do enterprise sales and capture the part of the market that was already overpriced and wasn't sophisticated enough to optimize itself.

leke 2y ago

I think AI services will be generally free because there is too much juicy data that can be mined and a risk someone might just make it easy to make your own AI "server" so to speak.

worstspotgain 2y ago

This was a huge concern in 2022, before open models with useful IQ levels were released. Today, it's hard to imagine prohibitive prices at the end-user level, because it's hard to envision an application that won't eventually have a "good enough" open/free on-device inference implementation. Two caveats: application-specific patents are still possible (and many torpedo patents are undoubtedly en route right now) and this argument might not hold at the middleman level (where wrapper apps might get their margins squeezed out.)

antupis 2y ago
I would say RAG + llama3 is already in that good enough level. Claude and gpt4-o are best but when you need steady performance or a cheap interface on a scale is it very hard to compete against llama3.

gnrlst 2y ago

I have pretty much the opposite take: years from now we'll tell our grandchildren that the AI APIs we used to use were paid BY THE TOKEN! We're in the dialup days of AI, where capabilities are in the hands of very few companies because hardware and training costs are prohibitively expensive. Sure, the apps we use are heavily subsidized by investment funding and the competition is very fierce. I'll also concede that 99% of the AI startups today will fail. But that doesn't mean only the 1% will be left: new ones will continuously enter the arena, compete for attention, and to do that they'll need to lower their prices. All the while, hardware costs will decrease, and incumbents like NVIDIA will inevitably grow stagnant and others will come to eat their lunch. It's the circle of (business) life.

deleted 2y ago

[deleted]

merryboxingday 2y ago

You guys are paying for AI services? The models are literally free, just run them yourself

verdverm 2y ago
I pay much less by using an API, and I get faster/higher quality results, than if I paid for the graphics card(s), electricity, and cooling myself. The ROI is not there for me Even the $20/month would take a few years to recoup the costs

hexo 2y ago

Of course. At current trend its going to consume ridiculous amount of electricity for no good reason (and make some more CO2). Funny that 60W old school light bulb is bad bad bad, but 400W gfx card is ok and armies of 3kW servers to produce bs responses to queries are also ok. Consuming ridiculous kilowatts to produce even more bs video is also "no problemo". Hm. Poor artists, I can see why they're furious about this.

Ukv 2y ago
> At current trend its going to consume ridiculous amount of electricity for no good reason (and make some more CO2). 5 joules per token[0] * 200 tokens per query * 10 queries per day * 365 days per year = 1.014kWh Which, going by the US's energy mix[1], means a year of someone's LLM usage would be about 0.4kg of CO2 - the same as a single cup of coffee[2]. Worth double-checking, since these are just napkin calculations and I could have made a large mistake. If it is correct (within an order of magnitude or two), that seems to me a relatively small amount of energy. Obviously still expensive to provide for free to hundreds of millions of users, as almost anything would be. [0]: https://arxiv.org/pdf/2310.03003 https://arxiv.org/pdf/2310.03003 [1]: https://www.eia.gov/tools/faqs/faq.php?id=74&t=11 https://www.eia.gov/tools/faqs/faq.php?id=74&t=11 [2]: https://www.co2everything.com/co2e-of/coffee https://www.co2everything.com/co2e-of/coffee
- hexo 2y ago
  I've read somewhere that half of googles datacenter energy consumption is ai. So I think the real numbers are probably somewhere else. Its not just about one query - its about everything that come with it, including production of hw including price manipulation (that means we [better say companies] have to produce more money [ie. make some more CO2] to even afford the hw in the first place) and soon also energy plants construction - and we can only hope thats not a coal one. Also, there's that pesky training that is hugely expensive on electricity. Oh dont get me wrong - there are good uses of neural networks, but the stuff that pushed so much is not that. An example of good one was nvidias sw (rtx voice) to clear up voice during calls, dunno if it exists or not.

cosmicradiance 2y ago

I feel that a specialized AI becomes too niche to be useful for the common folks. A general LLM model, that is "jack of all and master of none" may remain the go-to choice for the masses. These systems leave the last bits of intelligence to be filled by humans.

OhNoNotAgain_99 2y ago

[dead]