4 ms·

Something I haven’t been able to reconcile: If AI makes software easier to create, that will drive the price down. How are software companies going to make enou

Something I haven’t been able to reconcile: If AI makes software easier to create, that will drive the price down. How are software companies going to make enough revenue to pay for AI, when the amount of money being spent on AI is already multiples of the current total global expenditure on software? This demand for RAM is built on a foundation of sand, there will be a glut of capacity when it all shakes out.

locknitpicker 6mo ago
> If AI makes software easier to create, that will drive the price down. Supposedly AI drives down the cost of producing software,not the "price". > How are software companies going to make enough revenue to pay for AI, when the amount of money being spent on AI is already multiples of the current total global expenditure on software? Currently, the cost of AI is between $20/month and around $200/month per developer. I think the huge billions you're seeing in the news are the investment cost on AI companies, who are burning through cash to invest in compute infrastructure to allow both training and serving users. > This demand for RAM is built on a foundation of sand, there will be a glut of capacity when it all shakes out. Who knows? What I know is that I need >64GB of RAM to run local models, and that means most people will need to upgrade from their 8Gb/16GB setup to do the same. Graphics cards follow mostly the same pattern.
- zozbot234 6mo ago
  > Who knows? What I know is that I need >64GB of RAM to run local models, and that means most people will need to upgrade from their 8Gb/16GB setup to do the same. Graphics cards follow mostly the same pattern. Depends how big the models are, how fast you want them to run and how much context you need for your usage. If you're okay with running only smaller models (which are still very capable in general, their main limitation is world knowledge) making very simple inferences at low overall throughput, you can just repurpose the RAM, CPUs/iGPUs and storage in the average setup.
- adrian_b 6mo ago
  You need >64 GB of DRAM to run local models fast. You can run huge local models slowly with the weights stored on SSDs. Nowadays there are many computers that can have e.g. 2 PCIe 5.0 SSDs, which allow a reading throughput of 20 to 30 gigabyte per second, depending on the SSDs (or 1 PCIe 5.0 + 1 PCIe 4.0, for a throughput in the range 15-20 GB/s). There are still a lot of improvements that can be done to inference back-ends like llama.cpp to reach the inference speed limit determined by the SSD throughput. It seems that it is possible to reach inference speed in the range from a few seconds per token to a few tokens per second. That may be too slow for a chat, but it should be good enough for an AI coding assistant, especially if many tasks are batched, so that they can progress simultaneously during a single read pass over the SSD data.
  - zozbot234 6mo ago
    You can do that, but you're going to have rather low throughput unless you have lots of PCIe lanes to attach storage to. That's going to require either a HEDT or some kind of compute cluster. Batching inferences doesn't necessarily help that much since as models get sparser the individual inferences are going to share fewer experts. It does always help wrt. shared routing layers, of course.
- tomaskafka 6mo ago
  I got a 128 GB MBP, and the current models are fit enough to manage the calendar or do research on web (very slowly), not to be useful companions for coding as I hoped.
repeekad 6mo ago
Analyst says it would require a new $35/month subscription for every iPhone user, or a new $180/month for every Netflix subscriber. Claude Max subscriptions have gone up, but do you think every Netflix user will pay for one?.. https://www.tomshardware.com/tech-industry/artificial-intelligence/usd650-billion-in-annual-revenue-required-to-deliver-10-percent-return-on-ai-buildout-investment-j-p-morgan-claims-equivalent-to-usd35-payment-from-every-iphone-user-or-usd180-from-every-netflix-subscriber-in-perpetuity https://www.tomshardware.com/tech-industry/artificial-intell...
- avidiax 6mo ago
  This is probably the wrong denominator. There are more than a billion white-collar workers. Making them all just 10% more effective would possibly be worth more than $650B/year (~$650/worker).
lofaszvanitt 6mo ago
It's a trojan horse. They expect the world will get hooked on it.
NooneAtAll3 6mo ago
Everyone's betting on Jevons paradox the hope is that Ai is "the next semiconductor" and "the next internet"
nirui 6mo ago
> This demand for RAM is built on a foundation of sand Not exactly. LLMs are already quite useful today if you use them as a tool, so they are there to stay. The remaining problem is scalability, a.k.a. how to make LLMs cheap to use. But scalability is not really a requirement when you look the bigger picture. If smaller software company/projects can't afford to use AI, the bigger ones might just. Eventually they will discover variable use cases for such tech, even if it only serves big firms i.e. defense, resource extraction, war, finance etc. To the other end, if scalability is achieved, the use of LLM products will be cheaper too, so smaller project can also use them. But of course, if LLM usage is too cheap, then many were-to-be-consumers will just create software projects by themselves at their homes.
- rimliu 6mo ago
  Would they be considered as useful if users are required to pay non-subsidized price though.
  - nirui 6mo ago
    They are. The advanced workflows such as programming are not the only use cases for LLMs. There are much simpler work such as answering simple questions or arrange simple tasks. Both will improve productivity. And more advanced model should improve productivity even more.
tonyedgecombe 6mo ago
> This demand for RAM is built on a foundation of sand RAM is built on a foundation of sand.
- fransje26 6mo ago
  I laughed.
energy123 6mo ago
The usage of LLMs is continuing to increase ~exponentially. I'm going to bet on that rather than some half-baked scenario analysis that only takes into account one scenario and assigns a 100% probability to it.
- dannersy 6mo ago
  > The usage of LLMs is continuing to increase ~exponentially I would like a source for that statement. Additionally, I want to know by who? Because it certainly isn't end users. Inflating token usage doesn't make it any more economically viable if your user base, b2b or not, hasn't increased with it. On the contrary, that is a worse scenario for providers.
  - energy123 6mo ago
    > I would like a source for that statement The recent enterprise revenue numbers of Anthropic
    - dannersy 6mo ago
      So as I said, a self interested metric who also controls how many tokens it takes to get a desirable result from their models.
      - energy123 6mo ago
        Users are willingly paying for larger volumes of tokens. You are layering your own unproven interpretation onto that. I would have arrived at an opposite interpretation given the available facts. Models are becoming more token efficient for the same task, such as ChatGPT 5.3 versus 5.2 which halved the token count, and capabilities show a log relationship with the number of tokens since o1 preview was revealed in September 2024.
        dannersy 6mo ago
        No, you have gone off in your own tangent. The person you're responding to is talking about money and my point is that you're using a misleading metric. Even if the current user base is paying more for the "exponential token usage", it does not add up to the industry's cost of maintaining and building on this technology, especially since we are not taking into account what that token usage costs the provider. First you said Anthropic as your source, but now you're talking about OpenAI's ChatGPT, who are floundering for a product and user base, which they themselves claim will be profitable through subscriptions at numbers never seen before in a subscription business model.

locknitpicker 6mo ago

> If AI makes software easier to create, that will drive the price down. Supposedly AI drives down the cost of producing software,not the "price". > How are software companies going to make enough revenue to pay for AI, when the amount of money being spent on AI is already multiples of the current total global expenditure on software? Currently, the cost of AI is between $20/month and around $200/month per developer. I think the huge billions you're seeing in the news are the investment cost on AI companies, who are burning through cash to invest in compute infrastructure to allow both training and serving users. > This demand for RAM is built on a foundation of sand, there will be a glut of capacity when it all shakes out. Who knows? What I know is that I need >64GB of RAM to run local models, and that means most people will need to upgrade from their 8Gb/16GB setup to do the same. Graphics cards follow mostly the same pattern.

zozbot234 6mo ago
> Who knows? What I know is that I need >64GB of RAM to run local models, and that means most people will need to upgrade from their 8Gb/16GB setup to do the same. Graphics cards follow mostly the same pattern. Depends how big the models are, how fast you want them to run and how much context you need for your usage. If you're okay with running only smaller models (which are still very capable in general, their main limitation is world knowledge) making very simple inferences at low overall throughput, you can just repurpose the RAM, CPUs/iGPUs and storage in the average setup.
adrian_b 6mo ago
You need >64 GB of DRAM to run local models fast. You can run huge local models slowly with the weights stored on SSDs. Nowadays there are many computers that can have e.g. 2 PCIe 5.0 SSDs, which allow a reading throughput of 20 to 30 gigabyte per second, depending on the SSDs (or 1 PCIe 5.0 + 1 PCIe 4.0, for a throughput in the range 15-20 GB/s). There are still a lot of improvements that can be done to inference back-ends like llama.cpp to reach the inference speed limit determined by the SSD throughput. It seems that it is possible to reach inference speed in the range from a few seconds per token to a few tokens per second. That may be too slow for a chat, but it should be good enough for an AI coding assistant, especially if many tasks are batched, so that they can progress simultaneously during a single read pass over the SSD data.
- zozbot234 6mo ago
  You can do that, but you're going to have rather low throughput unless you have lots of PCIe lanes to attach storage to. That's going to require either a HEDT or some kind of compute cluster. Batching inferences doesn't necessarily help that much since as models get sparser the individual inferences are going to share fewer experts. It does always help wrt. shared routing layers, of course.
tomaskafka 6mo ago
I got a 128 GB MBP, and the current models are fit enough to manage the calendar or do research on web (very slowly), not to be useful companions for coding as I hoped.

repeekad 6mo ago

Analyst says it would require a new $35/month subscription for every iPhone user, or a new $180/month for every Netflix subscriber. Claude Max subscriptions have gone up, but do you think every Netflix user will pay for one?.. https://www.tomshardware.com/tech-industry/artificial-intelligence/usd650-billion-in-annual-revenue-required-to-deliver-10-percent-return-on-ai-buildout-investment-j-p-morgan-claims-equivalent-to-usd35-payment-from-every-iphone-user-or-usd180-from-every-netflix-subscriber-in-perpetuity https://www.tomshardware.com/tech-industry/artificial-intell...

avidiax 6mo ago
This is probably the wrong denominator. There are more than a billion white-collar workers. Making them all just 10% more effective would possibly be worth more than $650B/year (~$650/worker).

lofaszvanitt 6mo ago

It's a trojan horse. They expect the world will get hooked on it.

NooneAtAll3 6mo ago

Everyone's betting on Jevons paradox the hope is that Ai is "the next semiconductor" and "the next internet"

nirui 6mo ago

> This demand for RAM is built on a foundation of sand Not exactly. LLMs are already quite useful today if you use them as a tool, so they are there to stay. The remaining problem is scalability, a.k.a. how to make LLMs cheap to use. But scalability is not really a requirement when you look the bigger picture. If smaller software company/projects can't afford to use AI, the bigger ones might just. Eventually they will discover variable use cases for such tech, even if it only serves big firms i.e. defense, resource extraction, war, finance etc. To the other end, if scalability is achieved, the use of LLM products will be cheaper too, so smaller project can also use them. But of course, if LLM usage is too cheap, then many were-to-be-consumers will just create software projects by themselves at their homes.

rimliu 6mo ago
Would they be considered as useful if users are required to pay non-subsidized price though.
- nirui 6mo ago
  They are. The advanced workflows such as programming are not the only use cases for LLMs. There are much simpler work such as answering simple questions or arrange simple tasks. Both will improve productivity. And more advanced model should improve productivity even more.

tonyedgecombe 6mo ago

> This demand for RAM is built on a foundation of sand RAM is built on a foundation of sand.

fransje26 6mo ago
I laughed.

energy123 6mo ago

The usage of LLMs is continuing to increase ~exponentially. I'm going to bet on that rather than some half-baked scenario analysis that only takes into account one scenario and assigns a 100% probability to it.

dannersy 6mo ago
> The usage of LLMs is continuing to increase ~exponentially I would like a source for that statement. Additionally, I want to know by who? Because it certainly isn't end users. Inflating token usage doesn't make it any more economically viable if your user base, b2b or not, hasn't increased with it. On the contrary, that is a worse scenario for providers.
- energy123 6mo ago
  > I would like a source for that statement The recent enterprise revenue numbers of Anthropic
  - dannersy 6mo ago
    So as I said, a self interested metric who also controls how many tokens it takes to get a desirable result from their models.
    - energy123 6mo ago
      Users are willingly paying for larger volumes of tokens. You are layering your own unproven interpretation onto that. I would have arrived at an opposite interpretation given the available facts. Models are becoming more token efficient for the same task, such as ChatGPT 5.3 versus 5.2 which halved the token count, and capabilities show a log relationship with the number of tokens since o1 preview was revealed in September 2024.
      - dannersy 6mo ago
        No, you have gone off in your own tangent. The person you're responding to is talking about money and my point is that you're using a misleading metric. Even if the current user base is paying more for the "exponential token usage", it does not add up to the industry's cost of maintaining and building on this technology, especially since we are not taking into account what that token usage costs the provider. First you said Anthropic as your source, but now you're talking about OpenAI's ChatGPT, who are floundering for a product and user base, which they themselves claim will be profitable through subscriptions at numbers never seen before in a subscription business model.