4 ms·

> To me it feels like the AI inevitability will be much quicker. Based on what? We're only seeing linear improvements for increasing spending. There's no new

> To me it feels like the AI inevitability will be much quicker. Based on what? We're only seeing linear improvements for increasing spending. There's no new algorithm ideas on the horizon, just more and more hardware, in the hopes that if we throw enough RAM and CPU at the problem, it will suddenly become "AGI." No one has their eye on power budgets or sustainability or durability of the system. The human brain has such a high degree of energy efficiency that I don't think people understand the realities of competing with it digitally. The main problem "AI" seems to solve is that humans get bored with certain tasks. The language models obviously don't, but they do hallucinate, and checking for hallucinations is an exceedingly boring task. It's a coffin corner of bad ideas.

rdsubhas 1y ago
> Based on what? Internet did not have enough devices to reach people. At the height of 2002 only a fraction of people worldwide had an already expensive computer and an internet to go with it. I ran a e-commerce startup from 2005-2010. Having access to demand is a thing. Today everyone has access in their pockets. Go to small city in Africa, India and China, and observe how they use AI. See how perplexity has put AI answers in hundreds of millions of people's hands before Google in a matter of months. Forgive me for saying — but "Based on what" for comparing accelerated adoption between 2005 and 2025 — is discarding many huge elephants in the room, starting with that small thing you're reading this in your hand with, and the invisible thing that's sending you this comment.
tim333 1y ago
>no new algorithm ideas on the horizon The LLM algorithms seem pretty clunky to me, like a hack designed for text translation that surprised everyone by getting quite smart. The reason the human brain is so much more energy efficient is quite likely better design/algorithms. I was watching some video comparing brain and LLM function and almost tempted to try building something myself (https://youtu.be/3SUqBUGlyh8 https://youtu.be/3SUqBUGlyh8). I'm sure there are many more competent people looking at similar things.
- fatata123 1y ago
  [dead]
- ACCount37 1y ago
  Everyone says "those autoregressive transformer LLMs are obviously flawed", and then fails to come up with anything that outperforms them. I'm not too bullish on architectural gains. There are efficiencies to be had, but far closer to "+5% a year" than "+5000% in a single breakthrough". You can try to build a novel AI architecture, at a small scale. Just be ready. This field will kick your teeth in. ML doesn't like grand ideas and isn't kind to high aspirations.
  - fsmv 1y ago
    Physics is obviously incomplete and yet nobody can solve quantum gravity. Being obviously flawed doesn't mean the solution is obvious. That's the whole problem.
    - ACCount37 1y ago
      I think in this case, people tend to underrate just how capable and flexible the basic LLM architecture is. And, also, underrate how many gains are there in better training vs better architecture.
    - tim333 1y ago
      Not obvious but the brain manages to think in ways LLMs don't really and the design is presumably of fairly finite complexity to be encoded in DNA.
  - gizmo686 1y ago
    Most people are not ML researchers. Most of the AI industry is not AI researchers. Most of the AI spending is not going to AI researchers. AI researchers came up with an architectural improvement that made a lot of previously impossible stuff barely possible. Then, industry ran with it. Scaling that particular trick to the limits by throwing as much raw compute and data at it as humanly possible. You don't need to be an AI expert to know that there are probably more advances to be had and that funding foundational research is the way to get them.
pardon_me 1y ago
The current LLM models are too fallible and inefficient for everyday use. Energy requirements have become a major concern, bucking the trend of banking exponential computing gains through the efficiency improvements. Until recently, year-on-year global energy demands of data-centers were increasing linearly despite the exponential increase in computing power. This has changed since the cost of equipment and infrastructure has been close to "free" for the large corporations running a cycle of funny-money investment in a bubble. This has allowed them sell access to the computing and models for just above operational energy costs (to the extent of increasing global energy prices), whilst offering free accounts to harvest data. No small competitor could possibly compete with that model. The calculation for profitability (useful output for humans in a cost-benefit analysis) of the current setup is broken, and trades on our dreams of the future. Scaling computing forever simply does not work. It will never be profitable without further leaps forward in the technology--either more efficient models or hardware. By this time, the extent to the excessive investment in new data-centers will be clear.

rdsubhas 1y ago

> Based on what? Internet did not have enough devices to reach people. At the height of 2002 only a fraction of people worldwide had an already expensive computer and an internet to go with it. I ran a e-commerce startup from 2005-2010. Having access to demand is a thing. Today everyone has access in their pockets. Go to small city in Africa, India and China, and observe how they use AI. See how perplexity has put AI answers in hundreds of millions of people's hands before Google in a matter of months. Forgive me for saying — but "Based on what" for comparing accelerated adoption between 2005 and 2025 — is discarding many huge elephants in the room, starting with that small thing you're reading this in your hand with, and the invisible thing that's sending you this comment.

tim333 1y ago

>no new algorithm ideas on the horizon The LLM algorithms seem pretty clunky to me, like a hack designed for text translation that surprised everyone by getting quite smart. The reason the human brain is so much more energy efficient is quite likely better design/algorithms. I was watching some video comparing brain and LLM function and almost tempted to try building something myself (https://youtu.be/3SUqBUGlyh8 https://youtu.be/3SUqBUGlyh8). I'm sure there are many more competent people looking at similar things.

fatata123 1y ago
[dead]
ACCount37 1y ago
Everyone says "those autoregressive transformer LLMs are obviously flawed", and then fails to come up with anything that outperforms them. I'm not too bullish on architectural gains. There are efficiencies to be had, but far closer to "+5% a year" than "+5000% in a single breakthrough". You can try to build a novel AI architecture, at a small scale. Just be ready. This field will kick your teeth in. ML doesn't like grand ideas and isn't kind to high aspirations.
- fsmv 1y ago
  Physics is obviously incomplete and yet nobody can solve quantum gravity. Being obviously flawed doesn't mean the solution is obvious. That's the whole problem.
  - ACCount37 1y ago
    I think in this case, people tend to underrate just how capable and flexible the basic LLM architecture is. And, also, underrate how many gains are there in better training vs better architecture.
  - tim333 1y ago
    Not obvious but the brain manages to think in ways LLMs don't really and the design is presumably of fairly finite complexity to be encoded in DNA.
- gizmo686 1y ago
  Most people are not ML researchers. Most of the AI industry is not AI researchers. Most of the AI spending is not going to AI researchers. AI researchers came up with an architectural improvement that made a lot of previously impossible stuff barely possible. Then, industry ran with it. Scaling that particular trick to the limits by throwing as much raw compute and data at it as humanly possible. You don't need to be an AI expert to know that there are probably more advances to be had and that funding foundational research is the way to get them.

pardon_me 1y ago

The current LLM models are too fallible and inefficient for everyday use. Energy requirements have become a major concern, bucking the trend of banking exponential computing gains through the efficiency improvements. Until recently, year-on-year global energy demands of data-centers were increasing linearly despite the exponential increase in computing power. This has changed since the cost of equipment and infrastructure has been close to "free" for the large corporations running a cycle of funny-money investment in a bubble. This has allowed them sell access to the computing and models for just above operational energy costs (to the extent of increasing global energy prices), whilst offering free accounts to harvest data. No small competitor could possibly compete with that model. The calculation for profitability (useful output for humans in a cost-benefit analysis) of the current setup is broken, and trades on our dreams of the future. Scaling computing forever simply does not work. It will never be profitable without further leaps forward in the technology--either more efficient models or hardware. By this time, the extent to the excessive investment in new data-centers will be clear.