3 ms·

I think an interesting thing about recent AI developments is that its all happening right as we hit the diminishing returns side of another "exponential that's

I think an interesting thing about recent AI developments is that its all happening right as we hit the diminishing returns side of another "exponential that's actually a sigmoid" which is Moore's law. The naive expectation is that AI will slow down b/c Moore's law is coming to an end, but if you really think about the models and how they are currently implemented in silicon, they are still inefficient as hell. At some point someone will build a tensor processing chip that replaces all the digital matmuls with analogue logamp matmuls, or some breakthrough in memristors will start breaking down the barrier between memory and compute. With the right level of research funding in hardware, the ceiling for AI can be very high.

cyanydeez 5mo ago
they already did put a model into the silicon and it's crazy fast. https://chatjimmy.ai/ https://chatjimmy.ai/ I'm pretty sure there's a 3 year design goal starting this year that'll do that to any of the qwen, deepseek, etc models. There's a lot you could do with sped up models of these quality. It might even be bad enough that the real bubble is how much we don't need giant data centers when 80-90% of use cases could just be a silicon chip with a model rather than as you say, bloated SOTA
- clickety_clack 5mo ago
  It would be pretty cool to have interchangeable usb keys with models on them.
- LarsDu88 5mo ago
  And this is an asic that is still operating digitally. Imagine a chip with baked it weights that does its math analogue with 20x reduction in number of circuit elements needed to do a multiplication op. If there's a breakthrough in memristors, you could end up with another 20x reduction in circuit elements (get rid of memory bottlnecks, start doing multiplication ops as log transform voltage addition) The ceiling is ultra high for how far AI can go.
  - mordae 5mo ago
    Except weights will be unstable - temperature and frequency dependent - and we still have issues delivering analog circuits reliably to the spec. So it would take multiple attempts. But yeah, as soon as the digital models start to plateau, ASICs and then this will happen.
    - ACCount37 5mo ago
      You could SFT for temperature and frequency stability. I'm not even kidding. Modern ML systems already eat errors - what's one more error type for them to eat?
  - imtringued 5mo ago
    fp8 and fp4 are already the standard data types, where are you going to find a 20x reduction? At 1 sign bit, 1 exponent and 2 mantissa bits, there is barely any work done during multiplication into a bf16 accumulator. You're performing two shifted adds at this point.
throwaway27448 5mo ago
Even at orders of magnitude greater speed, we've still hit diminishing returns for quality of output. We simply haven't found anything like superhuman reasoning ability, just superhuman (potentially) reasoning speed.
- horsawlarway 5mo ago
  Possibly - but we've also seen that spending more tokens on a task can improve the quality of the output (reasoning, CoT, etc). So it's not impossible to have things that seem orthogonal, like generation speed or context length, have an impact on quality of result.
- energy123 5mo ago
  It's not that easy to assess diminishing returns with saturated benchmarks where asymptoting to 100% is mathematically baked in. I could point to the number of Erdos proofs being solved by AI going from 0 to many very recently as evidence for acceleration.
  - throwaway27448 5mo ago
    That is not evidence of acceleration, just of some measurable improvement compared to a previous model. After all, humans have made these breakthroughs since before recorded history—that never by itself implied accelerating intelligence.
    - energy123 5mo ago
      What would be evidence of acceleration? What would be evidence of diminishing returns? Both questions are hard to answer because it's difficult to avoid constructing a metric where the conclusion is already baked in.
- LarsDu88 5mo ago
  I disagree with this. Reinforcement learning with verifiable rewards training is actually the secret sauce that is leading Claude and GPT to automating software engineering tasks. All the easily verifiable domains such as mathematics, coding, and things that can be run inside a reasonable simulation are falling very very fast. By next year if not sooner, mathematicians will be wildly outpaced by LLMs for reasoning.
paulpauper 5mo ago
Moore's law is bypassed with volume--more datacenters
- torginus 5mo ago
  The argument I've heard about how special Moore's law is that if you take a baby crawling at 10cm/s and 1000x it, you get almost Mach 3, which is almost as fast as the SR71 Blackbird flies. yet with a 18 month cadence, that's 15 years of progress, which has happened multiple times during the existence of the VLSI industry.
rdedev 5mo ago
IMO we are either limited by data or reaching the limits of what's possible with a transformer architecture. Hardware will get us efficiency but I am not sure if it will lead to smarter models
- dehrmann 5mo ago
  My doubts in the architecture is how different they are from human intelligence. They need an inordinate amount of training data and lack any sort of generational architectural intelligence.
ToValueFunfetti 5mo ago
I suspect if you consider neurons as components of computation, you could draw an exponential of total computation in the world that goes back to the dawn of humanity, maybe further. Most of that would just be population, but interesting that digital computers start picking up the slack just as population growth slows.
imtringued 5mo ago
>but if you really think about the models and how they are currently implemented in silicon, they are still inefficient as hell. I'm not sure this follows? Research has gone into two bit quantizations that only need a scale factor per block and each parameter merely takes up two bits which means that the operations can be mapped directly onto adders rather than multipliers. >but if you really think about the models and how they are currently implemented in silicon, they are still inefficient as hell. The vast majority of analogue components are both space and energy inefficient. Digital won for a reason. You can simply keep scaling to lower voltages and smaller transistors since you only need to distingish between one and zero.

cyanydeez 5mo ago

they already did put a model into the silicon and it's crazy fast. https://chatjimmy.ai/ https://chatjimmy.ai/ I'm pretty sure there's a 3 year design goal starting this year that'll do that to any of the qwen, deepseek, etc models. There's a lot you could do with sped up models of these quality. It might even be bad enough that the real bubble is how much we don't need giant data centers when 80-90% of use cases could just be a silicon chip with a model rather than as you say, bloated SOTA

clickety_clack 5mo ago
It would be pretty cool to have interchangeable usb keys with models on them.
LarsDu88 5mo ago
And this is an asic that is still operating digitally. Imagine a chip with baked it weights that does its math analogue with 20x reduction in number of circuit elements needed to do a multiplication op. If there's a breakthrough in memristors, you could end up with another 20x reduction in circuit elements (get rid of memory bottlnecks, start doing multiplication ops as log transform voltage addition) The ceiling is ultra high for how far AI can go.
- mordae 5mo ago
  Except weights will be unstable - temperature and frequency dependent - and we still have issues delivering analog circuits reliably to the spec. So it would take multiple attempts. But yeah, as soon as the digital models start to plateau, ASICs and then this will happen.
  - ACCount37 5mo ago
    You could SFT for temperature and frequency stability. I'm not even kidding. Modern ML systems already eat errors - what's one more error type for them to eat?
- imtringued 5mo ago
  fp8 and fp4 are already the standard data types, where are you going to find a 20x reduction? At 1 sign bit, 1 exponent and 2 mantissa bits, there is barely any work done during multiplication into a bf16 accumulator. You're performing two shifted adds at this point.

throwaway27448 5mo ago

Even at orders of magnitude greater speed, we've still hit diminishing returns for quality of output. We simply haven't found anything like superhuman reasoning ability, just superhuman (potentially) reasoning speed.

horsawlarway 5mo ago
Possibly - but we've also seen that spending more tokens on a task can improve the quality of the output (reasoning, CoT, etc). So it's not impossible to have things that seem orthogonal, like generation speed or context length, have an impact on quality of result.
energy123 5mo ago
It's not that easy to assess diminishing returns with saturated benchmarks where asymptoting to 100% is mathematically baked in. I could point to the number of Erdos proofs being solved by AI going from 0 to many very recently as evidence for acceleration.
- throwaway27448 5mo ago
  That is not evidence of acceleration, just of some measurable improvement compared to a previous model. After all, humans have made these breakthroughs since before recorded history—that never by itself implied accelerating intelligence.
  - energy123 5mo ago
    What would be evidence of acceleration? What would be evidence of diminishing returns? Both questions are hard to answer because it's difficult to avoid constructing a metric where the conclusion is already baked in.
LarsDu88 5mo ago
I disagree with this. Reinforcement learning with verifiable rewards training is actually the secret sauce that is leading Claude and GPT to automating software engineering tasks. All the easily verifiable domains such as mathematics, coding, and things that can be run inside a reasonable simulation are falling very very fast. By next year if not sooner, mathematicians will be wildly outpaced by LLMs for reasoning.

paulpauper 5mo ago

Moore's law is bypassed with volume--more datacenters

torginus 5mo ago
The argument I've heard about how special Moore's law is that if you take a baby crawling at 10cm/s and 1000x it, you get almost Mach 3, which is almost as fast as the SR71 Blackbird flies. yet with a 18 month cadence, that's 15 years of progress, which has happened multiple times during the existence of the VLSI industry.

rdedev 5mo ago

IMO we are either limited by data or reaching the limits of what's possible with a transformer architecture. Hardware will get us efficiency but I am not sure if it will lead to smarter models

dehrmann 5mo ago
My doubts in the architecture is how different they are from human intelligence. They need an inordinate amount of training data and lack any sort of generational architectural intelligence.

ToValueFunfetti 5mo ago

I suspect if you consider neurons as components of computation, you could draw an exponential of total computation in the world that goes back to the dawn of humanity, maybe further. Most of that would just be population, but interesting that digital computers start picking up the slack just as population growth slows.

imtringued 5mo ago

>but if you really think about the models and how they are currently implemented in silicon, they are still inefficient as hell. I'm not sure this follows? Research has gone into two bit quantizations that only need a scale factor per block and each parameter merely takes up two bits which means that the operations can be mapped directly onto adders rather than multipliers. >but if you really think about the models and how they are currently implemented in silicon, they are still inefficient as hell. The vast majority of analogue components are both space and energy inefficient. Digital won for a reason. You can simply keep scaling to lower voltages and smaller transistors since you only need to distingish between one and zero.