4 ms·

It's quite telling that the 8B Taalas chip was already reticle-sized on TSMC N6. I mean, we're talking about a process that does ~100 MTr/mm², ROM needs about o

It's quite telling that the 8B Taalas chip was already reticle-sized on TSMC N6. I mean, we're talking about a process that does ~100 MTr/mm², ROM needs about one transistor per bit, but can probably be packed more densely than general logic. Something like, say, 150 megabit/mm² is not a lot. N6 has a 850 mm² reticle limit. This roughly tracks, the article says the chip has 8B parameters and apparently spends about half the area on ROM. There's a reason AI accelerators just use a ton of silicon area (each HBM3 die is >1000mm² of silicon). I imagine this is not terribly viable unless they make it a lot more space efficient e.g. using MLC ROM if they don't already, or use stacked dies with a ROM-optimized process. And then we're back to not cheap, though reticle chips were never in the cheap area to begin with.

Tuna-Fish 2mo ago
Taalas exploits the low cardinality to store one 4-bit weight with one transistor. (They are using metal layer traces for the ROM, and connecting an access transistor to light up one of 16 options.) Their system is honestly very efficient for the weights, the problem is the KV-cache. That's why HC1 only supports such short context, they use SRAM for that and spend most of what's left of the die for it. The recent advancements that made attention more efficient are probably going to be very useful for them.
- jrflo 2mo ago
  Do you really need a KV cache if inference is that fast though?
  - Tuna-Fish 2mo ago
    ... Yes. Quadratic is really bad for large enough n, and you need that big context for useful work.
- adityazero 2mo ago
  [flagged]
phonon 2mo ago
A full wafer like Cerebras is about 60x that, and N2P has about 3x the transistor density. So right now it's technically feasible to etch a 1.4 trillion parameter model. So roughly DeepSeek-V4-Pro class. Imagine that running a factory, for example.
- briansm 2mo ago
  Cerebras have special techniques to work around etching errors / bad cores on their wafers. This is possible since their wafers are effectively hundreds of identical copies of redundant cores. Can't do that for a globally unique model. Etching failure in that situation would be like brain-damage in a human, all sorts of weird effects would start appearing.
  - fulafel 2mo ago
    There's several ways to engineer around that as the errors are detectable. There's a big literature on how to trade off speed or transistors for error correction. [1] (Is Cerebras doing something novel? CPUs and memory blocks have been doing those things for a long time too, since the error rate is otherwise too high for normal size chips as well) [1] see eg https://www.vlsimentor.com/dft/redundancy-bisr https://www.vlsimentor.com/dft/redundancy-bisr to get some basic concepts
  - phonon 2mo ago
    A few hundred bad bits/transistors in a trillion+ parameter model would compromise its abilities not one iota...the models are inherently lossy and resistant to "brain damage"...
wtallis 2mo ago
> (each HBM3 die is >1000mm² of silicon) Did you mean that each HBM3 stack is that large? Because it only takes one glance to see that the memory chips are much smaller than reticle-sized GPUs they sit next to.

Tuna-Fish 2mo ago

Taalas exploits the low cardinality to store one 4-bit weight with one transistor. (They are using metal layer traces for the ROM, and connecting an access transistor to light up one of 16 options.) Their system is honestly very efficient for the weights, the problem is the KV-cache. That's why HC1 only supports such short context, they use SRAM for that and spend most of what's left of the die for it. The recent advancements that made attention more efficient are probably going to be very useful for them.

jrflo 2mo ago
Do you really need a KV cache if inference is that fast though?
- Tuna-Fish 2mo ago
  ... Yes. Quadratic is really bad for large enough n, and you need that big context for useful work.
adityazero 2mo ago
[flagged]

phonon 2mo ago

A full wafer like Cerebras is about 60x that, and N2P has about 3x the transistor density. So right now it's technically feasible to etch a 1.4 trillion parameter model. So roughly DeepSeek-V4-Pro class. Imagine that running a factory, for example.

briansm 2mo ago
Cerebras have special techniques to work around etching errors / bad cores on their wafers. This is possible since their wafers are effectively hundreds of identical copies of redundant cores. Can't do that for a globally unique model. Etching failure in that situation would be like brain-damage in a human, all sorts of weird effects would start appearing.
- fulafel 2mo ago
  There's several ways to engineer around that as the errors are detectable. There's a big literature on how to trade off speed or transistors for error correction. [1] (Is Cerebras doing something novel? CPUs and memory blocks have been doing those things for a long time too, since the error rate is otherwise too high for normal size chips as well) [1] see eg https://www.vlsimentor.com/dft/redundancy-bisr https://www.vlsimentor.com/dft/redundancy-bisr to get some basic concepts
- phonon 2mo ago
  A few hundred bad bits/transistors in a trillion+ parameter model would compromise its abilities not one iota...the models are inherently lossy and resistant to "brain damage"...

wtallis 2mo ago

> (each HBM3 die is >1000mm² of silicon) Did you mean that each HBM3 stack is that large? Because it only takes one glance to see that the memory chips are much smaller than reticle-sized GPUs they sit next to.