3 ms·

> Mythic M1 stores up to 80 million neural network weight parameters directly on-chip Which means connecting ~350 chiplets to run a Qwen 3.8 27b and over 30000

by mdp2021 2mo ago

> Mythic M1 stores up to 80 million neural network weight parameters directly on-chip Which means connecting ~350 chiplets to run a Qwen 3.8 27b and over 30000 chiplets to run Qwen3.8-2.4T-A95B. Cost? Space? Feasibility? Edit: wrong values, lost a zero... Edit: seemingly, the M1 is only part of the whole need. With the M1, you would run a feedforward pass of the NN but use the rest of the Von Neumann architecture to manage the data. The pass in the M1 will be lightning fast, the rest still a bottleneck. The M1 is almost explicitly not for LLMs.

sroussey 2mo ago
I think we are at the apex of Von Neumann machines. Once there is enough money to make alternatives, they will thrive. And AI is that catalyst.
imtringued 2mo ago
[dead]
dgfl 2mo ago
The vanguard chiplets even steps down to 30M… but they do claim to have a “Mead” design [1] designed to run GPT-3 in a single chip. 3D NAND flash can indeed routinely store hundreds of GB per die, so that’s proven. The question is about all the peripheral circuitry needed. Each attention block would need its own KV cache (i.e. SRAM or DRAM somewhere), plus DAC/ADC inputs and outputs, unless they figure out a way to keep it analog all the way (really cool but unlikely). I think this field is very interesting, at least from a technology point of view. Whether it works out or not will sadly be a matter of economics more than physics I fear. [1] https://www.mythic.ai/mead https://www.mythic.ai/mead
- mdp2021 2mo ago
  > unless they figure out a way to keep it analog all the way (really cool but unlikely) Very unlikely for the connection to the cache RAM, seemingly impossible if we want to get an articulate output :) > they do claim to have a “Mead” design [1] designed to run GPT-3 [ - i.e. to hold a 175b NN - ] in a single chip Careful: that is the /intention/, but the chip is just the storage (and CiM) for the NN and other parts are missing - explained in the Vanguard, not explained in the Mead. > Whether it works out or not will sadly be a matter of economics more than physics I fear Yes, but: -- what is not enabled today may often be tomorrow through advances, esp. in the economy of production; -- a very great point about this product seems to be that the chips can be rewritten, the NN is not etched and static (cpr. Taalas): that makes the practical implications extremely relevant, the demand would be "screaming mob" like; -- we have to go in that direction (of NNs in CiM) anyway, so it's just a matter of time, effort after effort we will get there.
- K0balt 2mo ago
  This might end up being a strong use case for FeFET FRAM, nonvolatile, high density, and sram fast. In an alternative universe where fram infra was as widespread as flash, memory is fast, cheap, and nonvolatile. It’s just that flash built the infrastructure and climbing that ladder is nearly impossible.
  - dgfl 2mo ago
    I wouldn't really say SRAM-fast. Maybe DRAM-fast? At least in write speed. But the thing is, you don't need high write speed for this application. And read speed should be the same between flash and FeFETs. Btw, I've seen some roadmaps in which foundries were heading towards integrating ferroelectrics and floating gates to accelerate switching. The two technologies are not mutually exclusive really, so I think we might see something in the relatively near future. The crazy memory market may help!

by mdp2021 2mo ago

sroussey 2mo ago

I think we are at the apex of Von Neumann machines. Once there is enough money to make alternatives, they will thrive. And AI is that catalyst.

imtringued 2mo ago

[dead]

dgfl 2mo ago

The vanguard chiplets even steps down to 30M… but they do claim to have a “Mead” design [1] designed to run GPT-3 in a single chip. 3D NAND flash can indeed routinely store hundreds of GB per die, so that’s proven. The question is about all the peripheral circuitry needed. Each attention block would need its own KV cache (i.e. SRAM or DRAM somewhere), plus DAC/ADC inputs and outputs, unless they figure out a way to keep it analog all the way (really cool but unlikely). I think this field is very interesting, at least from a technology point of view. Whether it works out or not will sadly be a matter of economics more than physics I fear. [1] https://www.mythic.ai/mead https://www.mythic.ai/mead

mdp2021 2mo ago
> unless they figure out a way to keep it analog all the way (really cool but unlikely) Very unlikely for the connection to the cache RAM, seemingly impossible if we want to get an articulate output :) > they do claim to have a “Mead” design [1] designed to run GPT-3 [ - i.e. to hold a 175b NN - ] in a single chip Careful: that is the /intention/, but the chip is just the storage (and CiM) for the NN and other parts are missing - explained in the Vanguard, not explained in the Mead. > Whether it works out or not will sadly be a matter of economics more than physics I fear Yes, but: -- what is not enabled today may often be tomorrow through advances, esp. in the economy of production; -- a very great point about this product seems to be that the chips can be rewritten, the NN is not etched and static (cpr. Taalas): that makes the practical implications extremely relevant, the demand would be "screaming mob" like; -- we have to go in that direction (of NNs in CiM) anyway, so it's just a matter of time, effort after effort we will get there.
K0balt 2mo ago
This might end up being a strong use case for FeFET FRAM, nonvolatile, high density, and sram fast. In an alternative universe where fram infra was as widespread as flash, memory is fast, cheap, and nonvolatile. It’s just that flash built the infrastructure and climbing that ladder is nearly impossible.
- dgfl 2mo ago
  I wouldn't really say SRAM-fast. Maybe DRAM-fast? At least in write speed. But the thing is, you don't need high write speed for this application. And read speed should be the same between flash and FeFETs. Btw, I've seen some roadmaps in which foundries were heading towards integrating ferroelectrics and floating gates to accelerate switching. The two technologies are not mutually exclusive really, so I think we might see something in the relatively near future. The crazy memory market may help!