7 ms·
Baking the base models on to ROM makes a lot of economic sense. SRAM for the KV cache & fine-tunes, not so much. Sure you’d get incredible speeds but it’s not s
by chorizo 2mo ago
Baking the base models on to ROM makes a lot of economic sense. SRAM for the KV cache & fine-tunes, not so much. Sure you’d get incredible speeds but it’s not scalable from a die-size or cost perspective.
Rather base model on ROM + KV cache on DRAM is much more scalable. Also this would work great for edge devices that have a 2-5 year lifecycle.
- adrianN 2mo agoIt is my understanding that just baking the model itself into silicon only gives moderate gains because memory bandwidth remains a bottleneck.
- chorizo 2mo agoThe big benefit is ROM cells require fewer components than DRAM. So the chips would be tiny, dense, cheap and consume far less power.
- klodolph 2mo agoI thought DRAM was pretty dense already. Is mask ROM that much denser?
- chorizo 2mo agoYes, each rom bit can be a transistor or even a diode with a decoder circuit. Simplest Dram cell is capacitor+transistor - and you need a clock, refresh circuit etc. Someday, I imagine model weights could even be encoded as analog resistors (memristors or similar) for even greater density
- ReactiveJelly 2mo agoHm. I wonder how many relays I'd need to make a physical MNIST classifier. That'd be dope
- Tuna-Fish 2mo agoIt's not even just that. If you just built the rom chips separately and swapped them for the RAM of a normal accelerator, it would not help at all. The trick is that every compute element in their system has it's own small pool of ROM, instead of putting all the ram behind a common pipe. ROM is just used because it's the densest kind of memory that can be fabricated on the same process as their logic.
- lelanthran 2mo agoDid you try using the the talaas chat? Something stupid like 18k tokens/second. Think it's called Askjimmy or similar.
- jgilias 2mo agoOh boy, thanks for sharing this, truly mind blowing. It was chatjimmy.ai
- TeMPOraL 2mo agoWhat is the model they're using there though? Interrogated, it claims it's a BERT variant and has capabilities around GPT-3 and below GPT-4. (Not that I believe it, it writes too well for GPT-3.) Hosted frontier models from two years ago would be much faster today, too.
- egorfine 2mo agoThey run Llama 3.1 8B.
- onion2k 2mo agoBaking the base models on to ROM makes a lot of economic sense. Less so for consumers though, because it'd mean the phone is out of date in 3 months when a better model comes along.
- ronsor 2mo agoIt's a perfect reason to get consumers to buy a new phone every year again! They got bored of the camera.
- miohtama 2mo agoThis stopped working some years ago. Now you sell the same phone with higher price tag.
- ben_w 2mo agoRight now the models are doubling in performance (by the METR time horizon metric at least) every 4 months, so 3 doublings in a year; conversely, I hear (not my field) it takes around a year to make a prototype IC and another year to turn that into mass production, i.e. if the next (late-2026 model) iPhone has a chip like this, it will likely be with, at best, a late-2024 set of weights. I think you can get open-weights models today that have performance equivalent to the SOTA-late-2024 while fitting in the RAM of a (high end) 2025-26 phone. At some point the music will stop on training bigger models, and when that happens it will make sense to have ROM weights (or 100% analog circuits given how noise-resistant LLMs are), but we'll know when that is because the investment bubble funding the training of new models will have burst.
- tesnorindian 2mo agoWe also have ReRAM (Analog Computing), which also holds a promising future given its efficiency and low power. Though ReRAM of larger size is still a research area.
- petra 2mo agoAMD can let the SRAM be on a different chip. Maybe even something similar to their 3D cache. that could increase density to 20B[1]. They could also move from 6nm to 2nm. that would probably increase density by another 3x to 60B. Add a bunch of chips together, and you get to a server that can run a 800B model, very fast and probably significantly cheaper than others. [1]https://www.eetimes.com/taalas-specializes-to-extremes-for-extraordinary-token-speed/ https://www.eetimes.com/taalas-specializes-to-extremes-for-e...
- coderbants 2mo agoI think the lifecycle for these chips could stretch far longer. If you're offering these models on a two year lifecycle, then you'd be able to stand up your top tier (wouldn't need to be frontier) at high speed. Run (for example) Kimi K3 on it and give it a brand name: AcmeAI Carbon Market it as your premier (only) model at high throughput. Two years later you stand up MSICs for the new state of the art with entirely new hardware, your lineup becomes: AcmeAI Nitrogen (top tier) AcmeAI Carbon (mid tier) If you just kept pushing the same model down your pricing tier over time you could still extract a lot of value from an old model, even years after it's been set in stone. Working on brand new code/frameworks? Pay to use the newest model. Working on legacy code? Use the lower tier models that will already know your legacy frameworks, pay far less and still get massive throughput. I've worked on a lot of government projects that this would be absolutely brilliant for. The other side of this is that agent harnesses are NOT set in stone, so even a legacy model with a knowledge cut-off that's years out of date can likely still be helped quite a bit by harness and fetch behaviours that are still developing rapidly. Especially at this kind of throughput.
- hypfer 2mo agoI'm somewhat doubtful that we will be seeing something as large as Kimi K3 in silicon any time soon. This tech can definitely scale up from the current 8B prototype, but - at least as far as my limited understanding of the tech involved goes - you cannot just ASIC a trillion weights model due to physical size constraints. ___ Specification HC1 Model Llama 3.1 8B (hardwired) Process TSMC 6nm Die size 815mm² ___ So the current prototype already pushes the limits of what we can fit on a single die, and that is already likely going to limit your yield.
- A_D_E_P_T 2mo agoKimi K3 is huge, though. Deepseek V4 Flash is a much more moderate model (284B total), and it works extremely well. Models of that size, and smaller, are just going to keep getting better and better. Presumably there's a threshold below which models are not generally useful or competitive, but if models-on-silicon can scale up to just 256B, that would be really remarkable.
- to11mtm 2mo agoThere's in-betweens; '1T SRAM' or eDRAM. Of course, 1T SRAM isn't really SRAM, but my understanding is it doesn't require external refresh like eDRAM, is a bit easier to fab on-die than eDRAM, and is half the mm2 per Megabit compared to real SRAM (15% more die size than eDRAM)...