5 ms·
I had a thought a while back: sell large local models burned onto fused compute / ROM chips. Like cartridges for old game consoles. Slot (or probably plug into
by api 3mo ago
I had a thought a while back: sell large local models burned onto fused compute / ROM chips. Like cartridges for old game consoles. Slot (or probably plug into USB-C) and go.
It’s an ASIC with the model wired into it so it’s very low power and fast.
I’d buy these. Say $100 for a frontier class model. Maybe more.
- fendy3002 3mo agoYour statement reminds me of Avenger's scene where Tony choosing Friday among other AI's catridges to use. That sounds good and practical to happen!
- Gecko4072 3mo agoThis would be very compelling. Can anyone share more details on how it would work? Only issue is that you are stuck at a certain point in time but that’s not a huge deal. Even just a good 27b model would be useful.
- FridgeSeal 3mo agoTalaas have done this with a llama 3 model. Runs at like, 16k/tokens a second oror something obscene. Very little power draw too. Doesn’t need hbm or lots of memory, because the hardware can just forward the data straight to the next layer and you don’t need to round trip through memory. They claim to be working on an approach to make the underlying hardware a bit more reusable between models.
- foobar10000 3mo agoYeah, if you have a fixed llm topology, you can just effectively burns 2 top layers of the chip as Rom (model weights) - which has a per area density even better than dram - so it’s just attention and kv streaming that is hbm to sram transfer. Most big model weights will not fit a single reticle sized chip - so you’d have prob 30 different chips to split the model . And you’d need super fast chip to chip comms for the all-reduce and similar. So scaling to 1T models is hard - and a long lead time - but can be very power efficient.
- all2 3mo agoThere a lot of ways this could work. 1) the hardest, custom silicon + MCU to manage the USB interface 2) not as hard, shared memory, NPU + MCU to manage inference and USB interface Theoretically you could do 2 with the right MCU, NPU, and memory combo. You'd stream/DMA the weights from memory into the NPU and then read the results with the MCU. From a user's perspective, it might take the form of an openAI API compatible endpoint that enumerates when they plug the USB device in. There would likely be some host-side software to ease the pain of trying to use a USB device as an HTTP API.
- i5heu 3mo agoLooks like 27b models are in reach. > “In the current generation, our density is 8 billion parameters on the hard wired part of the chip., plus the SRAM to allow us to do KV caches, adaptations like fine tuning, and etc. In our next generation, we would have the ability to go up to 20 billion parameters in a chip. Even with trillions of parameters, we’re talking about few tens of chips, which is a very, very small compared to anything else out there on the market today.” https://www.nextplatform.com/compute/2026/02/19/taalas-etches-ai-models-onto-transistors-to-rocket-boost-inference/4092140 https://www.nextplatform.com/compute/2026/02/19/taalas-etche... Edit: i do not know how reliable this page is... it has a lot of typos.. for me it does not look like it was written by a LLM
- theturtletalks 3mo agoTaalas[0] seems to be what you're talking about. 0. https://taalas.com/ https://taalas.com/
- jdw64 3mo agoWow, I'd really love it if that were the case. I'm already pretty satisfied with just GPT 5.6 as it is.
- TiredOfLife 3mo agoKinda already exists. demo https://chatjimmy.ai/ https://chatjimmy.ai/ https://news.ycombinator.com/item?id=47103661 https://news.ycombinator.com/item?id=47103661
- ktzar 3mo agoHallucinates on the first question I ask, as 90% of these models that try to take shortcuts.
- phildenhoff 3mo agoYou’re expecting the wrong thing. The demo demonstrates the insane inference rate of dedicated hardware. Iirc it’s llama 3 or something. Not a very good model by today’s standards. But it runs at 16k tokens per second, an order of magnitude above the competition. Imagine what’s possible if you had GLM-5.2 turned into a hardware chip like this.
- versteegen 3mo agoIt's a custom 3 bit quant of Llama 3.1 8B and other shortcuts. The quant is not good. Their newer arch switches to standard 4 bit quants, should be far better!
- Cantinflas 3mo agoYou need terabytes of memory to run a frontier class model
- calgoo 3mo agoI wonder, if you can run at 8k or 15k t/s, you could in theory run 10 or 20 agents (or more) at the same time and generate hundreds of versions, then just analyze them. Think thinking mode x1000 at least... Would be interesting to see how good it would be
- est 3mo agoIf you look closely, 80% of those terabytes of memory is stationary model weights.
- glaslong 3mo agoI love this for the popular sci-fi trope too, where you see some ship engineer swap one glowing crystal "compute core" for another. We could have the photonic AI model ASICs for real!
- jazzyjackson 3mo agoOr Galatea’s personality chip in Bicentennial Man :)
- api 3mo agoIn Gibson’s Neuromancer AIs burned onto chips were literally discussed and called “ROM constructs.” In that case some of them were images of human minds but with no persistent state or memory. So basically a static model version of consciousness uploading.
- taegee 3mo agohttps://chatjimmy.ai https://chatjimmy.ai
- tybit 3mo agoTaalas is developing this, but not for Frontier class models. I hope that if we can least get the easy 80% of work done on that sort of hardware, we can greatly reduce the demand for GPUs, HBM and energy to some extent.
- wild_egg 3mo agoThere is an amount of brute forcing that becomes possible at those speeds that I think could even take us beyond 80%. If we could have Qwen3.6-27B running at 15k t/s, run 100 attempts concurrently, select top-K solutions and synthesize a final result from them. There was a paper a while back that showed top-K selection like that with tiny models was able to reliably solve some 1M-step Tower of Hanoi when no frontier model could. Very big level up in capability just from horizontally scaling compute.
- yomismoaqui 3mo ago100 dumb folks don't make an Einstein
- wild_egg 3mo agoYou pull out Einstein when you need a breakthrough.
- mdp2021 3mo agoBut some (Meta, Anthropic) suggested that optimizing and extending the "<think>" process can produce extra value. (I do not know if that requires an improved underlying architecture - frontier models architectures are sometimes not public.)
- Schlagbohrer 3mo agoTo extend this metaphor or the "million monkeys typing at a million keyboards", it occurs to me that if you can just slightly increase the intelligence of each sub-agent by a tiny amount, the overall end product can shoot up in quality tremendously. If I can make my small fast AI model just a tiny, tiny bit more capable, and still run 100 of them or 1000 and run an evaluation model on top of that, the overall system capability will scale quickly with tiny increases in base model intelligence.
- glimshe 3mo agoInterestingly, you could easily run them from the said old consoles! You'd just need a bit of console code to interface (text input/output) with your fully independent LLM subsystem. Imagine Claude for the NES without Internet?
- smokel 3mo ago> I’d buy these. Say $100 for a frontier class model. Maybe more. Sure you would. Running frontier class models on current hardware costs in the order of tens of thousands of dollars. It is more likely that these custom ASICs will be priced competitively with that, and not with Super Mario Bros. Oh, and energy consumption will be in the same order.
- k33P1Tr3aL 3mo agoisn't this essentially an FPGA?
- whywhywhywhy 3mo agoFPGA is reprogrammable, ASIC only does one thing forever.
- szundi 3mo ago[dead]
- popalchemist 3mo agoHow very Cyberdyne.
- LarsDu88 3mo agoI believes the weights are burned as ROM microcode, but for an effective inference speedup, you do want to burn the architecture (matmuls, activation functions, MoE gates, etc) as well which will differ from model to model. It's not as simple as a weight swap between identical architectures. The speed gains are also from not having to route the weights through wiring like with ROM cartridges.
- tomaskafka 3mo agoSounds like something DARPA would be working on right now.
- yieldcrv 3mo agoThats happening, in progress We’ll see what the market chooses
- matheusmoreira 3mo agoI'd absolutely buy these chips as well.
- AussieWog93 3mo agoIs suspect you're looking at more like $100k than $100 for an ASIC with enough memory and compute to handle Fable. But the flip side is possibly 1000+ tok/s on a SOTA model, which would be game changing. Could make sense for datacentres or enterprise, but I don't think we'll be getting SOTA Game Boy carts this decade.
- Schlagbohrer 3mo agoThis is a fun idea but they'll be outdated in 6 months and it would take 12 months to develop, build and ship them assuming they can design and manufacture these at superhuman speeds.