5 ms·
I'm surprised people are surprised. Of course this is possible, and of course this is the future. This has been demonstrated already: why do you think we even h
by bsenftner 8mo ago
I'm surprised people are surprised. Of course this is possible, and of course this is the future. This has been demonstrated already: why do you think we even have GPUs at all?! Because we did this exact same transition from running in software to largely running in hardware for all 2D and 3D Computer Graphics. And these LLMs are practically the same math, it's all just obvious and inevitable, if you're paying attention to what we have, what we do to have what we have.
- the__alchemist 8mo agoI believe this is a CPU/GPU vs ASIC comparison, rather than CPU vs GPU. They have always(ish) coexisted, being optimized for different things: ASICs have cost/speed/power advantages, but the design is more difficult than writing a computer program, and you can't reprogram them. Generally, you use an ASIC to perform a specific task. In this case, I think the takeaway is the LLM functionality here is performance-sensitive, and has enough utility as-is to choose ASIC.
- GTP 8mo agoThe middle ground here would be an FPGA, but I belive you would need a very expensive one to implement an LLM on it.
- dogma1138 8mo agoFPGAs would be less efficient than GPUs. FPGAs don’t scale if they did all GPUs would’ve been replaced by FPGAs for graphics a long time ago. You use an FPGA when spinning a custom ASIC doesn’t makes financial sense and generic processor such as a CPU or GPU is overkill. Arguably the middle ground here are TPUs, just taking the most efficient parts of a “GPU” when it comes to these workloads but still relying on memory access in every step of the computation.
- jgalt212 8mo agoI thought it was because the number logic elements in a GPU is orders of magnitude higher than in a FPGA, rather than just processing speed. And GPU processing is inherently parallel so the GPU beats the FPGA just based on transistor count.
- dogma1138 8mo agoWith FPGA you are sacrificing performance for flexibility you are far less efficient in transistors for any given task than with a dedicated ASIC even if it’s a general compute ASIC like a GPU is today. The reason no one is building large FPGAs is that there is no market for them. If an H200 scale FPGA was viable we would have one.
- RobotToaster 8mo agoIt reminds me of the switch from GPUs to ASICs in bitcoin mining. I've been expecting this to happen.
- hkt 8mo agoHeh, I said this exact thing in another thread the other day. Nice to see I wasn't the only one thinking it.
- yunohn 8mo agoBut the BTC mining algorithm has not and will not change. That’s the only reason ASICs atleast make a bit of sense for crypto. AI being static weights is already challenged with the frequent model updates we already see - but may even be a relic once we find a new architecture.
- dangus 8mo agoSounds to me like there’s potential to use these for established models to provide cost/scale advantage while frontier models will run in the existing setup.
- yunohn 8mo agoIME llama et all require LoRA or fine-tuning to be usable. That's their real value vs closed source massive models, and their small size makes this possible, appealing, and doable on a recurring basis as things evolve. Again, rendering ASICs useless.
- fxnn 8mo agoRead the blog post. It mentions that their chip has a small SRAM which can store LoRA.
- yunohn 8mo agoNeither the blog nor Taalas' original post specify what speed to expect when using the SRAM in conjunction with the baked-in weights? To be taken seriously, that is really necessary to explain in detail, than a passing mention.
- IshKebab 8mo ago> Because we did this exact same transition from running in software to largely running in hardware for all 2D and 3D Computer Graphics. We transitioned from software on CPUs to fixed GPU hardware... But then we transitioned back to software running on GPUs! So there's no way you can say "of course this is the future".
- JKCalhoun 8mo ago"This has been demonstrated already…" I think burning the weights into the gates is kinda new. ("Weights to gates." "Weighted gates"? "Gated weights"?)
- dogma1138 8mo agoNot really new, this is 80’s-90’s Neuron MOS Transistor. It’s also not that different than how TPUs work where they have special registers in their PEs for weights.
- brookst 8mo agoGeights? Wates?
- learn_more 8mo agogweights
- Zetaphor 8mo agoIs this not effectively the same thing as a Bitcoin ASIC?
- pwarner 8mo agoI'd be kind of shocked if Nvidia isn't playing with this. I don't expect it's like super commercially viable today, but for sure things need to trend to radically more efficient AI solutions.
- saati 8mo agoThese are chips that become e-waste the second a better a model comes out, and nvidia is already limited by TSMC capacity.
- cyanydeez 8mo agoOnly in VC backed funding land. In the real world, theres talking refrigerators who dont need to know how to recite shakespeare.
- HPsquared 8mo agoOn the upside, Shakespeare isn't going to change soon.
- MarsIronPI 8mo agoSo you're saying we should burn Shakespeare onto a chip? /s
- throwthrowuknow 8mo agothese aren’t made for general chatbot use
- hamdingers 8mo agoThis is a ridiculous mindset. Llama 3.1 8B can do lots of things today and it'll still be able to do those things tomorrow. If you baked one of these into a smart speaker that could call tools to control lights and play music, it will still be able to do that when Llama 4 or 5 or 6 comes out.
- rembal 8mo agoIt's not certain this is the future: the obvious trade off is lack of flexibility: not only when a new model comes out, but also varying demand in the data centers - one day people want more LLM queries, another day more diffusion queries. Aaand, this blocks the holly grail of self improving models, beyond in-context learning. A realistic use case? More efficient vision based drone targeting in Ukraine/Taiwan/ whatevers next. That's the place where energy efficiency, processing speed, and also weight is most critical. Not sure how heavy ASICS are though, bit they should be proportional to the model size. I heard many complaints about onboard AI 'not being there yet', and this may change it. Not listing middle east as there is no serious jamming problem there.
- luckydata 8mo agoIt doesn't have be to true for all models to be useful. Thinking about small models running on phones or edge devices deployed in the field that would be a perfect use case for a "printed model".
- darkwater 8mo agoIn a not-too-distant future (5 years?) small LLMs will be good enough to be used as generic models for most tasks. And if you have a dedicated ASIC small enough to fit in an iPhone, you have a truly local AI device with the bonus point that you get something really new to sell in every new generation (i.e. acces to an even more powerful model)
- throwthrowuknow 8mo agoit doesn’t need to go in the phone if it only takes a few milliseconds to respond and is cheap
- hamdingers 8mo agoIt does if you care about who can access to your tokens
- yunwal 8mo ago
- theptip 8mo agoAre people surprised? I think the interesting point is the transition time. When is it ROI-positive to tape out a chip for your new model? There’s a bunch of fun infra to build to make this process cheaper/faster and I imagine MoE will bring some challenges.
- MarsIronPI 8mo agoDoesn't Google have custom TPUs that are kind of a halfway point between Taalas' approach and a generic GPU? I wonder if that kind of hardware will reach consumers. It probably will, though as I understand them NPUs aren't quite it.
- dyauspitr 8mo agoJob specific ASICs are are “old as time.”