4 ms·
I believe this is a CPU/GPU vs ASIC comparison, rather than CPU vs GPU. They have always(ish) coexisted, being optimized for different things: ASICs have cost/
by the__alchemist 8mo ago
I believe this is a CPU/GPU vs ASIC comparison, rather than CPU vs GPU. They have always(ish) coexisted, being optimized for different things: ASICs have cost/speed/power advantages, but the design is more difficult than writing a computer program, and you can't reprogram them.
Generally, you use an ASIC to perform a specific task. In this case, I think the takeaway is the LLM functionality here is performance-sensitive, and has enough utility as-is to choose ASIC.
- GTP 8mo agoThe middle ground here would be an FPGA, but I belive you would need a very expensive one to implement an LLM on it.
- dogma1138 8mo agoFPGAs would be less efficient than GPUs. FPGAs don’t scale if they did all GPUs would’ve been replaced by FPGAs for graphics a long time ago. You use an FPGA when spinning a custom ASIC doesn’t makes financial sense and generic processor such as a CPU or GPU is overkill. Arguably the middle ground here are TPUs, just taking the most efficient parts of a “GPU” when it comes to these workloads but still relying on memory access in every step of the computation.
- jgalt212 8mo agoI thought it was because the number logic elements in a GPU is orders of magnitude higher than in a FPGA, rather than just processing speed. And GPU processing is inherently parallel so the GPU beats the FPGA just based on transistor count.
- dogma1138 8mo agoWith FPGA you are sacrificing performance for flexibility you are far less efficient in transistors for any given task than with a dedicated ASIC even if it’s a general compute ASIC like a GPU is today. The reason no one is building large FPGAs is that there is no market for them. If an H200 scale FPGA was viable we would have one.
- RobotToaster 8mo agoIt reminds me of the switch from GPUs to ASICs in bitcoin mining. I've been expecting this to happen.
- hkt 8mo agoHeh, I said this exact thing in another thread the other day. Nice to see I wasn't the only one thinking it.
- yunohn 8mo agoBut the BTC mining algorithm has not and will not change. That’s the only reason ASICs atleast make a bit of sense for crypto. AI being static weights is already challenged with the frequent model updates we already see - but may even be a relic once we find a new architecture.
- dangus 8mo agoSounds to me like there’s potential to use these for established models to provide cost/scale advantage while frontier models will run in the existing setup.
- yunohn 8mo agoIME llama et all require LoRA or fine-tuning to be usable. That's their real value vs closed source massive models, and their small size makes this possible, appealing, and doable on a recurring basis as things evolve. Again, rendering ASICs useless.
- fxnn 8mo agoRead the blog post. It mentions that their chip has a small SRAM which can store LoRA.
- yunohn 8mo agoNeither the blog nor Taalas' original post specify what speed to expect when using the SRAM in conjunction with the baked-in weights? To be taken seriously, that is really necessary to explain in detail, than a passing mention.