3 ms·
Do they bake in the actual weights or the architecture? If it's just the architecture I don't understand where a speedup that considerable can come from.
by mysterEFrank 2y ago
Do they bake in the actual weights or the architecture? If it's just the architecture I don't understand where a speedup that considerable can come from.
- htthbjk 2y agoThey mention something about not wasting space for unnecessary memory.
- jasonni 2y agofrom their announcement, "Isn’t inference bottlenecked on memory bandwidth, not compute?", it seems weights are still in memory. It may have limit onchip cache for computing. Input tokens go through a batch pipeline to relieve memory bottleneck. Similar to Groq.