4 ms·
It's worth noting that the 4bit quants can run on cpu at ~reading speed, which should unlock many use cases - especially if we can precompute some of the result
by RandomBK 3y ago
It's worth noting that the 4bit quants can run on cpu at ~reading speed, which should unlock many use cases - especially if we can precompute some of the results async.
- minimaxir 3y agoThat assumes full CPU utilization, with no other tasks heavily using the CPU. In the case of high-end video games, that's unlikely.
- RandomBK 3y agoTrue, yet many games are effectively single-threaded anyways. The bigger problem is memory capacity and bandwidth, but I suspect folks will eventually figure out some sort of QoS setup to let the system crunch LLMs using otherwise unused/idle resources.
- somnic 3y agoResource constraints would be a concern, yeah, so if you were developing a game featuring LLMs (which would, at this point in their development and maturity, be a gimmick) you would keep that in mind and keep other demands on resources low.
- ilaksh 3y agoAlthough in my testing the 4bit reasoning was not nearly as good.