3 ms·
I find it super interesting that we're now in an era where we have LLM's that are quantized to binary weights - 1's and 0's. So effectively they're digital neur
by davedx 3mo ago
I find it super interesting that we're now in an era where we have LLM's that are quantized to binary weights - 1's and 0's. So effectively they're digital neural networks.
I assume that in addition to the significant memory savings, this should also lead to much simpler matrix multiplication operations? Could models like these run on CPU's efficiently, or does the geometry of the compute mean GPU's are still a better choice?
- jboss10 3mo agoMost of the time, the speed of these models are constrained by memory bandwidth. GPUs normally have much more memory bandwidth.
- davedx 3mo agoI'd expect the memory bandwidth to be the same for the CPU and GPU under a unified memory architecture like Apple silicon uses?
- kmacdough 3mo agoIts not about geometry, it's a parallel compute thing thing. CPUs typically don't have more than 10 or 20 cores. GPU have 100s to 1000s. Matmul is very well parallelized. More, lower power cores will always pay off handsomely.