3 ms·
People are working on it but it's just down to how many floating point operations can you do? I wonder if something like the Coral would help? I'd love having
by plagiarist 3y ago
People are working on it but it's just down to how many floating point operations can you do? I wonder if something like the Coral would help?
I'd love having an LLM on a Pi, but I'll have to settle for a larger machine I can turn on to get more compute. At least for the time being.
- filterfiber 3y agoThe current bottleneck for most current hardware is RAM capacity than memory bandwidth and last is FLOPS/TOPS. The coral has 8 MB of SRAM which uh, won't fit the 2GB+ that nearly any decent LLM require even after being quantized. LLMs are mostly memory and memory bandwidth limited right now.