4 ms·
I was expecting to see how they deploy, maximum model size and tokens/s.
by visarga 2y ago
I was expecting to see how they deploy, maximum model size and tokens/s.
- telgareith 2y agoAnswer: smol, and a fraction.
- bhouston 2y agoThe 1B to 3B models should be runnable on Raspberry PI 5 with 8GB. That seems reasonable. Probably equivalent to the new Apple Intelligence that runs locally on device in terms of performance if you can use the same model.
- telgareith 2y agoYou're joking right? a rpi5 isn't in the same league as Apple SoCs.