4 ms·

Already is possible. On a machine with 32GB ram, and NO gpu. Just need a large SSD or NVME. Streams from disk to memory. https://github.com/JustVugg/colibri ht

by nekusar 3mo ago

Already is possible. On a machine with 32GB ram, and NO gpu. Just need a large SSD or NVME. Streams from disk to memory. https://github.com/JustVugg/colibri https://github.com/JustVugg/colibri

khuey 3mo ago
Painfully slow tok/s though.
- NekkoDroid 3mo ago
  Isn't it more s/tok?
- nekusar 3mo ago
  I didn't say it was fast! But its also a 800B sized model running on a ram constrained system with no GPU. Techniques, GPUs, more ram, and faster disks can always speed it up. But the point being is they run on low end machines now. Its now an optimization problem, not a possibility assessment.
matheusmoreira 3mo ago
That's fucking awesome!