4 ms·
Painfully slow tok/s though.
by khuey 2mo ago
Painfully slow tok/s though.
- NekkoDroid 2mo agoIsn't it more s/tok?
- nekusar 2mo agoI didn't say it was fast! But its also a 800B sized model running on a ram constrained system with no GPU. Techniques, GPUs, more ram, and faster disks can always speed it up. But the point being is they run on low end machines now. Its now an optimization problem, not a possibility assessment.