4 ms·
That’s actually kinda impressive for an 8b model. Normally my experience with them is that they’re not really useful.
by Aeolun 1y ago
That’s actually kinda impressive for an 8b model. Normally my experience with them is that they’re not really useful.
- conwayanderson 1y agoOnly 2b active also - very fast
- lawlessone 1y agoCan run it on a phone then? Seems like it could be somewhat useful for people with poor eyesight or blindness
- conwayanderson 1y agoIn terms of size yes, but I think it needs some work to get the model in the right format couple people got it running on a raspberry pi though
- apwell23 1y agosorry what does it mean for only 2b to be active?
- simonw 1y agoMy understanding is that, while all 8B are loaded into memory, for each token inference step only 2B are selected and used - so tokens are produced faster because there is less computation needed. Hoping someone will correct me if that's not the right mental model!