2 ms·
What's the value running the smaller model too? Why not just the big model for everything? I note both are dense, as well.
by jauntywundrkind 4mo ago
What's the value running the smaller model too? Why not just the big model for everything? I note both are dense, as well.
- Ritewut 4mo agoTokens per second. The difference between 8B and something like 16B is not as big as you might think in practical usage and 8B is a lot faster and interactive than 16B but there are certain things where it is useful to farm it out to the large model.
- Natalia724 4mo agoAgree. For local coding help, latency often matters more than raw benchmark quality. A slightly weaker model that answers immediately changes how often you reach for it.
- cptskippy 4mo agoExactly this. Creating conversation titles and parsing HTML/JSON don't benefit from 27B models. The B70 can run both models comfortably side-by-side so it makes better use of time and resources.