5 ms·
what do you use "the most powerful language model for its size" for?
by yieldcrv 3y ago
what do you use "the most powerful language model for its size" for?
- ShrigmaMale 3y agoProbably a better candidate for local applications? Guessing this was a trial balloon against larger models coming though.
- snowram 3y ago7B models are small enough to be usable on a smartphone, so a local handheld assistant sounds like a use case.
- ComputerGuru 3y agoAre they? Unquantized, Llama 2 7b needs over 14GB of GPU (or shared) memory.
- polygamous_bat 3y ago"Unquantized" is the key word here: with quantization you can get a 4-8x improvement without much performance degradation.
- throeaaysn 3y ago[dead]
- minimaxir 3y ago"usable" is not the same as practical. Even running a quantized and optimized LLM on a smartphone would kill battery life at minimum.
- brucethemoose2 3y agoTry MLC-LLM. Its not as bad as you'd think. In the future(?), they will probably use the AI blocks instead of the GPU, which are very low power.
- linkpuff 3y agoAccording to their website, it's "Optimal for: low latency, text summarisation, classification, text completion, code completion"
- kirill5pol 3y agoOne big one is speculative decoding for larger models, the more “accurate”* the smaller model, the more the speed up on the bigger model * as in matches the token that the larger model would output
- sroussey 3y agoIt would be better if they also had a 70B model for this. They would need the same vocabulary, etc. What else?