4 ms·
how much faster is 3b in practice? Seems like an uncommon size, so it would make sense to have the title of "best 3b" lol
by brrrrrm 3y ago
how much faster is 3b in practice? Seems like an uncommon size, so it would make sense to have the title of "best 3b" lol
- lhl 3y agoThe rule of thumb is that inference speed halves with every doubling of parameter size (and obviously a doubling of memory size). You can check out real world performances on devices here: https://llm.mlc.ai/ https://llm.mlc.ai/