4 ms·
The rule of thumb is that inference speed halves with every doubling of parameter size (and obviously a doubling of memory size). You can check out real world
by lhl 3y ago
The rule of thumb is that inference speed halves with every doubling of parameter size (and obviously a doubling of memory size).
You can check out real world performances on devices here: https://llm.mlc.ai/ https://llm.mlc.ai/