2 ms·
This is just a VBR mode for neural networks. Not quite useful when inference is already quite slow.
by anticensor 1y ago
This is just a VBR mode for neural networks. Not quite useful when inference is already quite slow.
- vessenes 1y agoEven presuming this is an accurate summary, the conclusion is not accurate - most local LLM inference users are constantly trading off quality for speed, in that speed drops dramatically once RAM is full. So, if you think of speed at desired quality, this could be very useful.