3 ms·
4bit should take up less than this, there are quite a few shared parameters between experts. But unless you’re running bs=1 it will be painful vs 8x GPU as you
by qeternity 3y ago
4bit should take up less than this, there are quite a few shared parameters between experts.
But unless you’re running bs=1 it will be painful vs 8x GPU as you’re almost certain to be activating most/all of the experts in a batch.