3 ms·
Because there’s always some overhead during inference plus you don’t want to fill all your available RAM because you risk swapping to disk which will make every
by M4v3R 3y ago
Because there’s always some overhead during inference plus you don’t want to fill all your available RAM because you risk swapping to disk which will make everything slow to a crawl.
- swyx 3y agoso why is the overhead a 1/3 ratio instead of a constant amount? just testing the scaling assumption