3 ms·
whats the intuition for 2/3 of RAM?
by swyx 3y ago
whats the intuition for 2/3 of RAM?
- avereveard 3y agoyou need some leftover for holding the context
- M4v3R 3y agoBecause there’s always some overhead during inference plus you don’t want to fill all your available RAM because you risk swapping to disk which will make everything slow to a crawl.
- swyx 3y agoso why is the overhead a 1/3 ratio instead of a constant amount? just testing the scaling assumption