3 ms·
Why quantize something that is already very small (270mb)?
by Havoc 3y ago
Why quantize something that is already very small (270mb)?
- pietz 3y agoJust making up stuff here, but smaller models are great for serverless compute like functions, which would also benefit from lighter computation. Don't forget, some people are dealing with hundreds of millions of documents. Accelerating this by 4x may be worth a small performance hit.