2 ms·
That's the size on disk, my man. When you quantize it to a smaller float size you lose precision on the weights and so the model is smaller. Then here they `mma
by renewiltord 4y ago
That's the size on disk, my man. When you quantize it to a smaller float size you lose precision on the weights and so the model is smaller. Then here they `mmap` the file and it only needs 6 GiB of RAM!
- deleted 4y ago[deleted]
- gliptic 4y agoThe size mentioned is already quantized (and to integers, not floats). mmap obviously doesn't do any quantization.