3 ms·
I believe the HF pipeline can do this already and I assume each stage uses more than 4 GB vram. There are other tricks they open source community will come up w
by ShamelessC 3y ago
I believe the HF pipeline can do this already and I assume each stage uses more than 4 GB vram. There are other tricks they open source community will come up with though.
- brucethemoose2 3y ago8 bit (and 4 bit?) quantization is low hanging fruit, assuming its not already running in 8 bit. An 8GB requirement kinda sounds like they have already quantized the model though.