4 ms·
How much VRAM does it use during inference?
by djoldman 5y ago
How much VRAM does it use during inference?
- stellaathena 5y ago~40 GB with standard optimization. I suspect you can shrink it down more with some work, but it would require significant innovation to cram it into the next largest common chip size (24 GB, unless I’m misremembering)
- komuher 5y agoIs 40GB already on float16?
- stellaathena 5y agoYes