3 ms·
> It was tested on 4 (A100 80g) and 8 (V100 32g) GPUs, but is able to work with different configurations with ≈200GB of GPU memory in total which divide weight
by htrp 4y ago
> It was tested on 4 (A100 80g) and 8 (V100 32g) GPUs, but is able to work with different configurations with ≈200GB of GPU memory in total which divide weight dimensions correctly (e.g. 16, 64, 128).
so we looking at crazy prices just for inference. RIP to the first guy's cloud billing account who makes this public
- jhoelzel 4y agoso err the cheapest A100 i could find was EUR 10.579,79 . Suddenly that 3090 i wanted to get, does not seem so expensive....
- chrisMyzel 4y agoWith DeepSpeed an SSD and lots of RAM you should be able to run inference with a 8GB card. There’s a thread somewhere on https://discuss.huggingface.co/ https://discuss.huggingface.co/ doing the math around this