3 ms·
Why not 3090s? Same VRAM and cheaper. With both setups you'd be limited to 1B. By contrast, you can run 4-bit quants of Llama 70B on two {3,4}090s, and it's sti
by halyconWays 2y ago
Why not 3090s? Same VRAM and cheaper. With both setups you'd be limited to 1B. By contrast, you can run 4-bit quants of Llama 70B on two {3,4}090s, and it's still pretty lobotomized by modern standards.
You can also train your own model even without GPUs. Just depends on parameter size.
- sabareesh 2y agoIt is previous architecture and it doesnt support newer version of Flash Attention , fp8 training etc
- halyconWays 2y agoThat's fair. I did run into that issue when trying to speed up Hunyuan
- jszymborski 2y agoIt is, however, like 3x cheaper.