3 ms·
Considering it takes 8x A100 GPUs (80GB VRAM) to train GPT-2, I think it'll take far more than a single 4090.
by auspiv 2y ago
Considering it takes 8x A100 GPUs (80GB VRAM) to train GPT-2, I think it'll take far more than a single 4090.
- bufo 2y agoThe RTX 4090 has about the same BF16 Tensor Core TOPs than the A100, assuming 50% MFU (like the A100 40 GB PCIe) it would take 8x longer on 1 RTX 4090 vs 8x A100 80GB SXM, so 12 hours. Datasheet here for the TOPs https://images.nvidia.com/aem-dam/Solutions/geforce/ada/nvidia-ada-gpu-architecture.pdf https://images.nvidia.com/aem-dam/Solutions/geforce/ada/nvid... 50% MFU should be achievable on the 4090.
- anthonix1 2y agoNah, I reproduced on 4x 7900 XTX machine in 8.75 hours, so a single 7900 XTX (costs less than $1k) could do it in under 24 hours. Was hitting 55.4% MFU.
- benterix 2y agoDo you have a link to a description of your setup? I assume there are some quirks on the way and getting this done requires some substantial work. A submission on that would be very interesting, at least to me.