3 ms·
The graphs ranking GPUs may not be accurate, as they don't represent real-world results and have factual inaccuracies. For example: > Shown is raw relative per
by ipsum2 4y ago
The graphs ranking GPUs may not be accurate, as they don't represent real-world results and have factual inaccuracies. For example:
> Shown is raw relative performance of GPUs. For example, an RTX 4090 has about 0.33x performance of a H100 SMX for 8-bit inference. In other words, a H100 SMX is three times faster for 8-bit inference compared to a RTX 4090.
RTX 4090 GPUs are not able to use 8-bit inference (fp8 cores) because NVIDIA has not (yet) made the capability available via CUDA.
> 8-bit Inference and training are much more effective on Ada/Hopper GPUs because of Tensor Memory Accelerator (TMA) which saves a lot of registers
Ada does not have TMA, only Hopper does.
If people are interested I can run my own benchmarks on the latest 4090 and compare it to previous generations.
- p1esk 4y agoWhen people say “8 bit inference” they mean INT8, not FP8.
- ipsum2 4y agoIt's pretty clear from the article that 8-bit inference is referring to FP8, as the author claims int8 is similar to fp16 inference performance. > Ada/Hopper also have FP8 support, which makes in particular 8-bit training much more effective. I did not model numbers for 8-bit training because to model that I need to know the latency of L1 and L2 caches on Hopper/Ada GPUs, and they are unknown and I do not have access to such GPUs. On Hopper/Ada, 8-bit training performance can well be 3-4x of 16-bit training performance if the caches are as fast as rumored. For old GPUs, Int8 inference performance for old GPUs is close to the 16-bit inference performance.
- varunkmohan 4y agoWould be curious to see your benchmarks. Btw, Nvidia will be providing support for fp8 in a future release of CUDA - https://github.com/NVIDIA/TransformerEngine/issues/15 https://github.com/NVIDIA/TransformerEngine/issues/15 I think TMA may not matter as much for consumer cards given the disproportionate amount of fp32 / int32 compute that they have. Would be interesting to see how close to theoretical folks are able to get once CUDA support comes through.
- touisteur 4y agoWell every since I've read these papers doing FFTs faster than cuFFT using tensor cores (although FFT isn't supposed to be helped by more flops but only better memory bandwidth) and also fp32-level accuracy on convolutions with 3x tf32 tensor cores sweeps (available in cutlass) I'm quite ready to believe some hype about TMA. Anything improving memory bandwidth for any compute part of the GPU is welcome. Also I'd like for someone to crack open RT cores and get the ray-triangle intersection acceleration out of Optics. Have you seen the FLOPS on these things?
- touisteur 4y agoBefore I forget, here's the link to the tcFFT paper https://ar5iv.labs.arxiv.org/html/2104.11471v1 https://ar5iv.labs.arxiv.org/html/2104.11471v1 And the fp32-gemm-with-tf32 https://arxiv.org/abs/2203.03341 https://arxiv.org/abs/2203.03341
- lostmsu 4y agoCan you please run bench from https://github.com/karpathy/nanoGPT https://github.com/karpathy/nanoGPT ?
- onnodigcomplex 4y agoI'm not on my desktop, but I used nanoGPT extensively on my RTX-4090. I trained a GPT2-small but with a small context window (125 > 90M params) at batch sizes of 1536 using gradient accumulation (3*512). This runs at just a smidge over 1 it/s. Some notes - Gradient checkpointing and 2048 batch size in a single go allows ~10% performance improvement on a per sample basis. - torch.compile doesn't work for me yet (lowest cuda version I got my 4090 to run on was 11.8 but highest cuda version on which I got the model to compile is 11.7). - I did the optimalisations in https://arxiv.org/abs/2212.14034 https://arxiv.org/abs/2212.14034
- lostmsu 4y agoCan you share the code and numbers so that I could compare directly with my 3090? Do you train in fp16/bf16? Have you tried fp8?