AIUI an FP32 core is only 32 bits wide, but this is outside my area of expertise really. Also note that CPUs also have additional ALUs that can't do FMAs, FMA

AIUI an FP32 core is only 32 bits wide, but this is outside my area of expertise really. Also note that CPUs also have additional ALUs that can't do FMAs, FMA is just the most capable one. You're also repeating 2×512 / 4×256 — that's per core, you need to multiply by CPU core count. [also, note e.g. an 8-core CPU is much cheaper than a H100 card ;) — if anything you'd be comparing the highest end server CPUs here. An 192-core Zen5c is 8.2~10.5k€ open retail, an H100 is 32~35k€…] [reading through some random docs, a CPU core seems vaguely comparable to a SM; a SM might have 128 or 64 lanes (=FP32 cores) while a CPU only has 16 with AVX-512, but there is indeed also a notable clock difference and far more flexibility otherwise in the CPU core (which consumes silicon area)]