2 ms·
The free lunch isn't quite over, although significant advancements were made in parallel computing... it turns out that CPUs have been able to "auto-parallelize
by dragontamer 5y ago
The free lunch isn't quite over, although significant advancements were made in parallel computing... it turns out that CPUs have been able to "auto-parallelize" your sequential code all along. Just not nearly as efficiently as explicitly parallel methodologies.
In 2005, your typical CPU was a 2.2GHz Athlon 64 3700+. In 2021, your typical CPU is a Ryzen 5700x at 3.8 GHz.
Single-threaded performance is far better than 72% faster however. The Ryzen 5700x has far more L3 cache, far more instructions-per-clock, far more execution resources than the ol' 2005 era Athlon.
In fact, server-class EPYC systems are commonly in the 2GHz range, because low-frequency saves a lot on power and servers want lower power usage. Today's EPYCs are still far faster per-core than the Athlons of old.
-------------
This is because your "single thread" is executed more-and-more in parallel today. Thanks to the magic of "dependency cutting" compilers, the compiler + CPU auto-parallelizes your code and runs them on the 8+ execution pipelines found on modern CPU "cores".
Traditionally, the CPUs in the 90s had a singular pipeline. But the 90s and 00s brought forth out of order execution, as well as parallel execution pipelines (aka: superscalar execution). That means 2 or more pipelines execute your "sequential" code, yes, in parallel. Modern cores have more than 8 pipelines and are more than capable of 4+ or 6+ operations per clock tick.
This is less efficient than explicit, programmer given parallelism. But it is far easier to accomplish. The Apple M1 continues this tradition of wider execution. I'm not sure if "sequential" is dead quite yet (even if there's a huge amount of machine working to auto-translate sequential into parallel technically... our code is largely written in a "sequential" fashion)
-------------
But the big advancements after 2005 was the rise of GPGPU compute. It was always known that SIMD (aka: GPUs) were the most parallel systems, from the late 1980s and early 90s the SIMD supercomputers always had the most FLOPs.
OpenCL and CUDA really took parallelism more / SIMD mainstream. And indeed: these SIMD systems (be it AMD MI100 or NVidia A100) are far more efficient and far higher compute capabilities than anything else.
The only "competitor" on the supercomputer scale is the Fugaku supercomputer, with SVE (512-bit SIMD) ARM using HBM RAM (same as the high-end GPUs). SIMD seems like the obvious parallel compute methodology if you really need tons and tons of compute power.