> Only 40 years ago we thought one processor will be enough for all compute tasks, then we learned how to make software for bunch of CPUs, then we realized ther

> Only 40 years ago we thought one processor will be enough for all compute tasks, then we learned how to make software for bunch of CPUs, then we realized there are things that benefit from bunch of weaker cores and wrote software for that, then we realized you don't even need a single computer and you can parallelize your program to N computers with M processors. No, the history of parallel computing is a lot more chaotic than that! Parallel, distributed or vector computing graphics have been around since the early days and there are many fancy parallel processors architectures that never did well exactly because no ones knew how to program them. The Tera architecture (128 threads) comes to mind[1]. The Cell that was used in the playstation 3. It has 1 general purpose core and 8 accelerators that were vastly underexploited by game developers. The greatly anticipated Intel Larrabee (32 cores) never made it to the market. And even today the fate of the Xeon Phi (about 60 cores) remains very uncertain. In 2019 parallel computing is reserved to the same restricted set of easy to parallelize applications than 40 years ago (numerical simulations, 3D rendering, database servers and maybe few others) + deep learning. But most software is sequential, and most people would not notice if they had 1 or 2 cores instead of 8 or 16. That's because writing parallel programs is inherently and debugging is hell. [1] https://en.wikipedia.org/wiki/Cray_MTA https://en.wikipedia.org/wiki/Cray_MTA