5 ms·

Strong disagree. How is reality even remotely close to what your comment predicts? Only 40 years ago we thought one processor will be enough for all compute tas

Strong disagree. How is reality even remotely close to what your comment predicts? Only 40 years ago we thought one processor will be enough for all compute tasks, then we learned how to make software for bunch of CPUs, then we realized there are things that benefit from bunch of weaker cores and wrote software for that, then we realized you don't even need a single computer and you can parallelize your program to N computers with M processors. Your argument fails because it assumes everything is automate-able. It's not. Programming is not automate-able, at least not now. If you wait your C compiler to generate code that will run in L computers with N to M threading model you're gonna wait too long. This is what we software engineers are paid for. And it does seem to work, as explained above.

bnegreve 8y ago
> Only 40 years ago we thought one processor will be enough for all compute tasks, then we learned how to make software for bunch of CPUs, then we realized there are things that benefit from bunch of weaker cores and wrote software for that, then we realized you don't even need a single computer and you can parallelize your program to N computers with M processors. No, the history of parallel computing is a lot more chaotic than that! Parallel, distributed or vector computing graphics have been around since the early days and there are many fancy parallel processors architectures that never did well exactly because no ones knew how to program them. The Tera architecture (128 threads) comes to mind[1]. The Cell that was used in the playstation 3. It has 1 general purpose core and 8 accelerators that were vastly underexploited by game developers. The greatly anticipated Intel Larrabee (32 cores) never made it to the market. And even today the fate of the Xeon Phi (about 60 cores) remains very uncertain. In 2019 parallel computing is reserved to the same restricted set of easy to parallelize applications than 40 years ago (numerical simulations, 3D rendering, database servers and maybe few others) + deep learning. But most software is sequential, and most people would not notice if they had 1 or 2 cores instead of 8 or 16. That's because writing parallel programs is inherently and debugging is hell. [1] https://en.wikipedia.org/wiki/Cray_MTA https://en.wikipedia.org/wiki/Cray_MTA
Traster 8y ago
Maybe we have very different memories of the history of software. In my memory one of the defining issues of the 2000s was the move from the frequency race to the multi-core designs and how despite multi-core CPUs becoming common in the mid-2000s we still have significant software tools in the market today that STILL can't effectively parallelize to more than 1 core. The drive to multi-core design wasn't an epiphany that it was better, it was forced by the fact that we'd really reached the end of the road when it came to single core performance. To get performance we've been forced in that direction and it's still difficult. If you want to get performance on an Intel multi-core CPU you need to actually worry about concurrency which is already a serious complication versus writing software for a single threaded design. If you want to write software for a GPU not only are you going to need to worry about concurrency, you're probably going to have to actively think seriously about memory management as well as how to express MIMD constructs and often you're even talking about needing to go specifically to OpenCL to get that performance. Every time we add a hardware feature or concept that is not automatically handled by a compiler we create a new difficult problem for software engineers. It makes software harder to write, and very often those features go completely unused. Which is why it's going to be so important for benchmarks to really get into the meat and potatoes of what performance you will really get. Let me give you a concrete example - an Intel Stratix 10 can claim 10 TFLOPS of compute power on their website. The way they come up with that number is taking the number of DSP blocks on the FPGA, timesing that by 2 because they're Multiply-Accumulates and then timesing that by the maximum frequency the silicon can support which they (used to) claim is 1GHz. For the largest part that's 5760 x 2 x 800MHz = 9.2TFLOPS. (The reason for the discrepancy is they've found out the DSP block won't run at 1GHz but they haven't updated their headline marketing). Now, I've got friends who work on Intel FPGAs. Firstly, it'd be close to a miracle if you got 80% utilization. Secondly, frankly the chip literally won't work at 800MHz when full, it'll most probably run at 400MHz. So before you've even considered your algorithm, you've lost 60% of your FLOPS. Now onto the meat and potatoes, to compile a program for that device takes atleast 24 hours. So even accessing 40% of the peak performance requires such a huge amount of effort there will be a handful of companies that ever acheive that. That is why real benchmarks are important. It's not about peak performance, it's about what you can actually achieve with your design.

bnegreve 8y ago

> Only 40 years ago we thought one processor will be enough for all compute tasks, then we learned how to make software for bunch of CPUs, then we realized there are things that benefit from bunch of weaker cores and wrote software for that, then we realized you don't even need a single computer and you can parallelize your program to N computers with M processors. No, the history of parallel computing is a lot more chaotic than that! Parallel, distributed or vector computing graphics have been around since the early days and there are many fancy parallel processors architectures that never did well exactly because no ones knew how to program them. The Tera architecture (128 threads) comes to mind[1]. The Cell that was used in the playstation 3. It has 1 general purpose core and 8 accelerators that were vastly underexploited by game developers. The greatly anticipated Intel Larrabee (32 cores) never made it to the market. And even today the fate of the Xeon Phi (about 60 cores) remains very uncertain. In 2019 parallel computing is reserved to the same restricted set of easy to parallelize applications than 40 years ago (numerical simulations, 3D rendering, database servers and maybe few others) + deep learning. But most software is sequential, and most people would not notice if they had 1 or 2 cores instead of 8 or 16. That's because writing parallel programs is inherently and debugging is hell. [1] https://en.wikipedia.org/wiki/Cray_MTA https://en.wikipedia.org/wiki/Cray_MTA

Traster 8y ago

Maybe we have very different memories of the history of software. In my memory one of the defining issues of the 2000s was the move from the frequency race to the multi-core designs and how despite multi-core CPUs becoming common in the mid-2000s we still have significant software tools in the market today that STILL can't effectively parallelize to more than 1 core. The drive to multi-core design wasn't an epiphany that it was better, it was forced by the fact that we'd really reached the end of the road when it came to single core performance. To get performance we've been forced in that direction and it's still difficult. If you want to get performance on an Intel multi-core CPU you need to actually worry about concurrency which is already a serious complication versus writing software for a single threaded design. If you want to write software for a GPU not only are you going to need to worry about concurrency, you're probably going to have to actively think seriously about memory management as well as how to express MIMD constructs and often you're even talking about needing to go specifically to OpenCL to get that performance. Every time we add a hardware feature or concept that is not automatically handled by a compiler we create a new difficult problem for software engineers. It makes software harder to write, and very often those features go completely unused. Which is why it's going to be so important for benchmarks to really get into the meat and potatoes of what performance you will really get. Let me give you a concrete example - an Intel Stratix 10 can claim 10 TFLOPS of compute power on their website. The way they come up with that number is taking the number of DSP blocks on the FPGA, timesing that by 2 because they're Multiply-Accumulates and then timesing that by the maximum frequency the silicon can support which they (used to) claim is 1GHz. For the largest part that's 5760 x 2 x 800MHz = 9.2TFLOPS. (The reason for the discrepancy is they've found out the DSP block won't run at 1GHz but they haven't updated their headline marketing). Now, I've got friends who work on Intel FPGAs. Firstly, it'd be close to a miracle if you got 80% utilization. Secondly, frankly the chip literally won't work at 800MHz when full, it'll most probably run at 400MHz. So before you've even considered your algorithm, you've lost 60% of your FLOPS. Now onto the meat and potatoes, to compile a program for that device takes atleast 24 hours. So even accessing 40% of the peak performance requires such a huge amount of effort there will be a handful of companies that ever acheive that. That is why real benchmarks are important. It's not about peak performance, it's about what you can actually achieve with your design.