4 ms·

I usually like Herb Sutter's articles, but this feels a bit like special pleading to me. He made a prediction-- that the amount of hardware parallelism we'd ha

by cmccabe 14y ago

I usually like Herb Sutter's articles, but this feels a bit like special pleading to me. He made a prediction-- that the amount of hardware parallelism we'd have would be between 16 and 256-- and it didn't really turn out to be true. I would say that we now have something like between 8 and 12 way parallelism on most systems. If you take a 6-core i7 with hyperthreading, that's 12-way parallelism. Talking about specialty hardware like the Xeon Phi HPC accelerator (which most developers have never heard of, let alone programmed for), just doesn't make sense. Similarly, GPUs may be ubiquitous, but how many programmers have actually written for them? Very few. The reality is that CPU architects have done everything they possibly can to keep down the amount of parallelism. When you get transistors, you can use them for things like L1 cache, instead of adding more execution units. The bottom line is that it's hard to predict the future. We can read stuff from the past, like mailing list posts about how Linux is irrelevant because "we'll all be using Sparcs in a few years," but the lesson that it's hard to make predictions never really seems to sink in. I hope that we'll see more manycore chips in the future for developers to play with. The realities of physics seem to be dragging CPU architects kicking and screaming into the multicore world. But it may not be as fast as we once thought.

deleted 14y ago
[deleted]
sp332 14y ago
> GPUs may be ubiquitous, but how many programmers have actually written for them? Very few. It wasn't a prediction about the state of tools for programmers. The multi-threaded hardware is ubiquitous, but it's still hard to program for.
lambda 14y ago
But don't most machines that will be using CPUs that big have two sockets? Big servers these days come with 2 CPU sockets, 6 cores per CPU, 2 threads per code. So you do see 24 way parallelism pretty often, for server-side code at least. Now, this was pretty much the low end of his original prediction: 'If we stick with "just more of the same" as in Figure 2's extrapolation, we'd expect aggressive early hardware adopters to be running 16-core machines (possibly double that if they're aggressive enough to run dual-CPU workstations with two sockets), and we'd likely expect most general mainstream users to have 4-, 8- or maybe a smattering of 16-core machines (accounting for the time for new chips to be adopted in the marketplace).' It is true that new, mainstream systems offer 4 to 8 way parallelism, and that you see up to 24 way parallelism on the high end. So, his "more of the same" prediction was perfectly correct. We did not see the large jump that he hypothesized might happen. But if you notice, he said "But the gating factor is software that can use them effectively; specifically, the availability of scalable parallel mainstream killer applications. The only thing I can foresee that could prevent the widespread adoption of manycore mainstream systems in the next decade would be a complete failure to find and build some key parallel killer apps, ones that large numbers of people want and that work better with lots of cores." As you say, most programmers don't develop for these big many-core monstrosities. So he was right; software that takes advantage of multi-core machines is still a gating factor. The chips are available, but because that many cores does not help most software that hasn't been specially crafted for it, they are not mainstream. I think the one major fault is in his optimism that it's possible for most software to take good advantage of that many cores. Parallelism is hard. Correct software is more important than fast software. Heck, a lot of software these days is written in languages like JavaScript, Python, and Ruby, which are not known for their speed, but rather their ease of development. Telling people to get with the multi-core bandwagon, when really just writing large, correct software is in most cases more important that squeezing out the last ounce of performance, is not exactly productive. As GPUs have shown us, the best way to take advantage of that extra parallelism is to write special purpose libraries for the computationally intensive parts, and then just call out to that from simpler, less parallel code. So I'm not sure that the average programmer is going to spend that much time adapting their code for the parallel world, other than using existing libraries and frameworks for taking advantage of that. A database backed web app is a great example of this kind of parallelism. The hard parts of parallelism are generally handled by the database itself. The web app is usually written completely single threaded, but you can spin up multiple processes all talking to the same database to take advantage of your multiple cores. I suspect that is how we'll see parallelism play out in general. Most software written single threaded, with parallelism only by running multiple processes, talking to specially tailored databases, message queues, rendering libraries, numerical computing libraries, and the like.
- genwin 14y ago
  Good points. Parallel code is getting easier to write, e.g. the Go language, thus likely more common. I think web servers alone (serving more users per machine) could drive multi-core advances.

by cmccabe 14y ago

deleted 14y ago

[deleted]

sp332 14y ago

> GPUs may be ubiquitous, but how many programmers have actually written for them? Very few. It wasn't a prediction about the state of tools for programmers. The multi-threaded hardware is ubiquitous, but it's still hard to program for.

lambda 14y ago

But don't most machines that will be using CPUs that big have two sockets? Big servers these days come with 2 CPU sockets, 6 cores per CPU, 2 threads per code. So you do see 24 way parallelism pretty often, for server-side code at least. Now, this was pretty much the low end of his original prediction: 'If we stick with "just more of the same" as in Figure 2's extrapolation, we'd expect aggressive early hardware adopters to be running 16-core machines (possibly double that if they're aggressive enough to run dual-CPU workstations with two sockets), and we'd likely expect most general mainstream users to have 4-, 8- or maybe a smattering of 16-core machines (accounting for the time for new chips to be adopted in the marketplace).' It is true that new, mainstream systems offer 4 to 8 way parallelism, and that you see up to 24 way parallelism on the high end. So, his "more of the same" prediction was perfectly correct. We did not see the large jump that he hypothesized might happen. But if you notice, he said "But the gating factor is software that can use them effectively; specifically, the availability of scalable parallel mainstream killer applications. The only thing I can foresee that could prevent the widespread adoption of manycore mainstream systems in the next decade would be a complete failure to find and build some key parallel killer apps, ones that large numbers of people want and that work better with lots of cores." As you say, most programmers don't develop for these big many-core monstrosities. So he was right; software that takes advantage of multi-core machines is still a gating factor. The chips are available, but because that many cores does not help most software that hasn't been specially crafted for it, they are not mainstream. I think the one major fault is in his optimism that it's possible for most software to take good advantage of that many cores. Parallelism is hard. Correct software is more important than fast software. Heck, a lot of software these days is written in languages like JavaScript, Python, and Ruby, which are not known for their speed, but rather their ease of development. Telling people to get with the multi-core bandwagon, when really just writing large, correct software is in most cases more important that squeezing out the last ounce of performance, is not exactly productive. As GPUs have shown us, the best way to take advantage of that extra parallelism is to write special purpose libraries for the computationally intensive parts, and then just call out to that from simpler, less parallel code. So I'm not sure that the average programmer is going to spend that much time adapting their code for the parallel world, other than using existing libraries and frameworks for taking advantage of that. A database backed web app is a great example of this kind of parallelism. The hard parts of parallelism are generally handled by the database itself. The web app is usually written completely single threaded, but you can spin up multiple processes all talking to the same database to take advantage of your multiple cores. I suspect that is how we'll see parallelism play out in general. Most software written single threaded, with parallelism only by running multiple processes, talking to specially tailored databases, message queues, rendering libraries, numerical computing libraries, and the like.

genwin 14y ago
Good points. Parallel code is getting easier to write, e.g. the Go language, thus likely more common. I think web servers alone (serving more users per machine) could drive multi-core advances.