5 ms·

The point is that even the most parallel programs tend to have bottlenecks where only one thread can proceed. As you add more cores, these bottlenecks increase

by jbooth 14y ago

The point is that even the most parallel programs tend to have bottlenecks where only one thread can proceed. As you add more cores, these bottlenecks increase in relative importance to the parallel parts, which are easily dealt with by your additional parallel capacity.

marshray 14y ago
Did anyone (who writes parallel code or purchases CPUs professionally) really not know this? I'm not trying to pick on the article, I like it overall. I took an interest in the Anand article when I learned you could build a 64-core box for about $4000. This is the article it's in response to http://www.anandtech.com/show/5057/the-bulldozer-aftermath-delving-even-deeper http://www.anandtech.com/show/5057/the-bulldozer-aftermath-d... the crux of which is: Some, including sources inside AMD, have blamed Global Foundries for not delivering higher clocked SKUs. Sure, the clock speed targets for Interlagos were probably closer to 3GHz instead of 2.3GHz. But that does not explain why the extra integer cores do not deliver. We were promised up to 50% higher performance thanks to the 33% extra cores, but we got 20% at the most. So the real questions seem to be: Is Interlagos' perfomance disappointing on pure-parallel benchmarks as well, or just those with a noticeable serial bottleneck? If there is poor scalability, what is the cause? and perhaps most importantly: Will my kind of workload actually benefit from Interlagos' more cores? I think that probably what's happening is that even the pure parallel benchmarks are not scaling linearly relative to the previous generation of chips, the reason is a combination of architectural changes in the core itself and memory bus contention, and the target market for these chips is VPS hosts running hundreds of VM images.
- jbooth 14y ago
  I agree with everything you're saying, I just think the 'obvious point' is one that's papered over in the public discourse a lot because it's not exciting. It is interesting why they don't scale linearly and I agree that it's probably a combination of memory bus contention and/or the chip itself being a little slower. The whole bulldozer thing seemed like a marketing stunt rather than thoughtful pursuit of higher performance to me. Given the frequency and cost of cache miss rates, Intel HT seems to get the same work done on fewer watts.
  - marshray 14y ago
    I really don't think AMD would have bet their next generation of chips on something they believed was a marketing stunt. Once the upcoming chip changes get explained to the Marketing Department, however, maybe it kinda starts to sound like Marketing.
  - frozenport 14y ago
    Intel HT is often disabled on HPC clusters due to performance problems. Also many people refer to the IL as a 8 core CPU due to the problems mentioned above. Much like not using the HT, many codes disable half the AMD chip.
    - jbooth 14y ago
      Yeah, but it's really only on highly-optimized cache-coherent vector/matrix math that it's a problem. On just about every other piece of software, it works great. I feel like AMD saw the benchmarks available for matrix math and went for it, rather than thinking about what real-world performance profiles looked like.
CookWithMe 14y ago
"the most parallel programs" are those that run on supercomputers / huge clusters, and they don't have these bottlenecks... But for the average parallel programs, you are (currently) right, I guess.
- jbooth 14y ago
  You've still got a driver program that coordinates the massively parallel stages, most likely, and a less-scalable dispatch process where not all nodes are yet doing work.
  - CookWithMe 14y ago
    Yes, there definitely are programs where this holds true (as I have said), but there are others where it does not. A simple, but perfectly reasonable example is brute-force search for cracking a password (if you got the hash of the password). Your dispatch is simple: send every node 1) the hash, 2) the number of nodes, 3) it's own number. With 2) and 3), each node can easily figure out where it starts. You need no more communication until one of the nodes finds the password. See https://en.wikipedia.org/wiki/Embarrassingly_parallel https://en.wikipedia.org/wiki/Embarrassingly_parallel Again, you said "most parallel", and not "average parallel", and this is what I have commented on.
- marshray 14y ago
  Keep in mind, https://en.wikipedia.org/wiki/Amdahl%27s_law https://en.wikipedia.org/wiki/Amdahl%27s_law was posed at the very beginning of parallel computing. Seymour Cray later became famous both for his fast uniprocessor supercomputer and his quip "If you were plowing a field, which would you rather use: Two strong oxen or 1024 chickens?" For most embarrassingly parallel problems, a Hadoop or BOINC-like cluster made from commodity hardware may always be cheapest. But "real" supercomputers and clusters will spend the most money to reduce bottlenecks precisely because they want to run those programs that are the most limited by them.
  - CookWithMe 14y ago
    I know, I actually read the paper like a year ago. Actually, what was turned later into Amdahl's law is more or less a side-note in the paper :) The parent said "most parallel programs", which I translated to embarrassingly parallel problems. > But "real" supercomputers and clusters will spend the most money to reduce bottlenecks precisely because they want to run those programs that are the most limited by them. I agree, but from the talks I have heard most time (= money) is spent trying to work around these bottlenecks as good as possible in software (i.e. incremental algorithms, pipelining, ...) to keep the machines busy. Then again, I hear a lot more talks about software than hardware, so you may be very right. If you have any particular example in mind, I would love to read about it!
rm999 14y ago
> even the most parallel programs tend to have bottlenecks There's a big class of "embarrassingly parallel" tasks, which usually are only limited by how quickly you can move your data to where it needs to be.
- marshray 14y ago
  What makes something EP is that there is very little data (relative to the CPU time) that needs to be moved. For example, to compute a nice image of points near the Mandelbrot set, you only need to move the requested parameters (4 floats) to the CPU, let it churn for a long time, and retrieve the resulting pixels (3 bytes each).
  - rm999 14y ago
    I think EP describes a problem, whereas how much data you have to move around is dependent on your implementation. For example, SETI@home is an EP problem that could be run on a multicore machine completely in parallel without moving data. The fact that it is run on many computers over the internet doesn't change that it's EP. In fact, it is exactly because it is EP that it can be run @home.
    - marshray 14y ago
      I agree SETI@Home is very EP. What did I say to suggest it was not?
      - rm999 14y ago
        I'm not arguing that it's EP, I'm using it as an counterexample to your point. You can solve SETI by moving data around or without moving data around, but that doesn't change the inherent fact that it is a EP algorithm.
        marshray 14y ago
        But do we agree that if SETI required moving so much data around that it was clearly difficult to keep the CPUs fully utilized then it would not be an EP problem?

by jbooth 14y ago

marshray 14y ago

Did anyone (who writes parallel code or purchases CPUs professionally) really not know this? I'm not trying to pick on the article, I like it overall. I took an interest in the Anand article when I learned you could build a 64-core box for about $4000. This is the article it's in response to http://www.anandtech.com/show/5057/the-bulldozer-aftermath-delving-even-deeper http://www.anandtech.com/show/5057/the-bulldozer-aftermath-d... the crux of which is: Some, including sources inside AMD, have blamed Global Foundries for not delivering higher clocked SKUs. Sure, the clock speed targets for Interlagos were probably closer to 3GHz instead of 2.3GHz. But that does not explain why the extra integer cores do not deliver. We were promised up to 50% higher performance thanks to the 33% extra cores, but we got 20% at the most. So the real questions seem to be: Is Interlagos' perfomance disappointing on pure-parallel benchmarks as well, or just those with a noticeable serial bottleneck? If there is poor scalability, what is the cause? and perhaps most importantly: Will my kind of workload actually benefit from Interlagos' more cores? I think that probably what's happening is that even the pure parallel benchmarks are not scaling linearly relative to the previous generation of chips, the reason is a combination of architectural changes in the core itself and memory bus contention, and the target market for these chips is VPS hosts running hundreds of VM images.

jbooth 14y ago
I agree with everything you're saying, I just think the 'obvious point' is one that's papered over in the public discourse a lot because it's not exciting. It is interesting why they don't scale linearly and I agree that it's probably a combination of memory bus contention and/or the chip itself being a little slower. The whole bulldozer thing seemed like a marketing stunt rather than thoughtful pursuit of higher performance to me. Given the frequency and cost of cache miss rates, Intel HT seems to get the same work done on fewer watts.
- marshray 14y ago
  I really don't think AMD would have bet their next generation of chips on something they believed was a marketing stunt. Once the upcoming chip changes get explained to the Marketing Department, however, maybe it kinda starts to sound like Marketing.
- frozenport 14y ago
  Intel HT is often disabled on HPC clusters due to performance problems. Also many people refer to the IL as a 8 core CPU due to the problems mentioned above. Much like not using the HT, many codes disable half the AMD chip.
  - jbooth 14y ago
    Yeah, but it's really only on highly-optimized cache-coherent vector/matrix math that it's a problem. On just about every other piece of software, it works great. I feel like AMD saw the benchmarks available for matrix math and went for it, rather than thinking about what real-world performance profiles looked like.

CookWithMe 14y ago

"the most parallel programs" are those that run on supercomputers / huge clusters, and they don't have these bottlenecks... But for the average parallel programs, you are (currently) right, I guess.

jbooth 14y ago
You've still got a driver program that coordinates the massively parallel stages, most likely, and a less-scalable dispatch process where not all nodes are yet doing work.
- CookWithMe 14y ago
  Yes, there definitely are programs where this holds true (as I have said), but there are others where it does not. A simple, but perfectly reasonable example is brute-force search for cracking a password (if you got the hash of the password). Your dispatch is simple: send every node 1) the hash, 2) the number of nodes, 3) it's own number. With 2) and 3), each node can easily figure out where it starts. You need no more communication until one of the nodes finds the password. See https://en.wikipedia.org/wiki/Embarrassingly_parallel https://en.wikipedia.org/wiki/Embarrassingly_parallel Again, you said "most parallel", and not "average parallel", and this is what I have commented on.
marshray 14y ago
Keep in mind, https://en.wikipedia.org/wiki/Amdahl%27s_law https://en.wikipedia.org/wiki/Amdahl%27s_law was posed at the very beginning of parallel computing. Seymour Cray later became famous both for his fast uniprocessor supercomputer and his quip "If you were plowing a field, which would you rather use: Two strong oxen or 1024 chickens?" For most embarrassingly parallel problems, a Hadoop or BOINC-like cluster made from commodity hardware may always be cheapest. But "real" supercomputers and clusters will spend the most money to reduce bottlenecks precisely because they want to run those programs that are the most limited by them.
- CookWithMe 14y ago
  I know, I actually read the paper like a year ago. Actually, what was turned later into Amdahl's law is more or less a side-note in the paper :) The parent said "most parallel programs", which I translated to embarrassingly parallel problems. > But "real" supercomputers and clusters will spend the most money to reduce bottlenecks precisely because they want to run those programs that are the most limited by them. I agree, but from the talks I have heard most time (= money) is spent trying to work around these bottlenecks as good as possible in software (i.e. incremental algorithms, pipelining, ...) to keep the machines busy. Then again, I hear a lot more talks about software than hardware, so you may be very right. If you have any particular example in mind, I would love to read about it!

rm999 14y ago

> even the most parallel programs tend to have bottlenecks There's a big class of "embarrassingly parallel" tasks, which usually are only limited by how quickly you can move your data to where it needs to be.

marshray 14y ago
What makes something EP is that there is very little data (relative to the CPU time) that needs to be moved. For example, to compute a nice image of points near the Mandelbrot set, you only need to move the requested parameters (4 floats) to the CPU, let it churn for a long time, and retrieve the resulting pixels (3 bytes each).
- rm999 14y ago
  I think EP describes a problem, whereas how much data you have to move around is dependent on your implementation. For example, SETI@home is an EP problem that could be run on a multicore machine completely in parallel without moving data. The fact that it is run on many computers over the internet doesn't change that it's EP. In fact, it is exactly because it is EP that it can be run @home.
  - marshray 14y ago
    I agree SETI@Home is very EP. What did I say to suggest it was not?
    - rm999 14y ago
      I'm not arguing that it's EP, I'm using it as an counterexample to your point. You can solve SETI by moving data around or without moving data around, but that doesn't change the inherent fact that it is a EP algorithm.
      - marshray 14y ago
        But do we agree that if SETI required moving so much data around that it was clearly difficult to keep the CPUs fully utilized then it would not be an EP problem?