5 ms·

I agree, the single, superfast CPU model is outdated. It's a shame that multiprocessor programming is hard (or is it? It might also just be a vicious circle: 'n

by meuk 9y ago

I agree, the single, superfast CPU model is outdated. It's a shame that multiprocessor programming is hard (or is it? It might also just be a vicious circle: 'not many people do MPP because the facilities suck because not many people do MPP').

gaius 9y ago
Look at the way the Amiga did it. The CPU was effectively a peripheral for the system controller known as Agnus. https://en.m.wikipedia.org/wiki/MOS_Technology_Agnus https://en.m.wikipedia.org/wiki/MOS_Technology_Agnus
- walkingolof 9y ago
  Thats what the north/southbridge does, off-loading the CPU from all sorts of tasks (even if some of the tasks moved into the CPU die since the classic definition of north/southbridge was set)
jcranmer 9y ago
I could right now give you a CPU with a 100,000 ALUs on it. Your CPU with ~4 ALUs is probably going to beat it in performance, because you can easily keep all of your ALUs constantly fed with data, but I'm going to struggle to keep all 100,000 ALUs fed. We have some idea of how to do massively-parallel programming. That is, after all, what a supercomputer is. But one of the things we've found is that communication doesn't scale. For very low core counts, you can hook up each core to talk to every other one at the same latency--basically, O(N)-degree topology. For slightly higher core counts, you can keep it a hypercube, which is O(lg N) degree. But when you start thinking about a few hundred cores, you end up with a mesh, which is O(1)-degree. What that means is you struggle to scale problems that have very high communication costs compared to computation. Matrix multiplication is wonderful--you're doing O(N^1.5) computations for each data, so communication goes down as size goes up [1]. But a BFS graph traversal is horrible, because your communication costs go up if average degree is more than O(1). [1] This is why the TOP500 benchmark is to some degree bullshit. It's measuring performance on applications that are fundamentally computation-bound, not communication bound, so it tends to penalize machines that focus on improving communication bandwidth, which is often helpful for many HPC applications.

by meuk 9y ago

gaius 9y ago

Look at the way the Amiga did it. The CPU was effectively a peripheral for the system controller known as Agnus. https://en.m.wikipedia.org/wiki/MOS_Technology_Agnus https://en.m.wikipedia.org/wiki/MOS_Technology_Agnus

walkingolof 9y ago
Thats what the north/southbridge does, off-loading the CPU from all sorts of tasks (even if some of the tasks moved into the CPU die since the classic definition of north/southbridge was set)

jcranmer 9y ago

I could right now give you a CPU with a 100,000 ALUs on it. Your CPU with ~4 ALUs is probably going to beat it in performance, because you can easily keep all of your ALUs constantly fed with data, but I'm going to struggle to keep all 100,000 ALUs fed. We have some idea of how to do massively-parallel programming. That is, after all, what a supercomputer is. But one of the things we've found is that communication doesn't scale. For very low core counts, you can hook up each core to talk to every other one at the same latency--basically, O(N)-degree topology. For slightly higher core counts, you can keep it a hypercube, which is O(lg N) degree. But when you start thinking about a few hundred cores, you end up with a mesh, which is O(1)-degree. What that means is you struggle to scale problems that have very high communication costs compared to computation. Matrix multiplication is wonderful--you're doing O(N^1.5) computations for each data, so communication goes down as size goes up [1]. But a BFS graph traversal is horrible, because your communication costs go up if average degree is more than O(1). [1] This is why the TOP500 benchmark is to some degree bullshit. It's measuring performance on applications that are fundamentally computation-bound, not communication bound, so it tends to penalize machines that focus on improving communication bandwidth, which is often helpful for many HPC applications.