13 ms·

What’s Better Than 40 GPU-Based Computers? A Computer with 40 GPUs

24 points by sonabinu 8y ago

dekhn 8y ago
not really. better to spread those devices over mulitple domains, unless you have a crossbar switchbus that can handle terabytes/sec
tomatotomato37 8y ago
Isn't one of the limiting factors with putting everything on a single silicon wafer is the massive increase in defects the more surface area you use? Otherwise you could build an entire supercomputer on those enormous foot-wide wafers Edit: Nevermind, fixing that is the point of the article; I should stop skimming these things
- MrEldritch 8y ago
  I would recommend R-ing the TFA: >"One solution would be to use standard chip-making techniques to build all 40 GPUs on the same silicon wafer and add interconnects between them. But that’s the philosophy that killed Amdahl’s attempt in the 1980s. There is always the chance of a defect when you’re making a chip, and the likelihood of there being a defect increases with the size of the chip. If your chip is the size of a dinner plate, you’re almost guaranteed to have a system-killing flaw somewhere on it. So it makes more sense to start with normal-sized GPU chips that have already passed quality tests and find a technology to better connect them. The team believes they have that in a technology called silicon interconnect fabric (SiIF). SiIF replaces the circuit board with silicon, so there is no mechanical mismatch between the chip and the board and therefore no need for a chip package."
- dragontamer 8y ago
  It seems like that's the problem they solved. In effect, the wafer is just an interconnect. The GPUs are manufactured normally, but this technology is basically "just" a wafer-scale interconnect. (Kinda like how Vega chips from AMD have a silicon interposer to connect the HBM2 dies with the GPU. Except this interconnect is a whole freaking wafer.) The problem they now have is the heat that is emitted from such a beastly contraption. > Edit: Nevermind, fixing that is the point of the article; I should stop skimming these things The only reason why I'm in a better boat than you is because I had followed a discussion over at Reddit on this very story already. Mistakes when reading articles are pretty common, and don't worry about it. Anyone who gives you much trouble over that is just asking for Karma to smite them. Most of us are here to help each other understand the story.
  - cr0sh 8y ago
    > The GPUs are manufactured normally, but this technology is basically "just" a wafer-scale interconnect. I wonder if this is something like Intel's Foveros 3D chip stacking, where the wafer is treated more like a "PCB" (with the traces and such for signals/power) to the CPU chips bonded on top? Basically, a "PCB" made from silicon, where the components mounted on it are also bare CPU dies?
Traster 8y ago
I think our current trend to more heterogeneous compute units is really going to highlight the value of benchmarks. How many people are really doing TFlops of matrix multiply? I've always been wary of the idea of building extra ordinarily complicated hardware and rely on the software people to figure it out. It doesn't seem to work. It's too many degrees of freedom. Compilers are great at taking complex problems and fitting them to fixed solutions. Anything more is like handing off an NP-hard problem.
- gnulinux 8y ago
  Strong disagree. How is reality even remotely close to what your comment predicts? Only 40 years ago we thought one processor will be enough for all compute tasks, then we learned how to make software for bunch of CPUs, then we realized there are things that benefit from bunch of weaker cores and wrote software for that, then we realized you don't even need a single computer and you can parallelize your program to N computers with M processors. Your argument fails because it assumes everything is automate-able. It's not. Programming is not automate-able, at least not now. If you wait your C compiler to generate code that will run in L computers with N to M threading model you're gonna wait too long. This is what we software engineers are paid for. And it does seem to work, as explained above.
  - bnegreve 8y ago
    > Only 40 years ago we thought one processor will be enough for all compute tasks, then we learned how to make software for bunch of CPUs, then we realized there are things that benefit from bunch of weaker cores and wrote software for that, then we realized you don't even need a single computer and you can parallelize your program to N computers with M processors. No, the history of parallel computing is a lot more chaotic than that! Parallel, distributed or vector computing graphics have been around since the early days and there are many fancy parallel processors architectures that never did well exactly because no ones knew how to program them. The Tera architecture (128 threads) comes to mind[1]. The Cell that was used in the playstation 3. It has 1 general purpose core and 8 accelerators that were vastly underexploited by game developers. The greatly anticipated Intel Larrabee (32 cores) never made it to the market. And even today the fate of the Xeon Phi (about 60 cores) remains very uncertain. In 2019 parallel computing is reserved to the same restricted set of easy to parallelize applications than 40 years ago (numerical simulations, 3D rendering, database servers and maybe few others) + deep learning. But most software is sequential, and most people would not notice if they had 1 or 2 cores instead of 8 or 16. That's because writing parallel programs is inherently and debugging is hell. [1] https://en.wikipedia.org/wiki/Cray_MTA https://en.wikipedia.org/wiki/Cray_MTA

24 points by sonabinu 8y ago

not really. better to spread those devices over mulitple domains, unless you have a crossbar switchbus that can handle terabytes/sec

tomatotomato37 8y ago

Isn't one of the limiting factors with putting everything on a single silicon wafer is the massive increase in defects the more surface area you use? Otherwise you could build an entire supercomputer on those enormous foot-wide wafers Edit: Nevermind, fixing that is the point of the article; I should stop skimming these things

MrEldritch 8y ago
I would recommend R-ing the TFA: >"One solution would be to use standard chip-making techniques to build all 40 GPUs on the same silicon wafer and add interconnects between them. But that’s the philosophy that killed Amdahl’s attempt in the 1980s. There is always the chance of a defect when you’re making a chip, and the likelihood of there being a defect increases with the size of the chip. If your chip is the size of a dinner plate, you’re almost guaranteed to have a system-killing flaw somewhere on it. So it makes more sense to start with normal-sized GPU chips that have already passed quality tests and find a technology to better connect them. The team believes they have that in a technology called silicon interconnect fabric (SiIF). SiIF replaces the circuit board with silicon, so there is no mechanical mismatch between the chip and the board and therefore no need for a chip package."
dragontamer 8y ago
It seems like that's the problem they solved. In effect, the wafer is just an interconnect. The GPUs are manufactured normally, but this technology is basically "just" a wafer-scale interconnect. (Kinda like how Vega chips from AMD have a silicon interposer to connect the HBM2 dies with the GPU. Except this interconnect is a whole freaking wafer.) The problem they now have is the heat that is emitted from such a beastly contraption. > Edit: Nevermind, fixing that is the point of the article; I should stop skimming these things The only reason why I'm in a better boat than you is because I had followed a discussion over at Reddit on this very story already. Mistakes when reading articles are pretty common, and don't worry about it. Anyone who gives you much trouble over that is just asking for Karma to smite them. Most of us are here to help each other understand the story.
- cr0sh 8y ago
  > The GPUs are manufactured normally, but this technology is basically "just" a wafer-scale interconnect. I wonder if this is something like Intel's Foveros 3D chip stacking, where the wafer is treated more like a "PCB" (with the traces and such for signals/power) to the CPU chips bonded on top? Basically, a "PCB" made from silicon, where the components mounted on it are also bare CPU dies?

Traster 8y ago

I think our current trend to more heterogeneous compute units is really going to highlight the value of benchmarks. How many people are really doing TFlops of matrix multiply? I've always been wary of the idea of building extra ordinarily complicated hardware and rely on the software people to figure it out. It doesn't seem to work. It's too many degrees of freedom. Compilers are great at taking complex problems and fitting them to fixed solutions. Anything more is like handing off an NP-hard problem.

gnulinux 8y ago
Strong disagree. How is reality even remotely close to what your comment predicts? Only 40 years ago we thought one processor will be enough for all compute tasks, then we learned how to make software for bunch of CPUs, then we realized there are things that benefit from bunch of weaker cores and wrote software for that, then we realized you don't even need a single computer and you can parallelize your program to N computers with M processors. Your argument fails because it assumes everything is automate-able. It's not. Programming is not automate-able, at least not now. If you wait your C compiler to generate code that will run in L computers with N to M threading model you're gonna wait too long. This is what we software engineers are paid for. And it does seem to work, as explained above.
- bnegreve 8y ago
  > Only 40 years ago we thought one processor will be enough for all compute tasks, then we learned how to make software for bunch of CPUs, then we realized there are things that benefit from bunch of weaker cores and wrote software for that, then we realized you don't even need a single computer and you can parallelize your program to N computers with M processors. No, the history of parallel computing is a lot more chaotic than that! Parallel, distributed or vector computing graphics have been around since the early days and there are many fancy parallel processors architectures that never did well exactly because no ones knew how to program them. The Tera architecture (128 threads) comes to mind[1]. The Cell that was used in the playstation 3. It has 1 general purpose core and 8 accelerators that were vastly underexploited by game developers. The greatly anticipated Intel Larrabee (32 cores) never made it to the market. And even today the fate of the Xeon Phi (about 60 cores) remains very uncertain. In 2019 parallel computing is reserved to the same restricted set of easy to parallelize applications than 40 years ago (numerical simulations, 3D rendering, database servers and maybe few others) + deep learning. But most software is sequential, and most people would not notice if they had 1 or 2 cores instead of 8 or 16. That's because writing parallel programs is inherently and debugging is hell. [1] https://en.wikipedia.org/wiki/Cray_MTA https://en.wikipedia.org/wiki/Cray_MTA