3 ms·

Yes, it's pretty much exactly what you said. Algorithms like those for parallel solvers for systems of equations will divide the problem up into blocks of cell

Yes, it's pretty much exactly what you said. Algorithms like those for parallel solvers for systems of equations will divide the problem up into blocks of cells for parallel processing. But typically each cell's next state calculation needs some values from its close neighbours. That's fast and fine for cells inside a block, but for cells on the borders of blocks, that means communicating (doing "halo exchange") with neighbouring processors on every iteration of a solver. Even within one GPU, that means synchronising between the GPU's "cores" and reading state from the global memory (DRAM), which is orders of magnitude slower and more energy expensive than reading from the on-chip "cache"-style memory. These algorithms also don't do a lot of FLOPs per cell relative to the amount of memory that is accessed, so their performance is completely bound by memory bandwidth (how fast data can get to the processor) rather than compute performance (how fast the processor can operate on it). It's much worse when the problem doesn't fit in a single GPU: then the border tiles have to communicate with remote GPUs and CPUs, which is orders of magnitude slower and more energy intensive again. Then performance can become communication-bound. Because these algorithms are core to so many science applications (like differential equation solvers), there's a huge body of research around making them run better: overlapping communication and computation, swapping fatter halos less frequently, funky spatio-temporal tiling over blocks and loop iterations, etc. The Cerebras chip does amazingly well on these because its 400000 cores can access 18GiB of fast on-chip memory and communicate state over the fifos extremely fast and with very little energy relative to a GPU or multi-CPU solution. The power requirements are a tiny fraction of the GPU equivalent. But many of the problems come back when the problem being solved needs more than the 18GiB of on-chip memory, which sadly is most realistic simulations. Speed will still be limited by how quickly border data can be exchanged.

supernova87a 6y ago
Thanks for that! I figured I remembered at least something from undergrad. Now I'm also curious, if the 400,000 cores are arranged in this 12x7 grid configuration (as seen in the article photo), does that have any relation to the sequence / ordering of the simulation cells run on the chip? For example, if you have an x & y simulation space, do the corresponding ranges of the GPUs on the chip physically tend to map to the coordinates of the physical simulation cells (logic being that if you want simulation cells nearby each other to access results of neighboring cells efficiently, put them on neighboring GPUs)? I guess that then leads to the question of how you handle the 3rd dimension, but I suppose that's kind of just another slice in the other axis when you have to iterate over that.
- copy-pastries 6y ago
  These are great questions :-) If I remember correctly from the paper, the authors used quite small grids, but they were actually 3D, like your question. They gave each "slice" (one full dimension of the tensorx1x1) to a different core. Cores had to get border data from 8 surrounding memories, which would be a crazy ratio on most architectures, but works because memory accesses are 1 instruction cycle on the CS-1, or 1 cycle per hop if the local memory isn't right by that core. I think that the interconnects on the CS-1 were also toruses (wrapped around at the edges of the grid), which gives you some flexibility. In general, the decomposition of tiles (or blocks in 3d) to cores can follow lots of different strategies depending on what is best for the architecture and whether distributed cores have an all-to-all interconnect, or some sort of torus, or whether the block size can be made to suit some aspect of the processor (e.g. vector length), and the they is to balance the load equally over different cores. Depending on the problem, it might require more advanced graph partitioning algorithms than simple spatial decomposition.
- m_mueller 6y ago
  I haven't read the paper on Cerebras, but I can tell you a bit how it's done on GPU. For optimization reasons, the mapping on GPU cores tend to resemble the problem space. Typically you'd choose one thread for each x/y column or for each x/y/z cell, depending on how the data dependencies look like for a particular algorithm. These threads are then arranged on a grid. That grid should be at least one order of magnitude larger than the number of cores on your GPU. With that over-allocation you give enough to do for the on-chip scheduler in order to hide memory latency. Thus, unlike CPUs, GPUs become faster when using 'hyper-threading' thanks to the additional information about the problem space the scheduler can work with. Nowadays you can even have a unified grid for multi-GPU systems, though for larger clusters this is still a bit in its infancy (AFAIK, been out of the game for a couple of years) and engineers still tend to opt for MPI for internode communication setup. All in all, the way decomposition works onto such grids is actually one of the big strengths of GPU programming compared to CPU, because there it becomes complex very quickly with multi-threading as well as vectorization optimizations making a mess out of your code. GPU programming is harder than using a CPU naively, but at least for data parallel algorithms it's IMO much easier than using a CPU even close to optimally.
m_mueller 6y ago
nice summary. are you working in HPC? edit: In case you work for Cerebras, I gotta say, 18GB is indeed low for such a compute dense beast. I'd expect it to have access to at least 200GB off-chip high bandwidth memory.
- copy-pastries 6y ago
  Thanks! Yes, I work as an HPC researcher, but not for Cerebras. I just watched the presentation of this paper at the SC20 Supercomputing conference last week. I probably should have said, according to the specs, a CS-1 does have loads (TBs?) of off-chip DRAM to spill over into if the 18GiB isn't enough. That 18GiB is on-chip SRAM which you could think of as a giant 18GiB cache, which is astounding. Once you spill over that though, you're limited to how fast you can get data to the chip (probably multiple PCIe connections to a host CPU on the system with loads of RAM - I think the details are in the paper), so the results won't be nearly as impressive as this 10000x claim.
  - m_mueller 6y ago
    sounds quite capable actually, for anything that fits to its memory system. HPC is a small-ish community where you can tell quite immediately who's part of it from the way of thinking. I've switched to financial industry in a position where (other engineers think) performance doesn't matter much, but it has always stayed an important factor for the design decisions I make.

supernova87a 6y ago

Thanks for that! I figured I remembered at least something from undergrad. Now I'm also curious, if the 400,000 cores are arranged in this 12x7 grid configuration (as seen in the article photo), does that have any relation to the sequence / ordering of the simulation cells run on the chip? For example, if you have an x & y simulation space, do the corresponding ranges of the GPUs on the chip physically tend to map to the coordinates of the physical simulation cells (logic being that if you want simulation cells nearby each other to access results of neighboring cells efficiently, put them on neighboring GPUs)? I guess that then leads to the question of how you handle the 3rd dimension, but I suppose that's kind of just another slice in the other axis when you have to iterate over that.

copy-pastries 6y ago
These are great questions :-) If I remember correctly from the paper, the authors used quite small grids, but they were actually 3D, like your question. They gave each "slice" (one full dimension of the tensorx1x1) to a different core. Cores had to get border data from 8 surrounding memories, which would be a crazy ratio on most architectures, but works because memory accesses are 1 instruction cycle on the CS-1, or 1 cycle per hop if the local memory isn't right by that core. I think that the interconnects on the CS-1 were also toruses (wrapped around at the edges of the grid), which gives you some flexibility. In general, the decomposition of tiles (or blocks in 3d) to cores can follow lots of different strategies depending on what is best for the architecture and whether distributed cores have an all-to-all interconnect, or some sort of torus, or whether the block size can be made to suit some aspect of the processor (e.g. vector length), and the they is to balance the load equally over different cores. Depending on the problem, it might require more advanced graph partitioning algorithms than simple spatial decomposition.
m_mueller 6y ago
I haven't read the paper on Cerebras, but I can tell you a bit how it's done on GPU. For optimization reasons, the mapping on GPU cores tend to resemble the problem space. Typically you'd choose one thread for each x/y column or for each x/y/z cell, depending on how the data dependencies look like for a particular algorithm. These threads are then arranged on a grid. That grid should be at least one order of magnitude larger than the number of cores on your GPU. With that over-allocation you give enough to do for the on-chip scheduler in order to hide memory latency. Thus, unlike CPUs, GPUs become faster when using 'hyper-threading' thanks to the additional information about the problem space the scheduler can work with. Nowadays you can even have a unified grid for multi-GPU systems, though for larger clusters this is still a bit in its infancy (AFAIK, been out of the game for a couple of years) and engineers still tend to opt for MPI for internode communication setup. All in all, the way decomposition works onto such grids is actually one of the big strengths of GPU programming compared to CPU, because there it becomes complex very quickly with multi-threading as well as vectorization optimizations making a mess out of your code. GPU programming is harder than using a CPU naively, but at least for data parallel algorithms it's IMO much easier than using a CPU even close to optimally.

m_mueller 6y ago

nice summary. are you working in HPC? edit: In case you work for Cerebras, I gotta say, 18GB is indeed low for such a compute dense beast. I'd expect it to have access to at least 200GB off-chip high bandwidth memory.

copy-pastries 6y ago
Thanks! Yes, I work as an HPC researcher, but not for Cerebras. I just watched the presentation of this paper at the SC20 Supercomputing conference last week. I probably should have said, according to the specs, a CS-1 does have loads (TBs?) of off-chip DRAM to spill over into if the 18GiB isn't enough. That 18GiB is on-chip SRAM which you could think of as a giant 18GiB cache, which is astounding. Once you spill over that though, you're limited to how fast you can get data to the chip (probably multiple PCIe connections to a host CPU on the system with loads of RAM - I think the details are in the paper), so the results won't be nearly as impressive as this 10000x claim.
- m_mueller 6y ago
  sounds quite capable actually, for anything that fits to its memory system. HPC is a small-ish community where you can tell quite immediately who's part of it from the way of thinking. I've switched to financial industry in a position where (other engineers think) performance doesn't matter much, but it has always stayed an important factor for the design decisions I make.