4 ms·

Thanks for that! I figured I remembered at least something from undergrad. Now I'm also curious, if the 400,000 cores are arranged in this 12x7 grid configura

Thanks for that! I figured I remembered at least something from undergrad. Now I'm also curious, if the 400,000 cores are arranged in this 12x7 grid configuration (as seen in the article photo), does that have any relation to the sequence / ordering of the simulation cells run on the chip? For example, if you have an x & y simulation space, do the corresponding ranges of the GPUs on the chip physically tend to map to the coordinates of the physical simulation cells (logic being that if you want simulation cells nearby each other to access results of neighboring cells efficiently, put them on neighboring GPUs)? I guess that then leads to the question of how you handle the 3rd dimension, but I suppose that's kind of just another slice in the other axis when you have to iterate over that.

copy-pastries 6y ago
These are great questions :-) If I remember correctly from the paper, the authors used quite small grids, but they were actually 3D, like your question. They gave each "slice" (one full dimension of the tensorx1x1) to a different core. Cores had to get border data from 8 surrounding memories, which would be a crazy ratio on most architectures, but works because memory accesses are 1 instruction cycle on the CS-1, or 1 cycle per hop if the local memory isn't right by that core. I think that the interconnects on the CS-1 were also toruses (wrapped around at the edges of the grid), which gives you some flexibility. In general, the decomposition of tiles (or blocks in 3d) to cores can follow lots of different strategies depending on what is best for the architecture and whether distributed cores have an all-to-all interconnect, or some sort of torus, or whether the block size can be made to suit some aspect of the processor (e.g. vector length), and the they is to balance the load equally over different cores. Depending on the problem, it might require more advanced graph partitioning algorithms than simple spatial decomposition.
m_mueller 6y ago
I haven't read the paper on Cerebras, but I can tell you a bit how it's done on GPU. For optimization reasons, the mapping on GPU cores tend to resemble the problem space. Typically you'd choose one thread for each x/y column or for each x/y/z cell, depending on how the data dependencies look like for a particular algorithm. These threads are then arranged on a grid. That grid should be at least one order of magnitude larger than the number of cores on your GPU. With that over-allocation you give enough to do for the on-chip scheduler in order to hide memory latency. Thus, unlike CPUs, GPUs become faster when using 'hyper-threading' thanks to the additional information about the problem space the scheduler can work with. Nowadays you can even have a unified grid for multi-GPU systems, though for larger clusters this is still a bit in its infancy (AFAIK, been out of the game for a couple of years) and engineers still tend to opt for MPI for internode communication setup. All in all, the way decomposition works onto such grids is actually one of the big strengths of GPU programming compared to CPU, because there it becomes complex very quickly with multi-threading as well as vectorization optimizations making a mess out of your code. GPU programming is harder than using a CPU naively, but at least for data parallel algorithms it's IMO much easier than using a CPU even close to optimally.

copy-pastries 6y ago

These are great questions :-) If I remember correctly from the paper, the authors used quite small grids, but they were actually 3D, like your question. They gave each "slice" (one full dimension of the tensorx1x1) to a different core. Cores had to get border data from 8 surrounding memories, which would be a crazy ratio on most architectures, but works because memory accesses are 1 instruction cycle on the CS-1, or 1 cycle per hop if the local memory isn't right by that core. I think that the interconnects on the CS-1 were also toruses (wrapped around at the edges of the grid), which gives you some flexibility. In general, the decomposition of tiles (or blocks in 3d) to cores can follow lots of different strategies depending on what is best for the architecture and whether distributed cores have an all-to-all interconnect, or some sort of torus, or whether the block size can be made to suit some aspect of the processor (e.g. vector length), and the they is to balance the load equally over different cores. Depending on the problem, it might require more advanced graph partitioning algorithms than simple spatial decomposition.

m_mueller 6y ago

I haven't read the paper on Cerebras, but I can tell you a bit how it's done on GPU. For optimization reasons, the mapping on GPU cores tend to resemble the problem space. Typically you'd choose one thread for each x/y column or for each x/y/z cell, depending on how the data dependencies look like for a particular algorithm. These threads are then arranged on a grid. That grid should be at least one order of magnitude larger than the number of cores on your GPU. With that over-allocation you give enough to do for the on-chip scheduler in order to hide memory latency. Thus, unlike CPUs, GPUs become faster when using 'hyper-threading' thanks to the additional information about the problem space the scheduler can work with. Nowadays you can even have a unified grid for multi-GPU systems, though for larger clusters this is still a bit in its infancy (AFAIK, been out of the game for a couple of years) and engineers still tend to opt for MPI for internode communication setup. All in all, the way decomposition works onto such grids is actually one of the big strengths of GPU programming compared to CPU, because there it becomes complex very quickly with multi-threading as well as vectorization optimizations making a mess out of your code. GPU programming is harder than using a CPU naively, but at least for data parallel algorithms it's IMO much easier than using a CPU even close to optimally.