4 ms·

Neat, but wow look at the bus on that thing! Is Intel still using a basic crossbar for an interconnect? When are they going to roll out their new interconnect

Neat, but wow look at the bus on that thing! Is Intel still using a basic crossbar for an interconnect? When are they going to roll out their new interconnect technologies? (They clearly must be working on them... they know just as well as the rest of us do that the on-chip network they have now isn't going to scale.) These appear to be still built on the old 45nm process. It should be interesting to see what the 32nm westmere shrink look like. (The ones out so far are mostly only dual cores. They plan to do the rest of the line later in the year...)

wmf 17y ago
Actually, this is the first Intel chip with a ring bus.
- djcapelis 17y ago
  Ah, interesting. I assume there's multiple rings though, one can't really be satisfied with a 1x-4x difference in contacting other cores? (Assuming bidirectional.) Also rings still don't scale, so they're really going to have to get more clever. I wonder what the perf hit for the ring is going to be.
  - Andys 17y ago
    http://www.theregister.co.uk/2009/11/16/sgi_altix_uv_preview/ http://www.theregister.co.uk/2009/11/16/sgi_altix_uv_preview...
    - djcapelis 17y ago
      That's an off-chip interconnect, we're talking about on-chip interconnects. 8 cores is probably pushing the limits of Intel's current on-chip network. One option is to move closer (but probably not actually adopt) something like a scalar operand network used on some of the tile architectures.
      - sparky 17y ago
        It depends on the data sharing and memory access patterns you want to support. For instance, the 16-32-core Larrabee has a ring bus. So do GPUs with many, many cores (http://en.wikipedia.org/wiki/Radeon_R600#Memory_controllers http://en.wikipedia.org/wiki/Radeon_R600#Memory_controllers). What rings give up in terms of nonuniform point-to-point latency and diversity, they partially make up for in physical design simplicity. They are easily routed and verified, giving the circuit guys more time to go nuts on it and crank up the frequency more than you might be able to on a mesh, torus, or other more complex topology ( http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.22.1909&rep=rep1&type=pdf http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.22.... ). You're quite right that they're not infinitely scalable though; Intel came out with two chips at ISSCC using 2D meshes ( http://hothardware.com/News/Intel-Unveils-48Core-SingleChip-Cloud-Computer/ http://hothardware.com/News/Intel-Unveils-48Core-SingleChip-... http://www.theregister.co.uk/2010/01/28/isscc_chip_preview/ http://www.theregister.co.uk/2010/01/28/isscc_chip_preview/ ) for the 48-64-core regime. They also used a mesh for their Polaris test chip a couple years ago ( http://www.eetimes.com/news/latest/showArticle.jhtml?articleID=197004697 http://www.eetimes.com/news/latest/showArticle.jhtml?article... ). Beyond that, there is a ton of published work (network-on-chip, or NoC, is the keyword will help you find it) but not too many chips to prove or disprove their ideas. For what it's worth, most of the many-core chips that do exist keep it fairly simple. Ambric's MPPA had a mesh with configurable routing (http://www.nethra.us.com/technologies_mppa.php http://www.nethra.us.com/technologies_mppa.php), GPUs have a combination of pretty straightforward hierarchical interconnect and rings, and Azul's interconnect is a tree-like structure as well. One thing to consider is that, while more complex topologies can buy you something in a supercomputer, give you more bisection bandwdith, and better support point to point communication, it has turned out to be pretty hard to actually write correct programs in that style (lots of arbitrary peer-to-peer communication); this partially motivates simpler interconnects, where the programmer can more easily reason about what's going on. Another concern is that topologies that work great in a server room (3D) are nigh-unroutable on a chip (2D). The wraparound links in a torus are a great example of this; Blue Gene/L used a torus to great effect (http://www.google.com/url?sa=t&source=web&ct=res&cd=3&ved=0CBUQFjAC&url=https%3A%2F%2Fasc.llnl.gov%2Fcomputing_resources%2Fbluegenel%2Ftalks%2Fheidelberger.pdf&ei=p22WS9ysApTSNbyhqKgP&usg=AFQjCNGH-zFZu37JKAIQCY4KBdYrA-xabA&sig2=at2kAwIukcbbHFC2B3MjPA http://www.google.com/url?sa=t&source=web&ct=res&#38...), and the wraparound links drastically reduce worst-case and average point to point latency over a mesh, but those links mean giving up a large fraction of a metal layer on a chip, as opposed to a long cable in a machine room. Also, I would hazard a guess that the workloads these chips are intended for are mostly multiprogrammed, not multithreaded; if they don't share much data, the network's bisection bandwidth is not as much of an issue as the off-chip bandwidth. Regarding scalar operand networks as found in Raw/Tilera64: The specific idea (register-mapped networks) probably changes the programming model too much for Intel to adopt directly, but the Single-chip Cloud Computer (SCC) linked above, presented at ISSCC 2010, uses something sort of similar for message passing over their 2D mesh. The communication channels are memory-mapped rather than register-mapped, but the mechanism is similar.
duskwuff 17y ago
Some of the stuff in the middle is bound to be the 24MB cache.
- djcapelis 17y ago
  A lot of the stuff in the middle is the cache. But a lot is also interconnect, it's pretty easy to tell the cache, it looks like blocky things. :) There's a lot of interconnect on that chip. Which is not surprising if you're using the same type of on-chip network they've been doing. Clearly they're blocking some things together so it's probably not all to all, but it's not exactly the types of on-chip networks you can expect to scale a whole lot further than where they're at.

wmf 17y ago

Actually, this is the first Intel chip with a ring bus.

djcapelis 17y ago
Ah, interesting. I assume there's multiple rings though, one can't really be satisfied with a 1x-4x difference in contacting other cores? (Assuming bidirectional.) Also rings still don't scale, so they're really going to have to get more clever. I wonder what the perf hit for the ring is going to be.
- Andys 17y ago
  http://www.theregister.co.uk/2009/11/16/sgi_altix_uv_preview/ http://www.theregister.co.uk/2009/11/16/sgi_altix_uv_preview...
  - djcapelis 17y ago
    That's an off-chip interconnect, we're talking about on-chip interconnects. 8 cores is probably pushing the limits of Intel's current on-chip network. One option is to move closer (but probably not actually adopt) something like a scalar operand network used on some of the tile architectures.
    - sparky 17y ago
      It depends on the data sharing and memory access patterns you want to support. For instance, the 16-32-core Larrabee has a ring bus. So do GPUs with many, many cores (http://en.wikipedia.org/wiki/Radeon_R600#Memory_controllers http://en.wikipedia.org/wiki/Radeon_R600#Memory_controllers). What rings give up in terms of nonuniform point-to-point latency and diversity, they partially make up for in physical design simplicity. They are easily routed and verified, giving the circuit guys more time to go nuts on it and crank up the frequency more than you might be able to on a mesh, torus, or other more complex topology ( http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.22.1909&rep=rep1&type=pdf http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.22.... ). You're quite right that they're not infinitely scalable though; Intel came out with two chips at ISSCC using 2D meshes ( http://hothardware.com/News/Intel-Unveils-48Core-SingleChip-Cloud-Computer/ http://hothardware.com/News/Intel-Unveils-48Core-SingleChip-... http://www.theregister.co.uk/2010/01/28/isscc_chip_preview/ http://www.theregister.co.uk/2010/01/28/isscc_chip_preview/ ) for the 48-64-core regime. They also used a mesh for their Polaris test chip a couple years ago ( http://www.eetimes.com/news/latest/showArticle.jhtml?articleID=197004697 http://www.eetimes.com/news/latest/showArticle.jhtml?article... ). Beyond that, there is a ton of published work (network-on-chip, or NoC, is the keyword will help you find it) but not too many chips to prove or disprove their ideas. For what it's worth, most of the many-core chips that do exist keep it fairly simple. Ambric's MPPA had a mesh with configurable routing (http://www.nethra.us.com/technologies_mppa.php http://www.nethra.us.com/technologies_mppa.php), GPUs have a combination of pretty straightforward hierarchical interconnect and rings, and Azul's interconnect is a tree-like structure as well. One thing to consider is that, while more complex topologies can buy you something in a supercomputer, give you more bisection bandwdith, and better support point to point communication, it has turned out to be pretty hard to actually write correct programs in that style (lots of arbitrary peer-to-peer communication); this partially motivates simpler interconnects, where the programmer can more easily reason about what's going on. Another concern is that topologies that work great in a server room (3D) are nigh-unroutable on a chip (2D). The wraparound links in a torus are a great example of this; Blue Gene/L used a torus to great effect (http://www.google.com/url?sa=t&source=web&ct=res&cd=3&ved=0CBUQFjAC&url=https%3A%2F%2Fasc.llnl.gov%2Fcomputing_resources%2Fbluegenel%2Ftalks%2Fheidelberger.pdf&ei=p22WS9ysApTSNbyhqKgP&usg=AFQjCNGH-zFZu37JKAIQCY4KBdYrA-xabA&sig2=at2kAwIukcbbHFC2B3MjPA http://www.google.com/url?sa=t&source=web&ct=res&#38...), and the wraparound links drastically reduce worst-case and average point to point latency over a mesh, but those links mean giving up a large fraction of a metal layer on a chip, as opposed to a long cable in a machine room. Also, I would hazard a guess that the workloads these chips are intended for are mostly multiprogrammed, not multithreaded; if they don't share much data, the network's bisection bandwidth is not as much of an issue as the off-chip bandwidth. Regarding scalar operand networks as found in Raw/Tilera64: The specific idea (register-mapped networks) probably changes the programming model too much for Intel to adopt directly, but the Single-chip Cloud Computer (SCC) linked above, presented at ISSCC 2010, uses something sort of similar for message passing over their 2D mesh. The communication channels are memory-mapped rather than register-mapped, but the mechanism is similar.

duskwuff 17y ago

Some of the stuff in the middle is bound to be the 24MB cache.

djcapelis 17y ago
A lot of the stuff in the middle is the cache. But a lot is also interconnect, it's pretty easy to tell the cache, it looks like blocky things. :) There's a lot of interconnect on that chip. Which is not surprising if you're using the same type of on-chip network they've been doing. Clearly they're blocking some things together so it's probably not all to all, but it's not exactly the types of on-chip networks you can expect to scale a whole lot further than where they're at.