6 ms·

4 dies per package is a pretty interesting way of doing things - probably helps yields immensely, but I can't imagine it does anything good for intra-processor

4 dies per package is a pretty interesting way of doing things - probably helps yields immensely, but I can't imagine it does anything good for intra-processor latency. 142 ns to ping a thread on a different CCX within a die isn't too horrible, but I really want to know what sort of penalty you'll have from going to a different die within a package.

dom0 9y ago
MCMs aren't exactly new, especially not for bigger iron, though.
- greglindahl 9y ago
  No, and there is a whole zoo of them shipping now. Xeon Phi has Omni-Path on-package, for example, it's a separate chip.
sliken 9y ago
I think it's less about yield, than it is about amortizing a large R&D budget across more chips. Keep in mind that they are competing with Intel who ships huge volumes and is using this generations profit to fund the next generation fabs. So AMD pours everything into a single die and manages to hit the desktop (Ryzen with 1 die), workstation (thread ripper with 2 dies), and server (Epyc with 4 dies). All with a single wafer of silicon, and as a bonus the memory bandwidth, max memory capacity, and total number of cores scales to fit all 3 markets. Pretty crafty. This is far from new of course, various Power CPUs, Intel cpus as far back as the pentium pro, and of course the previous generation Xeons. Intel's strategy has been one die for most of their low end/low power chips (max 2 core/4T) and a larger die for their desktop (max 4c/8t) that use the same socket (LGA1151). Then Intel targets HEDT (High end desktop) and server with the same chip, same die, same socket, just marketing to differentiate the x-servies chips and the regular single/dual socket chips that share chipsets, sockets, and a lga2011 socket. AMD seems to have scared Intel pretty bad. They have held off on the skylake xeons, only shipping them to cloud providers while they wait for the AMD release before releasing skylake xeons to the masses. I'm glad to say AMD performance seems pretty good, SpecINT (using GCC) is around 50% faster than the similar Intel chip and SpecFP is even better. Seems fair unless you use the Intel compiler to compile all your binaries anyways. In fact the fastest AMD + gcc-6.2 is faster at SpecFP than the fastest Intel + Intel's compiler (1330 vs 1090 respectively).
- phire 9y ago
  Intel also shares dies across their laptop and desktop lines. I'm not sure how many dies there are, but they don't have much more than a 2 core and a 4 core version of each generation of silicon.
  - phonon 9y ago
    Skylake generation servers will have at least three different cores (LCC, HCC, XCC; low core count, high core count, and extreme core count) + a Xeon-D successor (presumably). You can see some info about them here, http://www.anandtech.com/show/11550/the-intel-skylakex-review-core-i9-7900x-i7-7820x-and-i7-7800x-tested/6 http://www.anandtech.com/show/11550/the-intel-skylakex-revie... (Bottom table)
- chx 9y ago
  Well, the Ryzen cores don't have 128 PCIe links, do they?
  - jacquesm 9y ago
    Is there any Intel core with 128 PCIe links then? I'm not aware of any. Iirc Intel is at 40 lanes / socket, which is more like 10 per core rather than 128.
  - snvzz 9y ago
    32 Ryzen (less actually usable due to chipset taking some), 64 Threadripper, 128 EPYC.
  - wyldfire 9y ago
    No, because if they did it would be wasted. Board manufacturers wouldn't lay out all of those lanes to slots because of the increased expense and limited market at that pricepoint that's interested in saturating all that I/O.
- steveh73 9y ago
  I have a skylake xeon from january 2016, and I think they had been around for a few months then?
sliken 9y ago
Well it's one hop over the in package Infinity Fabric which presumably should be less than the cross the package Infinity fabric. From my understanding though chip A in socket 1 is directly connected to chip A in socket 2. So the worst case would be chip A in socket 1 to chip B, C, or D in socket 2. I don't think that number is public yet.
cc439 9y ago
Intel did the same thing with their first Core 2 Quad processors. The Kentsfield generation may have had more limitations to it's multithreaded performance than the single-die Yorkfields but the performance penalty was so negligible that dual-die chips like the Q6600 stayed relevant as long as the next generation of single-die successors.
- kijin 9y ago
  When Intel did the dual-die thing with Kentsfield, AMD mocked them for not being "true quad-core" and refused to follow suit. They had quad-core dies back then, so they could have easily hit back with an 8-core part if they swallowed their pride and put more than one die on a chip! I think it was a terrible move, just when AMD was starting to become irrelevant again. 10 years later, Intel sticks to single-die chips, and AMD has no qualms stuffing a gazillion dies on a chip. Looks like they've learned a lesson or two :)
- velox_io 9y ago
  I forgot the Q6600 was dual-die, probably the best bang for buck CPU I've ever purchased (first popular quad-core & huge overclocking potential). I think AMD reusing the same dies is quite elegant (everything* scales as you add more cores). The inter-die bandwidth/ latency shouldn't really be a problem, because if more than 8 cores are dependant on the same data - locking issues would make any additional cores useless anyway. If they can get NUMA tuned correctly things should work nicely. *it's strange that they all have the same amount of L3 cache. Does anybody have contacts with AMD's marketing/ engineering department who can provide test units?
  - dr_zoidberg 9y ago
    Maybe AMD pulls something akin to the "Iris chip" from Intel and makes L4 standard in a few years. Speaking of which, never understood why Intel wouldn't push a bit more for Iris, when used as L4 it gave close to 20% increase in performance (due to lesser cache mises I believe).
WatchDog 9y ago
AMD's EPYC launch literature points out that the majority of the server market is multi socket. I imagine the intra-processor latency between dies must be better than the latency between sockets. So I think EPYC is a pretty compelling solution for the bulk of the market.
sixothree 9y ago
Didn't IBM do this with Power6?
ChuckMcM 9y ago
I think taking that approach can lead you astray. If you think of it as an "easy" way to build a four socket motherboard using only one actual socket it might make more sense. The "sweet spot" for the old Opteron was 4 CPUs (4 socket). You got to trade off parallelism, memory locality, and a bit of cache thrashing but you could run rings around the Intel offering at the time. Unfortunately it was both expensive and hot to run, this works out much better with regards to total system cost.

dom0 9y ago

MCMs aren't exactly new, especially not for bigger iron, though.

greglindahl 9y ago
No, and there is a whole zoo of them shipping now. Xeon Phi has Omni-Path on-package, for example, it's a separate chip.

sliken 9y ago

I think it's less about yield, than it is about amortizing a large R&D budget across more chips. Keep in mind that they are competing with Intel who ships huge volumes and is using this generations profit to fund the next generation fabs. So AMD pours everything into a single die and manages to hit the desktop (Ryzen with 1 die), workstation (thread ripper with 2 dies), and server (Epyc with 4 dies). All with a single wafer of silicon, and as a bonus the memory bandwidth, max memory capacity, and total number of cores scales to fit all 3 markets. Pretty crafty. This is far from new of course, various Power CPUs, Intel cpus as far back as the pentium pro, and of course the previous generation Xeons. Intel's strategy has been one die for most of their low end/low power chips (max 2 core/4T) and a larger die for their desktop (max 4c/8t) that use the same socket (LGA1151). Then Intel targets HEDT (High end desktop) and server with the same chip, same die, same socket, just marketing to differentiate the x-servies chips and the regular single/dual socket chips that share chipsets, sockets, and a lga2011 socket. AMD seems to have scared Intel pretty bad. They have held off on the skylake xeons, only shipping them to cloud providers while they wait for the AMD release before releasing skylake xeons to the masses. I'm glad to say AMD performance seems pretty good, SpecINT (using GCC) is around 50% faster than the similar Intel chip and SpecFP is even better. Seems fair unless you use the Intel compiler to compile all your binaries anyways. In fact the fastest AMD + gcc-6.2 is faster at SpecFP than the fastest Intel + Intel's compiler (1330 vs 1090 respectively).

phire 9y ago
Intel also shares dies across their laptop and desktop lines. I'm not sure how many dies there are, but they don't have much more than a 2 core and a 4 core version of each generation of silicon.
- phonon 9y ago
  Skylake generation servers will have at least three different cores (LCC, HCC, XCC; low core count, high core count, and extreme core count) + a Xeon-D successor (presumably). You can see some info about them here, http://www.anandtech.com/show/11550/the-intel-skylakex-review-core-i9-7900x-i7-7820x-and-i7-7800x-tested/6 http://www.anandtech.com/show/11550/the-intel-skylakex-revie... (Bottom table)
chx 9y ago
Well, the Ryzen cores don't have 128 PCIe links, do they?
- jacquesm 9y ago
  Is there any Intel core with 128 PCIe links then? I'm not aware of any. Iirc Intel is at 40 lanes / socket, which is more like 10 per core rather than 128.
- snvzz 9y ago
  32 Ryzen (less actually usable due to chipset taking some), 64 Threadripper, 128 EPYC.
- wyldfire 9y ago
  No, because if they did it would be wasted. Board manufacturers wouldn't lay out all of those lanes to slots because of the increased expense and limited market at that pricepoint that's interested in saturating all that I/O.
steveh73 9y ago
I have a skylake xeon from january 2016, and I think they had been around for a few months then?

sliken 9y ago

Well it's one hop over the in package Infinity Fabric which presumably should be less than the cross the package Infinity fabric. From my understanding though chip A in socket 1 is directly connected to chip A in socket 2. So the worst case would be chip A in socket 1 to chip B, C, or D in socket 2. I don't think that number is public yet.

cc439 9y ago

Intel did the same thing with their first Core 2 Quad processors. The Kentsfield generation may have had more limitations to it's multithreaded performance than the single-die Yorkfields but the performance penalty was so negligible that dual-die chips like the Q6600 stayed relevant as long as the next generation of single-die successors.

kijin 9y ago
When Intel did the dual-die thing with Kentsfield, AMD mocked them for not being "true quad-core" and refused to follow suit. They had quad-core dies back then, so they could have easily hit back with an 8-core part if they swallowed their pride and put more than one die on a chip! I think it was a terrible move, just when AMD was starting to become irrelevant again. 10 years later, Intel sticks to single-die chips, and AMD has no qualms stuffing a gazillion dies on a chip. Looks like they've learned a lesson or two :)
velox_io 9y ago
I forgot the Q6600 was dual-die, probably the best bang for buck CPU I've ever purchased (first popular quad-core & huge overclocking potential). I think AMD reusing the same dies is quite elegant (everything* scales as you add more cores). The inter-die bandwidth/ latency shouldn't really be a problem, because if more than 8 cores are dependant on the same data - locking issues would make any additional cores useless anyway. If they can get NUMA tuned correctly things should work nicely. *it's strange that they all have the same amount of L3 cache. Does anybody have contacts with AMD's marketing/ engineering department who can provide test units?
- dr_zoidberg 9y ago
  Maybe AMD pulls something akin to the "Iris chip" from Intel and makes L4 standard in a few years. Speaking of which, never understood why Intel wouldn't push a bit more for Iris, when used as L4 it gave close to 20% increase in performance (due to lesser cache mises I believe).

WatchDog 9y ago

AMD's EPYC launch literature points out that the majority of the server market is multi socket. I imagine the intra-processor latency between dies must be better than the latency between sockets. So I think EPYC is a pretty compelling solution for the bulk of the market.

sixothree 9y ago

Didn't IBM do this with Power6?

ChuckMcM 9y ago

I think taking that approach can lead you astray. If you think of it as an "easy" way to build a four socket motherboard using only one actual socket it might make more sense. The "sweet spot" for the old Opteron was 4 CPUs (4 socket). You got to trade off parallelism, memory locality, and a bit of cache thrashing but you could run rings around the Intel offering at the time. Unfortunately it was both expensive and hot to run, this works out much better with regards to total system cost.