8 ms·

A look beyond x86: OpenPOWER and AArch64

48 points by fcambus 12y ago

AnonNo15 12y ago
- Intel Xeon provides the best performance. - Power consumption of Intel systems at off, idle and full power states are less then of ARM64 and POWER8 ones - ARM 64-bit and Power 8 are currently providing a significantly lower performance / Watt than Intel Damn.
- TheLoneWolfling 12y ago
  Best performance for non memory-bandwidth-limited problems, yes. For memory-bandwidth-limited problems (such as you often see with HPC), power8 looks better.
- zurn 12y ago
  Note this is only about the specific configurations they tested and not necessarily about hte architectures in general (eg POWER8 is represented by 3 GHz chips here, they go up to 4+ GHz).
- userbinator 12y ago
  "at off, idle and full power states are less" Xeons consume less power even when off...? Am I not parsing this sentence correctly?
  - AnonNo15 12y ago
    Chassis power consumption with system(s) off: Xeon: 0W OpenPOWER: 11W HP Moonshot (Avoton): 121W HP Moonshot (ARM): 218W This is mostly due to chassis specifics, though.
    - userbinator 12y ago
      That is so amazingly high that I think something isn't right - I wonder what's consuming so much power. 100W+ is like a whole CPU at full load. Then again, servers are not usually turned off so perhaps they just kept all the CPUs powered up but in a halted state...
      - Sanddancer 12y ago
        Most likely, it's everything else on the motherboard. Think things like the IPMI chipset, which tends to be on all the time. If the boards have SFP+ transceivers, those may very well use a few watts waiting for Wake on Lan packets. Certain SCSI controllers will have their own Out of Band management chips that use a few watts. Then you have a fan spinning slowly to keep the power supply cool for handling all those parts at idle. Servers tend to have lots of bits and pieces that like being on all the time, so even when shut down, the motherboard still tends to be a hive of activity.
      - timthorn 12y ago
        Bear in mind that the Moonshot chassis also contains one or two, potentially 10GbE, switches.
        lvalsan 12y ago
        The tested Intel Xeon E3-1285L v3 (Haswell) and the IBM Power 8 systems are single mainboard systems. The Intel C2750 (Avoton) and the ARM 64-bit Applied Micro X-Gene 1 are cartridges inside an HP Moonshot chassis. All tested systems are fitted with a BMC. In addition the HP Moonshot chassis also have an internal network switch, uplink module, internal network fabric, chassis fans, chassis manager module, etc. The Intel Avoton chassis is having a 1 GbE internal network fabric and switch with a 6x SFP+ 10 GbE uplink module (one port used) while the ARM 64-bit X-Gene 1 chassis is having a 10 GbE internal network fabric and switch with a 4x QSFP+ uplink module (one port used). The purpose of the off power measurements is to see how much power the chassis infrastructure is drawing. The power consumption of the Intel Xeon E3-1285L v3 (Haswell) chassis with the system off is very low, below the threshold of the Racktivity smart metered PDU used to perform the power measurements. The PDU reports 0, but the real power consumption is in the order of a few Watts. The tested Power 8 processor is inside a Tyan reference system GN70-BP010. Being a reference system, not all components and firmwares have been fully optimised. In the mean time Tyan has launched the TN71-BP012 platform which is not a reference system and should be much more optimised and ready for production use. I would be curious to see what are the improvements, but I don't have any at hand for now.
    - deleted 12y ago
      [deleted]
- ksec 12y ago
  I think Power 8 and the ARM are still on older Gen node. Xeon is on 22nm, with the latest Xeon on 14nm Broadwell. So Intel wins on Fabs Tech. And this will not change for another 5 years.
  - exDM69 12y ago
    Semiconductor fabs competing with Intel are a bit behind but that's not the issue here, they were using a 40nm ARM chip but there are 28nm and 20nm fabs available.
  - lvalsan 12y ago
    The Intel chips (Xeon E3-1285L v3 - Haswell and Atom C2750 - Avoton) are both built on 22 nm. The Power 8 processor is using the same process size as the Intel parts, 22 nm. The ARM 64-bit Applied Micro X-Gene 1 SoC tested is on 40 nm. X-Gene 2 is on 28 nm and X-Gene 3 will be build using a 16 nm process. X-Gene 2 is apparently sampling now but we didn't get access to it yet, nor have we seen any publicly available platform using it. Once we'll get access (which should happen soon) to Xeon-D (Broadwell-DE) using a 14 nm process size, the results will be added to the repository at http://lvalsan.web.cern.ch/lvalsan/processor_benchmarking/ http://lvalsan.web.cern.ch/lvalsan/processor_benchmarking/
- exDM69 12y ago
  Yes, I was disappointed too but... The AArch64 chip used was a 40nm chip. The Intel chips it was up against are 20nm (or as low as 14nm). This has enormous power implications, and given that they're pretty close in power consumption with a huge difference in the fab process, Aarch64 is a strong contender as an architecture.
kev009 12y ago
Little bit disappointed by the POWER8 data. I guess it would start paying bigger dividends in scale up systems, but the power and thermals are a little shocking - I think there may need to be some tuning and AFAICT Linux may not be running on bare metal. Same for ARM, but I guess the sweet spot there is scale out. Intel has been putting out amazing chips since Haswell. We need the others to do well to keep them honest and innovative.
- m_mueller 12y ago
  The one big advantage of power architecture is memory bandwidth. Power8 has 230GB/s sustained on a 12 core chip while Xeon Haswell-EP has 102GB/s on 12 core. This means you need more parallelism (=harder to program) and more sockets to get the same speed on Xeons vs. Power8 for memory-bandwidth-limited problems (which >50% of HPC applications are). Power8 has I believe ~50% higher TDP than Xeons, so even in performance per watt for memory bandwidth limited problems, you are still ahead, especially considering all the components you need to power besides the socket. TLDR; These things are for a specific segment of HPC, albeit the one that doesn't care too much about Linpack performance. [1] http://en.wikipedia.org/wiki/POWER8 http://en.wikipedia.org/wiki/POWER8 [2] http://www.pugetsystems.com/blog/2014/09/08/Memory-Performance-for-Intel-Xeon-Haswell-EP-DDR4-596/ http://www.pugetsystems.com/blog/2014/09/08/Memory-Performan...
  - kev009 12y ago
    No doubt HPC is a big influence on design since they can sink big gov contracts, but IBM is claiming it's a serious competitor to scale out x86. They've done this in the past.. PowerPC (6xx), PReP, CHRP, PowerLinux, now OpenPOWER. It looks too similar this time where they cannot get things into gear lowering margins to gain volume. The Tyan box is basically a reference platform.. over priced and poorly designed (2U, 3.5" disks, several thousand dollars?!). Crossing my fingers something good will happen, but placing little stock in it.
  - StillBored 12y ago
    You should be careful on the memory bandwidth. The more recent xeons (2011v3, ddr4 2133) are rated 68 GB/s per socket. That is 136GB/s in a dual socket. But, what I really came here to say, is that POWER8 has nearly twice the single thread memory bandwidth available. From some of my tests a single thread can access ~40GB/sec of bandwidth vs the intel at ~20GB/sec. But this doesn't translate into clear cut wins even on codes which at first pass appear to be bandwidth limited. Mostly because the processor cores themselves are about 1/2 the speed in tight loopy code (think nbench) running out of cache. So, in our in house benchmarks the POWER ran anywhere from 75% as fast to roughly as fast. Only on a few tests did it actually manage to best the E5 xeon we were testing it against, and then never by more than 150%. Plus, the dynamic thread controls are cool, but beyond 4 threads/core I could never actually improve performance, and beyond 2x it was major diminishing returns.
ajdlinux 12y ago
Will shortly be starting at a new company where I get to do things with OpenPOWER. I've never even touched a POWER architecture system apart from the occasional PowerPC Mac, so this will be an interesting learning experience for me.
- e12e 12y ago
  That sounds like fun! Any more details? Why? How? What?
  - ajdlinux 12y ago
    I won't go into too much detail as it's still a while before I start - systems programming at a firm that is quite heavily invested in POWER, and does a fair chunk of work out of its Australian offices, of all places.
    - e12e 12y ago
      Ah, I see. Thanks for the update -- I must have missed the at-part. Thought you were "starting a company", not starting at a company :-) It'd be interesting to hear which company it is -- I don't really know who uses POWER these days... email in profile in case you're more comfortable sharing that privately. I of course understand if you feel you can't do that.
akuma73 12y ago
X-Gene 1 is in ancient 40nm technology. Haswell is 2 generations ahead in 22nm Finfet technology. The gap should close as the process technologies get closer.
- tw04 12y ago
  How do you see the process technologies getting closer? Intel has an insane R&D budget that they spend on their process technology. Nobody has managed to close the gap yet, what makes you think that's going to change?
  - Sanddancer 12y ago
    Samsung is making 14nm chips in volume and has licensed that tech to Globalfoundries, and has demoed 10nm. TSMC's got 16nm, and is planning on starting low scale production of 10nm later this year. So yeah, process tech is pretty similar between the various big chip makers.
WildUtah 12y ago
Page is unreadable on Mac Chrome. Content comes up and then turns into a plain title page with no additional content. Page does not scroll or respond to clicks anywhere.
- Sanddancer 12y ago
  They use a (imo silly) scrollless, etc setup to move through frames. Hit up and down to see the other pages.
- lvalsan 12y ago
  It may not be immediately obvious how to navigate the presentation. I've added a hint on the title slide that appears after three seconds of inactivity, with some details on how to move forward.

48 points by fcambus 12y ago

- Intel Xeon provides the best performance. - Power consumption of Intel systems at off, idle and full power states are less then of ARM64 and POWER8 ones - ARM 64-bit and Power 8 are currently providing a significantly lower performance / Watt than Intel Damn.

TheLoneWolfling 12y ago
Best performance for non memory-bandwidth-limited problems, yes. For memory-bandwidth-limited problems (such as you often see with HPC), power8 looks better.
zurn 12y ago
Note this is only about the specific configurations they tested and not necessarily about hte architectures in general (eg POWER8 is represented by 3 GHz chips here, they go up to 4+ GHz).
userbinator 12y ago
"at off, idle and full power states are less" Xeons consume less power even when off...? Am I not parsing this sentence correctly?
- AnonNo15 12y ago
  Chassis power consumption with system(s) off: Xeon: 0W OpenPOWER: 11W HP Moonshot (Avoton): 121W HP Moonshot (ARM): 218W This is mostly due to chassis specifics, though.
  - userbinator 12y ago
    That is so amazingly high that I think something isn't right - I wonder what's consuming so much power. 100W+ is like a whole CPU at full load. Then again, servers are not usually turned off so perhaps they just kept all the CPUs powered up but in a halted state...
    - Sanddancer 12y ago
      Most likely, it's everything else on the motherboard. Think things like the IPMI chipset, which tends to be on all the time. If the boards have SFP+ transceivers, those may very well use a few watts waiting for Wake on Lan packets. Certain SCSI controllers will have their own Out of Band management chips that use a few watts. Then you have a fan spinning slowly to keep the power supply cool for handling all those parts at idle. Servers tend to have lots of bits and pieces that like being on all the time, so even when shut down, the motherboard still tends to be a hive of activity.
    - timthorn 12y ago
      Bear in mind that the Moonshot chassis also contains one or two, potentially 10GbE, switches.
      - lvalsan 12y ago
        The tested Intel Xeon E3-1285L v3 (Haswell) and the IBM Power 8 systems are single mainboard systems. The Intel C2750 (Avoton) and the ARM 64-bit Applied Micro X-Gene 1 are cartridges inside an HP Moonshot chassis. All tested systems are fitted with a BMC. In addition the HP Moonshot chassis also have an internal network switch, uplink module, internal network fabric, chassis fans, chassis manager module, etc. The Intel Avoton chassis is having a 1 GbE internal network fabric and switch with a 6x SFP+ 10 GbE uplink module (one port used) while the ARM 64-bit X-Gene 1 chassis is having a 10 GbE internal network fabric and switch with a 4x QSFP+ uplink module (one port used). The purpose of the off power measurements is to see how much power the chassis infrastructure is drawing. The power consumption of the Intel Xeon E3-1285L v3 (Haswell) chassis with the system off is very low, below the threshold of the Racktivity smart metered PDU used to perform the power measurements. The PDU reports 0, but the real power consumption is in the order of a few Watts. The tested Power 8 processor is inside a Tyan reference system GN70-BP010. Being a reference system, not all components and firmwares have been fully optimised. In the mean time Tyan has launched the TN71-BP012 platform which is not a reference system and should be much more optimised and ready for production use. I would be curious to see what are the improvements, but I don't have any at hand for now.
  - deleted 12y ago
    [deleted]
ksec 12y ago
I think Power 8 and the ARM are still on older Gen node. Xeon is on 22nm, with the latest Xeon on 14nm Broadwell. So Intel wins on Fabs Tech. And this will not change for another 5 years.
- exDM69 12y ago
  Semiconductor fabs competing with Intel are a bit behind but that's not the issue here, they were using a 40nm ARM chip but there are 28nm and 20nm fabs available.
- lvalsan 12y ago
  The Intel chips (Xeon E3-1285L v3 - Haswell and Atom C2750 - Avoton) are both built on 22 nm. The Power 8 processor is using the same process size as the Intel parts, 22 nm. The ARM 64-bit Applied Micro X-Gene 1 SoC tested is on 40 nm. X-Gene 2 is on 28 nm and X-Gene 3 will be build using a 16 nm process. X-Gene 2 is apparently sampling now but we didn't get access to it yet, nor have we seen any publicly available platform using it. Once we'll get access (which should happen soon) to Xeon-D (Broadwell-DE) using a 14 nm process size, the results will be added to the repository at http://lvalsan.web.cern.ch/lvalsan/processor_benchmarking/ http://lvalsan.web.cern.ch/lvalsan/processor_benchmarking/
exDM69 12y ago
Yes, I was disappointed too but... The AArch64 chip used was a 40nm chip. The Intel chips it was up against are 20nm (or as low as 14nm). This has enormous power implications, and given that they're pretty close in power consumption with a huge difference in the fab process, Aarch64 is a strong contender as an architecture.

kev009 12y ago

Little bit disappointed by the POWER8 data. I guess it would start paying bigger dividends in scale up systems, but the power and thermals are a little shocking - I think there may need to be some tuning and AFAICT Linux may not be running on bare metal. Same for ARM, but I guess the sweet spot there is scale out. Intel has been putting out amazing chips since Haswell. We need the others to do well to keep them honest and innovative.

m_mueller 12y ago
The one big advantage of power architecture is memory bandwidth. Power8 has 230GB/s sustained on a 12 core chip while Xeon Haswell-EP has 102GB/s on 12 core. This means you need more parallelism (=harder to program) and more sockets to get the same speed on Xeons vs. Power8 for memory-bandwidth-limited problems (which >50% of HPC applications are). Power8 has I believe ~50% higher TDP than Xeons, so even in performance per watt for memory bandwidth limited problems, you are still ahead, especially considering all the components you need to power besides the socket. TLDR; These things are for a specific segment of HPC, albeit the one that doesn't care too much about Linpack performance. [1] http://en.wikipedia.org/wiki/POWER8 http://en.wikipedia.org/wiki/POWER8 [2] http://www.pugetsystems.com/blog/2014/09/08/Memory-Performance-for-Intel-Xeon-Haswell-EP-DDR4-596/ http://www.pugetsystems.com/blog/2014/09/08/Memory-Performan...
- kev009 12y ago
  No doubt HPC is a big influence on design since they can sink big gov contracts, but IBM is claiming it's a serious competitor to scale out x86. They've done this in the past.. PowerPC (6xx), PReP, CHRP, PowerLinux, now OpenPOWER. It looks too similar this time where they cannot get things into gear lowering margins to gain volume. The Tyan box is basically a reference platform.. over priced and poorly designed (2U, 3.5" disks, several thousand dollars?!). Crossing my fingers something good will happen, but placing little stock in it.
- StillBored 12y ago
  You should be careful on the memory bandwidth. The more recent xeons (2011v3, ddr4 2133) are rated 68 GB/s per socket. That is 136GB/s in a dual socket. But, what I really came here to say, is that POWER8 has nearly twice the single thread memory bandwidth available. From some of my tests a single thread can access ~40GB/sec of bandwidth vs the intel at ~20GB/sec. But this doesn't translate into clear cut wins even on codes which at first pass appear to be bandwidth limited. Mostly because the processor cores themselves are about 1/2 the speed in tight loopy code (think nbench) running out of cache. So, in our in house benchmarks the POWER ran anywhere from 75% as fast to roughly as fast. Only on a few tests did it actually manage to best the E5 xeon we were testing it against, and then never by more than 150%. Plus, the dynamic thread controls are cool, but beyond 4 threads/core I could never actually improve performance, and beyond 2x it was major diminishing returns.

ajdlinux 12y ago

Will shortly be starting at a new company where I get to do things with OpenPOWER. I've never even touched a POWER architecture system apart from the occasional PowerPC Mac, so this will be an interesting learning experience for me.

e12e 12y ago
That sounds like fun! Any more details? Why? How? What?
- ajdlinux 12y ago
  I won't go into too much detail as it's still a while before I start - systems programming at a firm that is quite heavily invested in POWER, and does a fair chunk of work out of its Australian offices, of all places.
  - e12e 12y ago
    Ah, I see. Thanks for the update -- I must have missed the at-part. Thought you were "starting a company", not starting at a company :-) It'd be interesting to hear which company it is -- I don't really know who uses POWER these days... email in profile in case you're more comfortable sharing that privately. I of course understand if you feel you can't do that.

akuma73 12y ago

X-Gene 1 is in ancient 40nm technology. Haswell is 2 generations ahead in 22nm Finfet technology. The gap should close as the process technologies get closer.

tw04 12y ago
How do you see the process technologies getting closer? Intel has an insane R&D budget that they spend on their process technology. Nobody has managed to close the gap yet, what makes you think that's going to change?
- Sanddancer 12y ago
  Samsung is making 14nm chips in volume and has licensed that tech to Globalfoundries, and has demoed 10nm. TSMC's got 16nm, and is planning on starting low scale production of 10nm later this year. So yeah, process tech is pretty similar between the various big chip makers.

WildUtah 12y ago

Page is unreadable on Mac Chrome. Content comes up and then turns into a plain title page with no additional content. Page does not scroll or respond to clicks anywhere.

Sanddancer 12y ago
They use a (imo silly) scrollless, etc setup to move through frames. Hit up and down to see the other pages.
lvalsan 12y ago
It may not be immediately obvious how to navigate the presentation. I've added a hint on the title slide that appears after three seconds of inactivity, with some details on how to move forward.