14 ms·

Google's POWER8 server motherboard

196 points by self 12y ago

mrweasel 12y ago
Funny layout. I would like to know why the PCI slots a spread out like that. I know Google don't have a standard rack setup, but still, it would make seens to have all the expantion ports the end of the board... No?
- Sanddancer 12y ago
  A layout like this means you can use both sides of the motherboard for I/O slots. So in a 1/2u box, you can get more than one or two expansion cards in place. The PCI slots themselves seem to be hammock connectors, which I was curious about too, and googling doesn't seem to have any info, unless it's too early/late and I'm missing the obvious.
- nuriaion 12y ago
  Maybe these Connectors are for daughterboards with Centaur Chips + RAM. (A Power8 can connect to 8 Centaur Chips where the RAM is connected)
  - justincormack 12y ago
    They must be, as there are otherwise no RAM chips on the board...
- deleted 12y ago
  [deleted]
z3phyr 12y ago
It would be great if somebody could list modern computers for personal use which are still based on Power architecture?
- msiebuhr 12y ago
  Both Xbox360 and PS3 use modified POWER-designs. Edit: Also, it's notable that both Xbox One and PS4 switched to use x64.
  - riffraff 12y ago
    Wii and WiiU also have POWER-based designs IIRC.
    - carey 12y ago
      Both are PowerPC 750-based according to Wikipedia, which is part of why the Wii U runs games released for the Wii. Note that PowerPC, as used in the Wii, Wii U and old Macs, is not exactly the same as POWER, as in this announcement. The POWER architecture is used by IBM AIX and AS/400 servers, and by the PS3 in its Cell variant.
      - DCKing 12y ago
        There is contradictory evidence whether the Cell's PPE was 'PowerPC' or just a 'Power' core. In any case, it could run PowerPC software. Incidentally, three of those exact same cores are used as the CPU of the Xbox 360. For reference, the PowerPC 750 derivatives in the GameCube/Wii/Wii U are in the same family as the PowerPC G3 used in Macs around the turn of the century. It is also related to the CPU running Curiosity on Mars. So yeah, although Nintendo is a big customer of the Power architecture at the moment, they're not really breaking new ground.
        DiabloD3 12y ago
        The Cell is the world's first in order execution PowerPC. Its similar in design to the G3 family but has a very high clock speed. They stripped a lot out of the chip design (such as the out of order execution pipeline, a lot of the cache brains, etc) to get the core as small and as low power as possible while relying on modern compilers to make the magic happen. I'm not entirely sure they succeeded in their goals, but with how well SPEs are used in PS3 games, I'm not sure it matters.
jmnicolas 12y ago
It's a bit short on details imo : where are the specs, the benchmarks etc ?
- jacquesm 12y ago
  It's actually quite impressive that google would open up this much of their secret sauce, a lot can be gleaned from looking at this board. You can bet that this is not exactly revision one (and you can bet as well that this is likely not their latest and greatest, no need to show off more than you have to, competitive edges are pretty thin). When I see stuff like this it is painfully clear that from a technological perspective a company like duck-duck-go has a huge amount of defensible moat to cross before they can begin to be a serious contender. Think about it for a second: the company that you're trying to compete with is operating at such economies of scale that it can afford to have its own custom motherboards + non-standard expansion boards made.
fh973 12y ago
This is significant. With POWER back in the game, and ARM server CPUs arriving, Intel will again have competition.
- ithkuil 12y ago
  I think currently there are few big customers that can afford the overhead of porting their code and dependencies to a different architecture. For example I don't expect cloud providers to have a huge marked soon for non x86 architectures. Well there are JVM or other VM users which in theory could not care, as long as you don't need some native library. In the past the battle with intel had to be played by providing an alternative implementation of the x86 instruction set for precisely the same reason: legacy. The mobile market proved you can achieve good performance with ARM and especially better power per performance. I really can't wait to see some more fights in this arena.
  - justincormack 12y ago
    Many Linux distros have Power support; Ubuntu just added little endian ppc64 support for additional compatibility. JVM is available, and most other software. It is not that much work...
    - ithkuil 12y ago
      Cool, having the same endianness helps. Sure, it's doable, I have a few ppc at home, but I know first hand stories of small/medium companies just not wanting to risk that. They just have higher costs at maintaining some dependencies, custom builds etc. You never know when you will get some new version of something, like jdk8. Then you have things like missing Go compiler (yes there is gccgo for ppc but still not everything works the same). It's perfect for enthusiast, it's ok for companies with strong investment in IT infrastructure. I just wonder what is the best way to convince those small companies that there is no problem. Perhaps the tools/distros etc are starting to mature at the right point and this will soon no longer be a big practical problem.
      - 12y ago
bhouston 12y ago
Can someone explain the benefits of POWER8 as compared to Intel? I though the volume of POWER8 chips being low (as compared to the exceedingly powerful Intel and Arm chips) would mean that innovation in that area would be low as well.
- rwmj 12y ago
  POWER is fast. I have remote access to a 64-way POWER7 server through work and it really rocks.
  - AnthonyMouse 12y ago
    That's the interesting thing about POWER. Uses 250 watts? No problem. Costs $5000? Whatever. They only seem to have one design criteria: It has to be fast.
  - StillBored 12y ago
    The problem with POWER7 has been the IO connectivity. The use of the GX++ bus has been a huge bottleneck (something they are obviously fixing for POWER8). The theoretical bandwidth on GX++ is 20Gbit and so its basically the equivalent of a single x4 PCIE 2.0 slot. This was borderline bad in 2010 when POWER7 was released, now it looks even worse.
- Sanddancer 12y ago
  Ridiculous parallelism. A POWER8 chip has 12 cores, and each core can handle 8 threads. As a result, these chips can keep the pipeline pretty much always full, and provide massive performance boosts to things like database servers.
  - joosters 12y ago
    Sun's Niagara line of processors had similar numbers of threads (64/128), but they often had lacklustre performance. With this amount of parallelism, it becomes very difficult to keep all the threads busy, even for highly scalable programs. You'd get hit by all kinds of problems, like blockages due to memory throughput, or shared resources (IIRC the Niagara threads shared FPUs and other processing blocks, so to get them all running 100% of the time you'd have to manage the workload of your tasks insanely carefully) On top of that, so much stuff just doesn't scale well. On the early Niagaras, even ssh-ing into the machine was noticeably slow. Oh? your crypto doesn't use all 64 threads? Hard luck!
ksec 12y ago
So Presumably, Google will manufacture their own POWER8 CPU. But Who made them? TSMC? GloFo? Not IBM since IBM will be exiting Fab business in the near future. I am going to guess this Dual CPU variant will be aiming at Intel Xeon E5 v2 Series. The 10 - 12 Core version cost from anywhere between $1200 - $2600. Although Google do get huge discount for buying directly from Intel and their volume. Assuming the cost to made each 12 Core POWER8 to be $200, that is a potentially cost saving of $1000 per CPU, and $2000 per Server. The last estimate were around 1 - 1.5 Million Servers at google in 2012 and 2M+ in 2013. May be they are approaching 3M in 2014/15. Even with most of those are low power CPU for storage or other needs. One million CPU made themselves could be savings of up to a billion. Could this, kick start the server and Enterprise Industry to buy POWER8 CPU at much cheaper price? And Once there are enough momentum and software optimization ( JVM ) it could filter down to Web Hosting industry as well. In the best case scenario, this means big trouble for Intel.
- solarexplorer 12y ago
  What would be the point for Google to build their own CPU? How would it be different from what IBM has to offer?
  - barkingcat 12y ago
    The parent poster is referring to IBM leaving the cpu fab business, so of course a Google produced CPU (most likely farmed out to a fab) is better than a ghost cpu produced by a shut down factory by non-existent fab technicians/engineers.
    - wmf 12y ago
      Realistically, you don't shut down a multi-billion-dollar state-of-the-art fab. It may be sold to another company but it will continue to operate.
    - protomyth 12y ago
      Is it confirmed IBM will stop chip fab?
cliveowen 12y ago
Between people shifting from pc to arm-powered phones and major data-center users doing their best to cut costs this is shaping up to be a tough decade for Intel.
- fidotron 12y ago
  In all seriousness, I would not want to be leading Intel right now as I can't imagine what they could actually do to escape this. Hindsight makes Itanium look like even more of a disaster, when that energy in that era should have gone into evolving the x86 platform for the future. Without AMD doing what they did (x86-64) I wonder where Intel would actually stand in the server market today.
  - orbifold 12y ago
    From what I heard modern Intel chips basically only keep up the x86 instruction set as a facade and the architecture beneath is different (much larger number of registers etc.). Wouldn't it potentially be a good idea to do a clean redesign of the "frontend" and eliminate all the legacy support?
    - lnanek2 12y ago
      That's basically what Itanium was and it failed vs. the AMD competitor that kept support.
    - zhemao 12y ago
      Yes, modern Intel CPUs use a RISC-like architecture underneath. The CPU contains a decoder unit which converts x86 instructions to RISC-like micro-ops. Getting rid of x86 support would not be a good idea. It's their "legacy support" which has allowed them to dominate the desktop and server market. Porting your software to a new architecture can be a real pain.
      - mikeash 12y ago
        I wonder if it would be reasonable to come up with a more modern ISA (or just borrow somebody else's, like AArch64?) and offer that as an alternate front end. Keep the x86 decoder front end that they have now. Add another one for the better ISA. Add another mode that kicks the CPU into that ISA. Current x86-64 OSes already generally support two architectures: x86-64 and i386. This would just be a third one. Then everything could move to the new ISA incrementally, and legacy software could keep on working forever using the legacy decoder. I'd guess that the x86 ISA is no longer enough of a bottleneck to justify it. Throw enough transistors at the problem and perhaps it doesn't matter anymore whether your ISA makes any sense.
nkurz 12y ago
In case it helps, the larger context of this story is that IBM has spent a couple billion dollars developing a new server CPU (POWER8) that is just about to come on the market: http://www.forbes.com/sites/alexkonrad/2014/04/23/ibm-debuts-new-power-servers-and-new-open-platform-partnership-with-google/ http://www.forbes.com/sites/alexkonrad/2014/04/23/ibm-debuts... They've also formed a consortium to promote this processor, of which Google is a flagship member (http://openpowerfoundation.org/ http://openpowerfoundation.org/). The expectation (or hope, or fear, depending on your point of view) is that Google may be designing their future server infrastructure around this chip. This motherboard is some of the first concrete evidence of this. The chip is exciting to a lot of people not just because it offer competition to Intel, but because it's the first potentially strong competitor to x86/x64 to appear in the server market for quite a while. By the specs, it's really quite a powerhouse: http://www.extremetech.com/computing/181102-ibm-power8-openpower-x86-server-monopoly http://www.extremetech.com/computing/181102-ibm-power8-openp...
- raverbashing 12y ago
  However, the "Google model" of computation involves a huge amount of cheap "light" servers, instead of a few "big" servers (on which the Power model was based) Well, the Power architecture had some success in Apple products, but ended with the inability of IBM to scale production and produce parts that consumed less power
  - SSLy 12y ago
    I think they could use this on edge servers, as well with those algorithms, that they couldn't distribute enough.
  - justincormack 12y ago
    Google's servers are not that light, and this is a dual socket one, so 20-32 cores or so, rather than a huge Power 16 socket board which are the real scale up ones, so it is not that much more scale up. You get more IO bandwidth out of Power than Intel.
    - 12y ago
listic 12y ago
I wonder if POWER8 based servers will be available for the mass market? I'm not sure whether Google is interested in commoditizing POWER8 servers or just participates in the OpenPOWER foundation to ensure that POWER-based servers will suit their needs. The fact that Google is open about their new motherboard hints at the former, but it's not much. I wonder how non-Google-scale developer could even potentially get to use POWER-based servers. Will they be available from the regular dedicated server hosting companies? What OS could they run? RHEL does support POWER platform, but for a hefty price: https://www.redhat.com/apps/store/server/ https://www.redhat.com/apps/store/server/ CentOS doesn't, presumably because all the POWER hardware CentOS developers could get is either very expensive or esoteric. That likely means I don't have to consider using POWER-based servers for at least 3 years, right?
- jacquesm 12y ago
  Depending on your definition of 'mass' you can buy them now: http://www-03.ibm.com/press/us/en/pressrelease/43702.wss http://www-03.ibm.com/press/us/en/pressrelease/43702.wss and http://www-03.ibm.com/systems/power/hardware/s812l-s822l/browse.html http://www-03.ibm.com/systems/power/hardware/s812l-s822l/bro... I can't stand it how their 'buy now' link for a product with a listed price then links to a 'get a quote' form. If they didn't do stuff like that I might have bought one of their machines instead of the HP that is currently churning away happily (32 cores, 192G of RAM, quite the little beast).
  - listic 12y ago
    What OS would you run on a POWER server? Which model and configuration of HP server did you buy and for how much? I actually had popular dedicated server hosting providers in mind, e.g. Leaseweb. http://www.leaseweb.com/en/dedicated-servers http://www.leaseweb.com/en/dedicated-servers Re: linked servers. Thanks for the concrete info! $8K for 10-core / 3.4 GHz POWER8 with 32 GB RAM and 2x300 GB 10K rpm drives. Those have to be some freakishly good 10 cores, to justify that kind of a price at least for _some_ use cases.
    - jacquesm 12y ago
teepo 12y ago
Would these be too pricey as hypervisors for cloud compute? It seems to me to be ideal for CPU thread intensive applications like databases, on-demand transcoding. What are some use cases for a server like this for Google? I'd love to see these available in the IBM Cloud (SoftLayer) but I think they will be too pricey and reserved for enterprise.
- jameshk 12y ago
  If these are cheap enough (someone said $200 a pop with bulk discounts) then cloud providers will get a big boost.
- huslage 12y ago
  You can also logically partition these beasts into multiple real servers. Who needs a hypervisor when you can have 96 "real" servers sharing the same hardware?
  - sp332 12y ago
    Memory bandwidth would be a nightmare, not to mention every other kind of I/O.
    - mzs 12y ago
      There are some tricks for IO: http://www.redbooks.ibm.com/redpieces/abstracts/redp5065.html http://www.redbooks.ibm.com/redpieces/abstracts/redp5065.htm...
  - mikeash 12y ago
    Pardon my ignorance: what's the difference between using a hypervisor and running "real" servers that share hardware?
foxhill 12y ago
250W TDP in a package that size.. as the article correctly states, it's about how many FLOPs you can get inside a rackmount case. that TDP alone is going to mean that you wont be able to put that many in a single case. a dual socket board, 500W on CPUs, 600W with everything else.. the power supply would have to be something special, but the biggest challenge there would be getting the energy (ala heat) back out of the box.. GPUs have similar TDPs and issues - that's why the HSFs on top of them are so massive (and hence GPUs have a bit of an advantage here - they have the entire PCIE board to fit their cooling hardware on) finally, 4.5ghz? what the hell? in one clock cycle, a beam of light wouldn't even get half way across the board (EDIT: not chip). branch/cache/TLB misses may literally kill any reasonable performance you might hope to get out of it. intel get around this by having years of market leading research in branch predictors, caching models, etc. and it's going to be no mean feat to match that. i know IBM aren't exactly new to this game. but AFAIK x86 has always been faster, clock for clock, than POWER. that said, i hope my concerns are misplaced. i'm hoping intel get some competition in the server room. it will be of benefit to everyone.
- ISL 12y ago
  Light travels 30 cm/nanosecond. 30/4.5 = 6.6 cm, larger than the chip.
- yaakov34 12y ago
  Light would travel about 660 millimetres in 0.22 nanoseconds, and the chip is about 25 millimetres on the side, so a beam of light could run a few laps around the chip in one clock cycle, or bounce off the sides 20-30 times. Maybe you wanted to say across the motherboard? I don't think 4.5 GHz is somehow ridiculous when 3 GHz is routine (and POWER7 was 4.2 GHz). Hundreds of cycles of latency when accessing anything off the chip is now routine - that's the world we live in now. I think that the biggest problem is that IBM is not able to make the investments (especially in semiconductor manufacturing) to match Intel's rate of bringing technology to market. The current POWER7 is a 45-nm device if I remember correctly, and this 22-nm POWER8 is not yet on the market. Intel has been selling 22-nm Haswells for how long now? And of course the POWER7 chips have been up against next-generation semiconductors for most of their life. EDIT: I see that IBM started selling POWER8 systems a few days ago. That's close to a year later than Haswell, and what's more, this chip is likely to compete against 14-nm processors for most of its lifetime.
mark_l_watson 12y ago
Two things. First, slightly off topic: is there anyway this could be a negotiating position with Intel, on price? Second: while many CPU cores (with enough IO) is great for large Borg map reduce jobs, I am curious to see if Google will develop/use better software technology for running general purpose jobs more efficiently on many cores. Properly written Java and Haskell (which I think Google uses a bit in house) help, but the area seems ripe for improvement.
- sp332 12y ago
  Google is a flagship partner in the Power8 consoritium, so I doubt it's just for leverage against Intel.
  - fludlight 12y ago
    Google's position as a major backer of the competition gives them credibility at the negotiating table. Intel won't give them better terms unless Google can demonstrate a viable alternative.
sp332 12y ago
Does that say "little-endian support"? Like you just set a flag and all your math switches from big-endian to little-endian?
- dekhn 12y ago
  there are a number of software-switchable processor arches that support bi-endian.
- termain 12y ago
  I believe Power has long been a bi-endian architecture. I gather it's a switch thrown (in either software or hardware) at startup.
  - klodolph 12y ago
    The old PowerPC processors did this by flipping the low bits of memory addresses when in little-endian mode, but the data lanes had to be reversed to make this work. So back in the day, it only meant that you could use the same chip for a little-endian design but not the same motherboard. I don't think that's how newer POWER processors work, though.
    - protomyth 12y ago
      The 970 (G5) didn't have the little-endian mode. Virtual PC was impacted.
    - KMag 12y ago
      Correction: a modified memory controller wasn't necessary as long as all of your memory accesses were naturally aligned. As I remember, in little-endian mode, unaligned accesses would also trap to the kernel, so kernel authors could include code that would fix things up, at a huge performance penalty for unaligned access. Most architectures that support unaligned access have a small penalty for unaligned access anyway, and some architectures (Does anyone remember Netscape Navigator on Solaris SPARC crashing with SIGBUS much more often than the same Navigator release crashing on x86? At least Solaris 6/7 didn't include kernel code to emulate support for aligned memory access on SPARC.) don't support it, so it's best to avoid unaligned memory access in C code. I don't recall the JVM specification forcing a particular object layout on an implementation, and I believe most JVM implementations naturally align all object fields rather than packing them for minimum space usage. I believe an implementation could reorder the fields in order to optimally pack them while avoiding unaligned accesses, at the cost of breaking any hand optimization of locality of reference made by the programmer. However, I think the space savings for almost all programs would be very meager.
peterfisher 12y ago
I love when google announces something through Google+
zurn 12y ago
So they're saying it's easier to use a brand new incompatible little endian Linux personality, with associated new toolchains and new ports of low level stuff etc compared to the standard Linux PPC64 stuff... Sounds kind of surprising even if IBM did some of the bringup work ahead of time, but maybe they've got little endian assumptions baked in many internal protocols/apps.
- rbanffy 12y ago
  Linux has supported POWER for ages. Is endianness such a big issue? Why?
  - sparkie 12y ago
    Endianness is an issue because programmers ignore it - they think "undefined behavior" is a synonym for "not yet standardized", and the mentality of "works on my machine" typically trumps concerns of portability. This isn't a concern for low level developers, such as the kernel developers - they understand the concerns and take care to implement code in portable ways. The issue is with user-space developers who think C and C++ are a good choice of language, and they have no qualms using bitfields, unguarded compiler pragmas, violating strict aliasing rule, and failing to specify the endianness their protocols use in the protocol itself (BoMs are not universally used) - also there is often a failure to provide the endianness conversions in implementations of such protocols where necessary. Not to mention a complete lack of standard way to test the endianness of the current machine, which typically requires violating the strict aliasing rule to check.
    - rbanffy 12y ago
      I believe it's safe to assume having more endian-diversity is then a good thing. Bugs in software and protocols will be exposed and eventually corrected. Since most Linux distros fully support a very diverse set of machines, endianness is usually not a problem with most of the software that's already part of a Linux distro. As for software developed inside Google, they hire smart people. They'll manage.
  - 12y ago
cdi 12y ago
Large photo of this motherboard: https://www.flickr.com/photos/ibmevents/14051347355/sizes/o/ https://www.flickr.com/photos/ibmevents/14051347355/sizes/o/ They've masked all the chips with something black. Are they hiding chips they are using, or is this something for thermal dissipation?
- wmf 12y ago
  Looks like typical Google "secret transparency". You can look but you won't learn anything.
Corrado 12y ago
I think its interesting that they didn't include the "traditional" mouse/keyboard/VGA ports. Not particularly surprised since this is a server motherboard, but still interesting. I think I do see an HDMI connector in the lower right next to a tall silver port (possible USB connector).

196 points by self 12y ago

Funny layout. I would like to know why the PCI slots a spread out like that. I know Google don't have a standard rack setup, but still, it would make seens to have all the expantion ports the end of the board... No?

Sanddancer 12y ago
A layout like this means you can use both sides of the motherboard for I/O slots. So in a 1/2u box, you can get more than one or two expansion cards in place. The PCI slots themselves seem to be hammock connectors, which I was curious about too, and googling doesn't seem to have any info, unless it's too early/late and I'm missing the obvious.
nuriaion 12y ago
Maybe these Connectors are for daughterboards with Centaur Chips + RAM. (A Power8 can connect to 8 Centaur Chips where the RAM is connected)
- justincormack 12y ago
  They must be, as there are otherwise no RAM chips on the board...
deleted 12y ago
[deleted]

z3phyr 12y ago

It would be great if somebody could list modern computers for personal use which are still based on Power architecture?

msiebuhr 12y ago
Both Xbox360 and PS3 use modified POWER-designs. Edit: Also, it's notable that both Xbox One and PS4 switched to use x64.
- riffraff 12y ago
  Wii and WiiU also have POWER-based designs IIRC.
  - carey 12y ago
    Both are PowerPC 750-based according to Wikipedia, which is part of why the Wii U runs games released for the Wii. Note that PowerPC, as used in the Wii, Wii U and old Macs, is not exactly the same as POWER, as in this announcement. The POWER architecture is used by IBM AIX and AS/400 servers, and by the PS3 in its Cell variant.
    - DCKing 12y ago
      There is contradictory evidence whether the Cell's PPE was 'PowerPC' or just a 'Power' core. In any case, it could run PowerPC software. Incidentally, three of those exact same cores are used as the CPU of the Xbox 360. For reference, the PowerPC 750 derivatives in the GameCube/Wii/Wii U are in the same family as the PowerPC G3 used in Macs around the turn of the century. It is also related to the CPU running Curiosity on Mars. So yeah, although Nintendo is a big customer of the Power architecture at the moment, they're not really breaking new ground.
      - DiabloD3 12y ago
        The Cell is the world's first in order execution PowerPC. Its similar in design to the G3 family but has a very high clock speed. They stripped a lot out of the chip design (such as the out of order execution pipeline, a lot of the cache brains, etc) to get the core as small and as low power as possible while relying on modern compilers to make the magic happen. I'm not entirely sure they succeeded in their goals, but with how well SPEs are used in PS3 games, I'm not sure it matters.

jmnicolas 12y ago

It's a bit short on details imo : where are the specs, the benchmarks etc ?

jacquesm 12y ago
It's actually quite impressive that google would open up this much of their secret sauce, a lot can be gleaned from looking at this board. You can bet that this is not exactly revision one (and you can bet as well that this is likely not their latest and greatest, no need to show off more than you have to, competitive edges are pretty thin). When I see stuff like this it is painfully clear that from a technological perspective a company like duck-duck-go has a huge amount of defensible moat to cross before they can begin to be a serious contender. Think about it for a second: the company that you're trying to compete with is operating at such economies of scale that it can afford to have its own custom motherboards + non-standard expansion boards made.

fh973 12y ago

This is significant. With POWER back in the game, and ARM server CPUs arriving, Intel will again have competition.

ithkuil 12y ago
I think currently there are few big customers that can afford the overhead of porting their code and dependencies to a different architecture. For example I don't expect cloud providers to have a huge marked soon for non x86 architectures. Well there are JVM or other VM users which in theory could not care, as long as you don't need some native library. In the past the battle with intel had to be played by providing an alternative implementation of the x86 instruction set for precisely the same reason: legacy. The mobile market proved you can achieve good performance with ARM and especially better power per performance. I really can't wait to see some more fights in this arena.
- justincormack 12y ago
  Many Linux distros have Power support; Ubuntu just added little endian ppc64 support for additional compatibility. JVM is available, and most other software. It is not that much work...
  - ithkuil 12y ago
    Cool, having the same endianness helps. Sure, it's doable, I have a few ppc at home, but I know first hand stories of small/medium companies just not wanting to risk that. They just have higher costs at maintaining some dependencies, custom builds etc. You never know when you will get some new version of something, like jdk8. Then you have things like missing Go compiler (yes there is gccgo for ppc but still not everything works the same). It's perfect for enthusiast, it's ok for companies with strong investment in IT infrastructure. I just wonder what is the best way to convince those small companies that there is no problem. Perhaps the tools/distros etc are starting to mature at the right point and this will soon no longer be a big practical problem.
    - 12y ago

bhouston 12y ago

Can someone explain the benefits of POWER8 as compared to Intel? I though the volume of POWER8 chips being low (as compared to the exceedingly powerful Intel and Arm chips) would mean that innovation in that area would be low as well.

rwmj 12y ago
POWER is fast. I have remote access to a 64-way POWER7 server through work and it really rocks.
- AnthonyMouse 12y ago
  That's the interesting thing about POWER. Uses 250 watts? No problem. Costs $5000? Whatever. They only seem to have one design criteria: It has to be fast.
- StillBored 12y ago
  The problem with POWER7 has been the IO connectivity. The use of the GX++ bus has been a huge bottleneck (something they are obviously fixing for POWER8). The theoretical bandwidth on GX++ is 20Gbit and so its basically the equivalent of a single x4 PCIE 2.0 slot. This was borderline bad in 2010 when POWER7 was released, now it looks even worse.
Sanddancer 12y ago
Ridiculous parallelism. A POWER8 chip has 12 cores, and each core can handle 8 threads. As a result, these chips can keep the pipeline pretty much always full, and provide massive performance boosts to things like database servers.
- joosters 12y ago
  Sun's Niagara line of processors had similar numbers of threads (64/128), but they often had lacklustre performance. With this amount of parallelism, it becomes very difficult to keep all the threads busy, even for highly scalable programs. You'd get hit by all kinds of problems, like blockages due to memory throughput, or shared resources (IIRC the Niagara threads shared FPUs and other processing blocks, so to get them all running 100% of the time you'd have to manage the workload of your tasks insanely carefully) On top of that, so much stuff just doesn't scale well. On the early Niagaras, even ssh-ing into the machine was noticeably slow. Oh? your crypto doesn't use all 64 threads? Hard luck!

ksec 12y ago

So Presumably, Google will manufacture their own POWER8 CPU. But Who made them? TSMC? GloFo? Not IBM since IBM will be exiting Fab business in the near future. I am going to guess this Dual CPU variant will be aiming at Intel Xeon E5 v2 Series. The 10 - 12 Core version cost from anywhere between $1200 - $2600. Although Google do get huge discount for buying directly from Intel and their volume. Assuming the cost to made each 12 Core POWER8 to be $200, that is a potentially cost saving of $1000 per CPU, and $2000 per Server. The last estimate were around 1 - 1.5 Million Servers at google in 2012 and 2M+ in 2013. May be they are approaching 3M in 2014/15. Even with most of those are low power CPU for storage or other needs. One million CPU made themselves could be savings of up to a billion. Could this, kick start the server and Enterprise Industry to buy POWER8 CPU at much cheaper price? And Once there are enough momentum and software optimization ( JVM ) it could filter down to Web Hosting industry as well. In the best case scenario, this means big trouble for Intel.

solarexplorer 12y ago
What would be the point for Google to build their own CPU? How would it be different from what IBM has to offer?
- barkingcat 12y ago
  The parent poster is referring to IBM leaving the cpu fab business, so of course a Google produced CPU (most likely farmed out to a fab) is better than a ghost cpu produced by a shut down factory by non-existent fab technicians/engineers.
  - wmf 12y ago
    Realistically, you don't shut down a multi-billion-dollar state-of-the-art fab. It may be sold to another company but it will continue to operate.
  - protomyth 12y ago
    Is it confirmed IBM will stop chip fab?

cliveowen 12y ago

Between people shifting from pc to arm-powered phones and major data-center users doing their best to cut costs this is shaping up to be a tough decade for Intel.

fidotron 12y ago
In all seriousness, I would not want to be leading Intel right now as I can't imagine what they could actually do to escape this. Hindsight makes Itanium look like even more of a disaster, when that energy in that era should have gone into evolving the x86 platform for the future. Without AMD doing what they did (x86-64) I wonder where Intel would actually stand in the server market today.
- orbifold 12y ago
  From what I heard modern Intel chips basically only keep up the x86 instruction set as a facade and the architecture beneath is different (much larger number of registers etc.). Wouldn't it potentially be a good idea to do a clean redesign of the "frontend" and eliminate all the legacy support?
  - lnanek2 12y ago
    That's basically what Itanium was and it failed vs. the AMD competitor that kept support.
  - zhemao 12y ago
    Yes, modern Intel CPUs use a RISC-like architecture underneath. The CPU contains a decoder unit which converts x86 instructions to RISC-like micro-ops. Getting rid of x86 support would not be a good idea. It's their "legacy support" which has allowed them to dominate the desktop and server market. Porting your software to a new architecture can be a real pain.
    - mikeash 12y ago
      I wonder if it would be reasonable to come up with a more modern ISA (or just borrow somebody else's, like AArch64?) and offer that as an alternate front end. Keep the x86 decoder front end that they have now. Add another one for the better ISA. Add another mode that kicks the CPU into that ISA. Current x86-64 OSes already generally support two architectures: x86-64 and i386. This would just be a third one. Then everything could move to the new ISA incrementally, and legacy software could keep on working forever using the legacy decoder. I'd guess that the x86 ISA is no longer enough of a bottleneck to justify it. Throw enough transistors at the problem and perhaps it doesn't matter anymore whether your ISA makes any sense.

nkurz 12y ago

In case it helps, the larger context of this story is that IBM has spent a couple billion dollars developing a new server CPU (POWER8) that is just about to come on the market: http://www.forbes.com/sites/alexkonrad/2014/04/23/ibm-debuts-new-power-servers-and-new-open-platform-partnership-with-google/ http://www.forbes.com/sites/alexkonrad/2014/04/23/ibm-debuts... They've also formed a consortium to promote this processor, of which Google is a flagship member (http://openpowerfoundation.org/ http://openpowerfoundation.org/). The expectation (or hope, or fear, depending on your point of view) is that Google may be designing their future server infrastructure around this chip. This motherboard is some of the first concrete evidence of this. The chip is exciting to a lot of people not just because it offer competition to Intel, but because it's the first potentially strong competitor to x86/x64 to appear in the server market for quite a while. By the specs, it's really quite a powerhouse: http://www.extremetech.com/computing/181102-ibm-power8-openpower-x86-server-monopoly http://www.extremetech.com/computing/181102-ibm-power8-openp...

raverbashing 12y ago
However, the "Google model" of computation involves a huge amount of cheap "light" servers, instead of a few "big" servers (on which the Power model was based) Well, the Power architecture had some success in Apple products, but ended with the inability of IBM to scale production and produce parts that consumed less power
- SSLy 12y ago
  I think they could use this on edge servers, as well with those algorithms, that they couldn't distribute enough.
- justincormack 12y ago
  Google's servers are not that light, and this is a dual socket one, so 20-32 cores or so, rather than a huge Power 16 socket board which are the real scale up ones, so it is not that much more scale up. You get more IO bandwidth out of Power than Intel.
  - 12y ago

listic 12y ago

I wonder if POWER8 based servers will be available for the mass market? I'm not sure whether Google is interested in commoditizing POWER8 servers or just participates in the OpenPOWER foundation to ensure that POWER-based servers will suit their needs. The fact that Google is open about their new motherboard hints at the former, but it's not much. I wonder how non-Google-scale developer could even potentially get to use POWER-based servers. Will they be available from the regular dedicated server hosting companies? What OS could they run? RHEL does support POWER platform, but for a hefty price: https://www.redhat.com/apps/store/server/ https://www.redhat.com/apps/store/server/ CentOS doesn't, presumably because all the POWER hardware CentOS developers could get is either very expensive or esoteric. That likely means I don't have to consider using POWER-based servers for at least 3 years, right?

jacquesm 12y ago
Depending on your definition of 'mass' you can buy them now: http://www-03.ibm.com/press/us/en/pressrelease/43702.wss http://www-03.ibm.com/press/us/en/pressrelease/43702.wss and http://www-03.ibm.com/systems/power/hardware/s812l-s822l/browse.html http://www-03.ibm.com/systems/power/hardware/s812l-s822l/bro... I can't stand it how their 'buy now' link for a product with a listed price then links to a 'get a quote' form. If they didn't do stuff like that I might have bought one of their machines instead of the HP that is currently churning away happily (32 cores, 192G of RAM, quite the little beast).
- listic 12y ago
  What OS would you run on a POWER server? Which model and configuration of HP server did you buy and for how much? I actually had popular dedicated server hosting providers in mind, e.g. Leaseweb. http://www.leaseweb.com/en/dedicated-servers http://www.leaseweb.com/en/dedicated-servers Re: linked servers. Thanks for the concrete info! $8K for 10-core / 3.4 GHz POWER8 with 32 GB RAM and 2x300 GB 10K rpm drives. Those have to be some freakishly good 10 cores, to justify that kind of a price at least for _some_ use cases.
  - jacquesm 12y ago

teepo 12y ago

Would these be too pricey as hypervisors for cloud compute? It seems to me to be ideal for CPU thread intensive applications like databases, on-demand transcoding. What are some use cases for a server like this for Google? I'd love to see these available in the IBM Cloud (SoftLayer) but I think they will be too pricey and reserved for enterprise.

jameshk 12y ago
If these are cheap enough (someone said $200 a pop with bulk discounts) then cloud providers will get a big boost.
huslage 12y ago
You can also logically partition these beasts into multiple real servers. Who needs a hypervisor when you can have 96 "real" servers sharing the same hardware?
- sp332 12y ago
  Memory bandwidth would be a nightmare, not to mention every other kind of I/O.
  - mzs 12y ago
    There are some tricks for IO: http://www.redbooks.ibm.com/redpieces/abstracts/redp5065.html http://www.redbooks.ibm.com/redpieces/abstracts/redp5065.htm...
- mikeash 12y ago
  Pardon my ignorance: what's the difference between using a hypervisor and running "real" servers that share hardware?

foxhill 12y ago

250W TDP in a package that size.. as the article correctly states, it's about how many FLOPs you can get inside a rackmount case. that TDP alone is going to mean that you wont be able to put that many in a single case. a dual socket board, 500W on CPUs, 600W with everything else.. the power supply would have to be something special, but the biggest challenge there would be getting the energy (ala heat) back out of the box.. GPUs have similar TDPs and issues - that's why the HSFs on top of them are so massive (and hence GPUs have a bit of an advantage here - they have the entire PCIE board to fit their cooling hardware on) finally, 4.5ghz? what the hell? in one clock cycle, a beam of light wouldn't even get half way across the board (EDIT: not chip). branch/cache/TLB misses may literally kill any reasonable performance you might hope to get out of it. intel get around this by having years of market leading research in branch predictors, caching models, etc. and it's going to be no mean feat to match that. i know IBM aren't exactly new to this game. but AFAIK x86 has always been faster, clock for clock, than POWER. that said, i hope my concerns are misplaced. i'm hoping intel get some competition in the server room. it will be of benefit to everyone.

ISL 12y ago
Light travels 30 cm/nanosecond. 30/4.5 = 6.6 cm, larger than the chip.
yaakov34 12y ago
Light would travel about 660 millimetres in 0.22 nanoseconds, and the chip is about 25 millimetres on the side, so a beam of light could run a few laps around the chip in one clock cycle, or bounce off the sides 20-30 times. Maybe you wanted to say across the motherboard? I don't think 4.5 GHz is somehow ridiculous when 3 GHz is routine (and POWER7 was 4.2 GHz). Hundreds of cycles of latency when accessing anything off the chip is now routine - that's the world we live in now. I think that the biggest problem is that IBM is not able to make the investments (especially in semiconductor manufacturing) to match Intel's rate of bringing technology to market. The current POWER7 is a 45-nm device if I remember correctly, and this 22-nm POWER8 is not yet on the market. Intel has been selling 22-nm Haswells for how long now? And of course the POWER7 chips have been up against next-generation semiconductors for most of their life. EDIT: I see that IBM started selling POWER8 systems a few days ago. That's close to a year later than Haswell, and what's more, this chip is likely to compete against 14-nm processors for most of its lifetime.

mark_l_watson 12y ago

Two things. First, slightly off topic: is there anyway this could be a negotiating position with Intel, on price? Second: while many CPU cores (with enough IO) is great for large Borg map reduce jobs, I am curious to see if Google will develop/use better software technology for running general purpose jobs more efficiently on many cores. Properly written Java and Haskell (which I think Google uses a bit in house) help, but the area seems ripe for improvement.

sp332 12y ago
Google is a flagship partner in the Power8 consoritium, so I doubt it's just for leverage against Intel.
- fludlight 12y ago
  Google's position as a major backer of the competition gives them credibility at the negotiating table. Intel won't give them better terms unless Google can demonstrate a viable alternative.

sp332 12y ago

Does that say "little-endian support"? Like you just set a flag and all your math switches from big-endian to little-endian?

dekhn 12y ago
there are a number of software-switchable processor arches that support bi-endian.
termain 12y ago
I believe Power has long been a bi-endian architecture. I gather it's a switch thrown (in either software or hardware) at startup.
- klodolph 12y ago
  The old PowerPC processors did this by flipping the low bits of memory addresses when in little-endian mode, but the data lanes had to be reversed to make this work. So back in the day, it only meant that you could use the same chip for a little-endian design but not the same motherboard. I don't think that's how newer POWER processors work, though.
  - protomyth 12y ago
    The 970 (G5) didn't have the little-endian mode. Virtual PC was impacted.
  - KMag 12y ago
    Correction: a modified memory controller wasn't necessary as long as all of your memory accesses were naturally aligned. As I remember, in little-endian mode, unaligned accesses would also trap to the kernel, so kernel authors could include code that would fix things up, at a huge performance penalty for unaligned access. Most architectures that support unaligned access have a small penalty for unaligned access anyway, and some architectures (Does anyone remember Netscape Navigator on Solaris SPARC crashing with SIGBUS much more often than the same Navigator release crashing on x86? At least Solaris 6/7 didn't include kernel code to emulate support for aligned memory access on SPARC.) don't support it, so it's best to avoid unaligned memory access in C code. I don't recall the JVM specification forcing a particular object layout on an implementation, and I believe most JVM implementations naturally align all object fields rather than packing them for minimum space usage. I believe an implementation could reorder the fields in order to optimally pack them while avoiding unaligned accesses, at the cost of breaking any hand optimization of locality of reference made by the programmer. However, I think the space savings for almost all programs would be very meager.

peterfisher 12y ago

I love when google announces something through Google+

zurn 12y ago

So they're saying it's easier to use a brand new incompatible little endian Linux personality, with associated new toolchains and new ports of low level stuff etc compared to the standard Linux PPC64 stuff... Sounds kind of surprising even if IBM did some of the bringup work ahead of time, but maybe they've got little endian assumptions baked in many internal protocols/apps.

rbanffy 12y ago
Linux has supported POWER for ages. Is endianness such a big issue? Why?
- sparkie 12y ago
  Endianness is an issue because programmers ignore it - they think "undefined behavior" is a synonym for "not yet standardized", and the mentality of "works on my machine" typically trumps concerns of portability. This isn't a concern for low level developers, such as the kernel developers - they understand the concerns and take care to implement code in portable ways. The issue is with user-space developers who think C and C++ are a good choice of language, and they have no qualms using bitfields, unguarded compiler pragmas, violating strict aliasing rule, and failing to specify the endianness their protocols use in the protocol itself (BoMs are not universally used) - also there is often a failure to provide the endianness conversions in implementations of such protocols where necessary. Not to mention a complete lack of standard way to test the endianness of the current machine, which typically requires violating the strict aliasing rule to check.
  - rbanffy 12y ago
    I believe it's safe to assume having more endian-diversity is then a good thing. Bugs in software and protocols will be exposed and eventually corrected. Since most Linux distros fully support a very diverse set of machines, endianness is usually not a problem with most of the software that's already part of a Linux distro. As for software developed inside Google, they hire smart people. They'll manage.
- 12y ago

cdi 12y ago

Large photo of this motherboard: https://www.flickr.com/photos/ibmevents/14051347355/sizes/o/ https://www.flickr.com/photos/ibmevents/14051347355/sizes/o/ They've masked all the chips with something black. Are they hiding chips they are using, or is this something for thermal dissipation?

wmf 12y ago
Looks like typical Google "secret transparency". You can look but you won't learn anything.

Corrado 12y ago

I think its interesting that they didn't include the "traditional" mouse/keyboard/VGA ports. Not particularly surprised since this is a server motherboard, but still interesting. I think I do see an HDMI connector in the lower right next to a tall silver port (possible USB connector).