9 ms·
The Future Google Rackspace Power9 System
- nickpeterson 11y agoHey Google, sell these to other companies :)
- chronid 11y agoWell, these are not power9, but in theory... :P http://www.penguincomputing.com/products/rackmount-servers/openpower-servers/ http://www.penguincomputing.com/products/rackmount-servers/o...
- nullc 11y agoAnyone have any idea on the rough prices of these systems? Or is it the "if you have to ask, it's too much for you" as seem to be the case with the IBM power systems?
- loeg 11y agoTalos plan to, if demand allows, sell you a bare bones POWER8 (CPU, heatsink, and mainboard) for $3,700 USD. https://raptorengineeringinc.com/TALOS/prerelease.php https://raptorengineeringinc.com/TALOS/prerelease.php
- nullc 11y agoSadly that's $1000 more than they were originally talking about; still low compared to IBMs prices, but much harder to justify for most applications.
- loeg 11y agoYes, very hard to justify when a high-end 4-core Xeon outperforms on some benchmarks[0] and costs half as much or less. Not to mention vastly mature open source floating point, compiler, etc, support. As well as existing (invalid, but still) programs that assume x86isms. [0]: https://www.phoronix.com/scan.php?page=article&item=talos-workstation&num=3 https://www.phoronix.com/scan.php?page=article&item=talos-wo...
- 0xcde4c3db 11y agoMost POWER8 stuff does seem to be in the "contact us" category. The only reference point I know of is that Tyan has (had?) a reference platform for 2850 USD, but that's basically beta quality and not intended to be production hardware. http://www.tyan.com/campaign/openpower/ http://www.tyan.com/campaign/openpower/
- _delirium 11y agoIBM has public pricing for most of their POWER8 servers. The price is high, but public. :) http://www-03.ibm.com/systems/power/hardware/linux.html http://www-03.ibm.com/systems/power/hardware/linux.html
- ajdlinux 11y agoSome IBM Power Systems machines can now be ordered online (at least for US customers) - https://www.ibm.com/marketplace/cloud/big-data-infrastructure/purchase/us/en-us https://www.ibm.com/marketplace/cloud/big-data-infrastructur.... [Disclosure: IBMer]
- mtgx 11y agoWistron will probably sell them: http://www.anandtech.com/show/10230/ibm-nvidia-and-wistron-develop-new-openpower-hpc-server-with-power8-cpus-nvlink http://www.anandtech.com/show/10230/ibm-nvidia-and-wistron-d...
- crudbug 11y agoIt will be great if Dell / HP / Cisco / Lenovo others start forging some POWER gear .. For a platform to succees, it should provide low barrier of entry - may be a low capacity P9 system at a low cost ~ 1K ? Will be a better strategy for OF. I know Power is targeting cloud computing applications, but IBM should consider low cost entry level gear for getting some market share at the lower level which can transition to higher margin markets .
- transfire 11y agoI am surprised. I thought 64-bit ARM was the newness headed to the server farms.
- DannyBee 11y agoIt will be there eventually. It is definitely not there now, despite what some may have you think :)
- StreamBright 11y agoDo you have some data to back this prediction? What is the biggest advantage over x86 server processors?
- DannyBee 11y agoWhich, that it isn't there now? Or that it will be there eventually? The x86 server processors have too much legacy they can't get rid of, and that limits how far they can push it.
- wyldfire 11y agoARM instruction set is pretty mature but the system architecture for servers is less so, IMO. I think there's little commonality for bootstrapping the various SoCs.
- bogomipz 11y agoIs there enough juice in ARM chips to power servers and compete with Intel offerings like Skylake/Haswell?
- virtuallynathan 11y agoI wonder if the inclusion of NVLink in Power 8+ will cause Power to excel in ML applications. It could well be quite a bit faster than x86 just due to the memory/interconnect bandwidth.
- PeCaN 11y agoNVLink and CAPI[1] both have huge potential for machine learning. However, a lot of the benefits of NVLink for ML come from GPU-to-GPU NVLink, which doesn't require CPU support. 1. CAPI doesn't seem to get mentioned to much around here, but imagine an FPGA directly accessing some shared system memory. It's neat.
- DiabloD3 11y agoIt does seem to require support in the PCI-E host controller, which for both modern Intel and those POWER machines, is on die on the CPU. So, it "requires" CPU support, just not in the way usually meant.
- mikehollinger 11y agoCorrect; the CPU and the end-point accelerator both must cooperate to negotiate the CAPI link. Disclaimer: I work on this with some very smart people @ IBM. Opinions are my own. When a PCIe device is in CAPI mode, the PCIe protocol is used as a transport layer, but the CAPI protocol rides on top, and hardware in the CPU's PHB (the CAPP unit) and hardware in the accelerator (the PSL in this case) cooperate to present the common address space to the process and to the accelerator itself. [1] If a CAPI-capable card's plugged in to a non-CAPI-capable slot, it remains a PCI card. If a non-CAPI card's plugged in to a CAPI-capable system, it remains a PCI card. If both sides match on protocol versions and the kernel contains the cxl driver, the kernel will switch the slot into CAPI mode, and the CAPP unit and PSL effectively take over the PCI link on either side. [1] http://events.linuxfoundation.org/sites/events/files/slides/lcjp15_mackerras.pdf http://events.linuxfoundation.org/sites/events/files/slides/... - see page 13+ for some GPU / NVLink materials, and page 24+ for CAPI materials (oh - and I worked on the product who's data is quoted on page 29 [2]! [2] https://www.ibm.com/developerworks/community/blogs/fe313521-2e95-46f2-817d-44a4f27eba32/entry/power8_capi_flash_in_memory_expansion_to_speed_data_access?lang=en https://www.ibm.com/developerworks/community/blogs/fe313521-...
- bluedino 11y agoIBM reps love to throw around the "Google is switching to IBM" line. Can they possibly compete with IBM on price? Why isn't AMD trying to reach this market?
- rdtsc 11y agoAMD would still have an AMD64 architecture though? Or are you thinking they should come up with a new competing architecture.
- derefr 11y agoThese POWER chips are an open design; thus, anyone who wants Google as a customer and has a chip fab ready to go (like, say, AMD) could, in theory just fab some up to sell to Google.
- WoodenChair 11y agoAMD doesn't have a chip fab - they are fabless now.
- monksy 11y agoOh wow. Is there a good reason for this?
- snuxoll 11y agoMaintaining a fab when you can't justify enough orders for chips to keep it running around the clock is expensive. As part of the spinoff they had penalties because they weren't purchasing enough wafers from GF anyway, but it resulted in less bleeding than owning and managing it as a subsidiary.
- mgo 11y agoThey're broke and they need at least $750MM to $1B in the bank to operate effectively as a company.
- voltagex_ 11y agoWhat's the current state of Power development in the Linux kernel like? I thought it was only IBM holding the fort (via ozLabs) but this could be a big boost.
- cpeterso 11y agoWhy does Facebook's Open Rack use a nonstandard rack size? That seems like an obvious barrier for adoption of hardware that was designed to be a commodity.
- DiabloD3 11y agoRacks were designed to fit telecom hardware originally. The Open Rack size is designed around common computer hardware sizes. It isn't a barrier for adoption because swapping racks out of a datacenter is easy, and they fit on standard datacenter floor tiles. What is a barrier is that damned 48V. Disclaimer: I run a hosting company.
- rdtsc 11y ago> What is a barrier is that damned 48V. Interesting what is the issue with 48V, I saw equipment for it seemed to be overpriced. Remember pricing out some stuff and as soon as 48V option for power came into play then price rose quite a bit. Or is it that voltage is not high enough to be efficient for a large data center?
- DiabloD3 11y agoThe voltage is high enough. The way Facebook's solution is, is you have triplets of racks: left, right racks hold computers, middle rack holds network, power distribution, and UPS. The computers have extremely simplistic power supplies that basically can't fail, and just DC->DC transform from 48v to 12, 5, and 3.3; and the large scale power supplies that convert three phase 240v (or whatever you're supplying it with) from the datacenter to 48V are much higher efficiency than the ones that would have been in the server (which you would have fed them, usually, something like single phase 208v). Redundancy is supplied by just hooking multiple transformers in the middle rack to the + and - terminals on each PSU, instead of a convoluted multi-module redundant PSU (which always uses a single backplane, and backplanes in redundant PSUs fail surprisingly frequently). The total round trip efficiency of this system is about as high as you can realistically get. 80Plus Titanium is 90-95% efficient (depending on load), but has efficiency losses in rack level distribution which 48V tries to correct. However, 48V DC can be very dangerous to work with, and a lot of tech workers refuse to work with it. Now, if you believe it is dangerous or not (I've seen arguments stating that it is no more dangerous than single phase 208v) is immaterial, this is the opinion of a lot of workers. The cost of 48V gear is expensive if you're not in a datacenter already setup to handle it. Facebook obviously doesn't have this problem because they build entire datacenters from scratch. I personally don't believe in it because it doesn't buy me anything that single phase 208v doesn't give me, I do not pay enough in electricity to have the overhead of dealing with it.
- ksec 11y agoThey are going up against the coming Xeon E5 Broadwell + FPGA. Power9 do offer more memory per Rack. But I dont see how Intel cant adopt with better memory controller. To simply put, what are the incentive to switch over to Power9 platform?
- dman 11y agoHaving viable options to Intel would be very helpful for Google when negotiating bulk rates for the Xeon processors.
- thrownaway2424 11y agoYeah exactly. Google waves these around from time to time to have something over which to negotiate.
- bogomipz 11y agoKind of like how Google does with Google Fiber? They seem to engage in this just enough to keep carriers in in limited markets but not much more.
- ksec 11y agoIf Zen was as good as the rumored, then waving a AMD Zen Server processor would be much more useful.
- bogomipz 11y agoWhats the Zen offering? This is the first I've heard of it. Unfortunate name considering theres already Xen.
- ksec 11y agoNext generation uArch from AMD. Which promise 40% IPC improvement . Again it won't be as fast as Intel, but at least AMD is within Reach for a price war, where as now even if AMD is 50% cheaper it makes little sense to use them.
- mozumder 11y agoKinda amazing that they can fit 2 Power9's as well as 2 FHFL PCIEx16 slots, along with 15 drives and 2TB of memory in 1 rack unit.
- xiaopingguo 11y agoWasn't Google at one point all about commodity/consumer level hardware for their servers? Seems a huge turnaround.
- sargun 11y agoThis is still largely at commodity prices / performance points. It's been quite some time since any of their hardware has looked consumer-oriented, but comparing this to what enterprises buy, it's apples and oranges. [1] http://shop.oreilly.com/product/0636920041528.do http://shop.oreilly.com/product/0636920041528.do [2] http://research.google.com/pubs/pub35290.html http://research.google.com/pubs/pub35290.html
- nl 11y agoTheNextPlatform is a pretty bad site. They rehash - badly - information available elsewhere, and add a hyperactive spin on it all. Here's the truth: Google uses lots of compute power (insightful!) Google isn't shifting to Power. Google does have an active R&D program looking at Power. TheNextPlatform misses the whole point here: That Zaius board has 32 DDR4 slots (commercially available servers from eg Dell max out at 24) and it has 2 NVLINK slots! (!!) Those NVLINK slots are what Intel should be worried about, because that's where Google is prepared to pay money. They are building computers that lock themselves into NVidia and doing it gladly. Intel better find a way to compete with NVidia on deep learning.
- exhilaration 11y agoFor anyone else wondering what NVLINK is, some links to save you a Google search: https://blogs.nvidia.com/blog/2014/11/14/what-is-nvlink/ https://blogs.nvidia.com/blog/2014/11/14/what-is-nvlink/ http://www.tomshardware.com/news/nvidia-nvlink-boosts-performance,28989.html http://www.tomshardware.com/news/nvidia-nvlink-boosts-perfor... I'd be curious to hear what Intel is developing to compete with this.
- bogomipz 11y agoDo you have similar links for CAPI? I feel like that along with NVLINK seem to be present in all the POWER literature.
- oneweekwonder 11y agoTo add to the above, From Wikipedia[0]: NVLink is a communications protocol developed by Nvidia. NVLink specifies a point-to-point connection between a CPU and a GPU and also between a GPU and another GPU. NVLink products introduced to date focus on the high-performance application space. and [1]: NVLink – a power-efficient high-speed bus between the CPU and GPU, and between multiple GPUs. Allows much higher transfer speeds than those achievable by using PCI Express; estimated to provide between 80 and 200 GB/s. [0]: https://en.wikipedia.org/wiki/NVLink https://en.wikipedia.org/wiki/NVLink [1]: https://en.wikipedia.org/wiki/GeForce#PASCAL https://en.wikipedia.org/wiki/GeForce#PASCAL