6 ms·
Cloudflare CEO: Why we’re switching to ARM-based servers
- Aissen 9y agoA blog post is coming, so I might have jumped the shark here. An interesting note from a Cloudflare engineer: Most powerful Xeon is the 28 core Platinum 2180 at $10k RSP and >200W TDP. Due to the nature of Intel turbo boost it almost always operates at TDP under load. Our ARM is the Centriq 2452 at less than $1400 RSP, 46 cores and 120W TDP, that it never hits. Beats Xeon 9/10 workloads. https://twitter.com/thecomp1ler/status/976617883164921857 https://twitter.com/thecomp1ler/status/976617883164921857
- dvfjsdhgfv 9y agoThis is amazing! We're witnessing the beginning of the demise of Intel in the data canter, right in front of our eyes. I had thought this would take decades... I just hope that thanks to large companies like CloudFlare buying large numbers of ARM servers their prices finally start going down to a reasonable level.
- sigi45 9y agoI don't think so. I'm guessing that the xeons feature set is not used at all and this specific payload is just much better on that arm. There is a big chance, that whatever cloudflare is doing on that machine, is massive parallel and simple.
- Arnt 9y agoThese nine specific payloads, if you take the tweet's words literally.
- thecompilr 9y agoThose are indeed workloads, specific for a webserver and even more specific for CF. Many other workloads that we don't care about that Intel can handily win of course. We don't have use for most Intel extensions, like AVX512 and barely use any AVX2. Intel is the king of HPC and single threaded performance for a few more years. Although ARM can potentially leapfrog Intel in the HPC space, if someone implements the Salable Vector Instruction extension in 1024 or 2048 bit.
- pdpi 9y agoMy understanding is that some large parts of the feature set aren't usable anyhow. E.g. there's enough of a ramp up/ramp down power cost for the vector units that you'd never want to compile your binaries with SSE/AVX turned on unless your workload makes near constant use of those units.
- sigi45 9y agoI have never heard that one before. Do you have any sources for that?
- Jenya_ 9y agohttps://www.quora.com/Why-are-AVX-instructions-so-power-hungry https://www.quora.com/Why-are-AVX-instructions-so-power-hung... https://news.ycombinator.com/item?id=13925039 https://news.ycombinator.com/item?id=13925039
- pdpi 9y agoThanks for the quotes. Apparently I slightly misremembered the problem. From the second one, this is what I was thinking about: https://news.ycombinator.com/item?id=13926839 https://news.ycombinator.com/item?id=13926839
- znpy 9y ago> I had thought this would take decades... Well it took decades...
- yellowapple 9y ago"I had thought this would take decades.." It is taking decades at this rate. Or at least a decade. Not to long ago when Intel was yanking the rug out from under the SPARC and POWER and MIPS and $RANDOM_PROPRIETARY_MAINFRAME_CISC datacenter marketshares.
- thisisit 9y agoAny ideas on what is the performance Cavium Thunder X2 CPUs?
- jazoom 9y agoFYI: "jump the shark" has a completely different meaning to "jump the gun", which is what you wanted to say.
- ksec 9y agoAt $8K savings per server, that is 8M if you have 1000 server at Cloudflare. Of course you have to invest a lot of people into software and testing as well. As well as many other time consuming things that were likely standard on Intel Platform. Still, those wont cost anywhere close to 8M. To be honest I am very much surprised at how cheap Qualcomm are selling their chip. This is going to be the start of avalanche. Once the software improves and make a self sustainable ecosystem, everything else will come. I still dont believe Qualcomm could achieve their 5% Server target by 2020 goal. But it is still a good start in longer terms success. What is more worrying is Intel's growth and Cash Cow is now at risk in the mid term. It will be interesting to see how Intel react to this. There is serious price pressure from AMD EPYC and Qualcomm Centriq. Intel DC 2018 may ship lower unit YoY and lower revenue / unit YoY. PC market is being attacked by AMD, while not at risk, Intel is definitely earning less as they increase core count. Personally I would have liked to see POWER9 or RISC-V in server space, but both still missing in action.
- Algent 9y agoSince we are speaking about CF, they usually a do lot of crypto. Which, unless I'm wrong, is accelerated by a lot on Intel CPUs. I'm really curious at what the blog post will reveal on this.
- Deathmax 9y agoThere is a blog post from CF a few months back with benchmarks. https://blog.cloudflare.com/arm-takes-wing/ https://blog.cloudflare.com/arm-takes-wing/
- deleted 9y ago[deleted]
- thecompilr 9y agoEncryption is one thing Intel has a lead in, both in single thread, and as a platform. But even there QC is sufficiently fast, and crypto although extensively used actually uses very little CPU. https://blog.cloudflare.com/how-expensive-is-crypto-anyway/ https://blog.cloudflare.com/how-expensive-is-crypto-anyway/
- walrus01 9y agoMid priced xeons: all sorts of motherboards available from eight different Taiwan based OEMs. AMD: I need some threadripper or server Zen motherboards, again there's 4+ OEMs with many designs. Both amd and Intel: all sorts of stuff available that meets open compute platform dimensions for motherboards. Or for ordinary 1U servers, or atx size boards that can be mounted in DIY blade enclosures. ARM: where are the motherboards I can buy right now in small to low quantities, and what are their specs/price? Who makes them? It I want to order 40 or 400 tomorrow... where do I go? Edit: I just spent ten minutes googling for arm centriq platform motherboards and cannot find a single product I can actually buy. As compared to how easily I could start the process of ordering a hundred 1U xeon based servers in the next half hour. At a massive scale such as cloudflare, sure you can afford to custom order thousands of motherboards. But for a lot of operators who are not nearly as big as them, the existing economies of scale in the x86-64 platform are a greater advantage. Right now I'd be willing to bet that dual socket current generation xeon motherboards are manufactured in much greater numbers, as much as a ratio of 500:1, compared to ARM server motherboards. Being able to mix and match whitebox parts from Taiwan and China sources to DIY bare metal blade hypervisors is a huge advantage.
- Aissen 9y agoAll true. This is temporary, when(if?) this gets more traction, you'll find OEMs and distributors having what you want.
- walrus01 9y agoMany years ago there was an attempt to get third party manufacturers to start building motherboards for the IBM power4 architecture. Never came to fruition.
- yellowapple 9y agoThose attempts are still happening in this day and age of POWER8 and beyond.
- majewsky 9y ago
- robeastham 9y agoSaw that the Centriq 2434 (40 cores) is listed at RRP $888. Does anyone know if you can get hold of a Centriq cpu & compatiple motherboard to build a local server or two? I fancy one in our lab rack to play around with. I'm assuming that this is not possible and that they are currently just for datacentre partners who make a large order. But would be great to hear otherwise.
- chomp 9y agoIf you ask your distributor rep, they should be able to procure a single unit or two for testing. We did this when Calxeda was a thing.
- exikyut 9y agoThis doesn't answer the supply chain question, but this is the info page on the chip: https://www.qualcomm.com/products/qualcomm-centriq-2400-processor https://www.qualcomm.com/products/qualcomm-centriq-2400-proc...
- ballen 9y agoAsk your HPE or SuperMicro reps.
- bjackman 9y agoNot Centriq but I think there are boards with Cavium ARM chips that you can buy as a consumer (can't find link to cite so may be wrong, but may be a useful avenue for your search).
- jbarham 9y agoWill be even more interesting when RISC-V becomes a contender in the HP space.
- ZirconiumX 9y agoI don't know why this is being downvoted. One of the workloads that Cloudflare does is encryption, and RISC-V has been planning a vector extension that can be reconfigured, so that the jump from 128-bit to 256-bit native execution is not an ISA change and doesn't require recompiling.
- imtringued 9y agoDoesn't every CPU manufacturer do this? AMD offers 256 bit vector instructions but they take twice as long because the Zen cores can only process 128 bit at a time.
- walrus01 9y agoFor a bit more on what the manufacturer reference design looks like: https://www.nextplatform.com/2017/12/06/deep-dive-qualcomms-centriq-arm-server-ecosystem/ https://www.nextplatform.com/2017/12/06/deep-dive-qualcomms-...
- jakozaur 9y agoNot surprised given: 1. A lot of cloud services are abstracting CPU architecture. E.g. you don't care on which CPU AWS S3 runs. 2. ARM is using less power than x86. 3. ARM license fees are so low comparing to Intel high gross margins. Whole mobile world runs on ARMs and little profit is made on CPUs there comparing to Intel cash cow server chips.
- stephenr 9y agoPoint 1 is not really related to “the cloud”. If you use a storage apppiance - be it a local NAS or S3 or B2 - you likely aren’t interested in the cpu running it. A better comparison would be if you didn’t care what cpu Amazon Lambda ran on. Which, maybe you don’t, if you use it with an interpreted language.
- ofrzeta 9y ago1. Is it? Actually I do not want to recompile all software. Did you try running Docker or Kubernetes with ARM? It's there but there's not a whole lot of software/images you can run.
- exikyut 9y agoSoftware follows hardware. Software is just commentary, particularly throwaway software.