12 ms·

Why Cloudflare Chose AMD EPYC for Gen X Servers

284 points by jgrahamc 7y ago

bluedino 7y ago
Title is "Technical Details of Why Cloudflare Chose AMD EPYC for Gen X Servers"
- justin66 7y ago
  The intriguing thing is that the guy who submitted the story is the Cloudflare CTO...
- PhantomGremlin 7y ago
  Blog post is by Nitin Rao, Head of Global Infrastructure at Cloudflare. Submission and editorialized title change is by John Graham-Cumming, CTO at Cloudflare. Not sure what to make of that.
  - sudhirj 7y ago
    jgrahamc is more active on HN, and he seems to keep tabs on all Cloudflare discussions and answers questions when asked. Don't see what's odd about it. Haven't noticed Nitin here.
    - Thorrez 7y ago
      It's not odd that he submitted it. It's odd that he changed the title, given the HN guidelines against changing titles. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- dang 7y ago
  We've reverted the title to that, in accordance with the site guidelines. Submitted title was "Why Cloudflare abandoned Intel for AMD for its latest servers".
jasode 7y ago
Yes, there are lots of stories of individuals and companies choosing AMD over Intel. For me, what's more interesting would be any contrarian view listing the compelling reasons to pay the higher price for Intel instead of AMD. Off the top my head, some reasons would be: - Intel AVX-512 instruction set - Intel 4 and 8 sockets What realistic workloads have a better performance/cost advantage on Intel vs AMD EPYC? If you're a home user (gamer, programmer), is there any reason to buy Intel today? I'm about to buy another motherboard & CPU and considering AMD for the first time. In the past, I avoided it because AMD had an incompatibility (e.g. VMware) that kept me on Intel. (E.g. previous example of AMD showstopping bug affecting VMware: https://communities.vmware.com/thread/456094 https://communities.vmware.com/thread/456094)
- pedrocr 7y ago
  > If you're a home user (gamer, programmer), is there any reason to buy Intel today? There's definitely not much. I believe absolute peak single-core performance can still be slightly higher on Intel which is important for gaming, but maybe the last few AMD chips also killed that. And there are also some tools that are not well optimized or don't yet fully support AMD yet. For example: https://github.com/mozilla/rr/issues/2034 https://github.com/mozilla/rr/issues/2034 This year's release of Zen 3 looks very promising. Hopefully RAM manufacturers wake up and we also get a bunch more ECC options that aren't so overpriced and slow.
  - MrStonedOne 7y ago
    > There's definitely not much. I believe absolute peak single-core performance can still be slightly higher on Intel https://www.cpubenchmark.net/singleThread.html https://www.cpubenchmark.net/singleThread.html It's still weird to see amd anywhere near the top of this list, a few years ago the first few screenfulls were intel.
    - pedrocr 7y ago
      >>but maybe the last few AMD chips also killed that. It's definitely quite close but PassMark just shows that it's a tie and it's not necessarily representative of game workloads. It all depends on how much overclocking headroom there is. The enthusiasts for this kind of thing will throw great cooling at these chips and get the best out of them. I wouldn't be surprised if the advantage AMD now has in fabrication meant it also won there though.
MikusR 7y ago
Two years ago they switched to ARM https://news.ycombinator.com/item?id=16646701 https://news.ycombinator.com/item?id=16646701
- jgrahamc 7y ago
  We tried to switch to ARM. We maintain our ARM port of all our code for the day there are servers we can use. However, we are in production with AMD now. From one of the posts: "Readers of our blog might remember our excitement around ARM processors. We even ported the entirety of our software stack to run on ARM, just as it does with x86, and have been maintaining that ever since even though it calls for slightly more work for our software engineering teams. We did this leading up to the launch of Qualcomm’s Centriq server CPU, which eventually got shuttered. While none of the off-the-shelf ARM CPUs available this moment are interesting to us, we remain optimistic about high core count offerings launching in 2020 and beyond, and look forward to a day when our servers are a mix of x86 (Intel and AMD) and ARM."
  - MikusR 7y ago
    In two years followed by "Why we are switching from Intel to RISC-V" and two years after that "Why we are switching from Intel to POWER".
    - myself248 7y ago
      No love for itanium? Aww.
      - colejohnson66 7y ago
        I know this is sarcasm, but: I personally never used Itanium, but the features it had at the time were very compelling. Unfortunately, it was too ambitious, and now it’s faster to do OOO execution instead of making the compiler do all the heavy lifting.
        7y ago
lukevp 7y ago
I’m a big fan of CloudFlare. I own a little stock, and I moved our company over to it after running on a competitor for about a year. CloudFlare has been much better. Whenever I use the front end, it just feels like it’s made by people who actually need to use this stuff, and it is so straightforward. The api is great too. When we were with the competition it was ridiculous how often we had to have an engineer assist us with setup because the front end didn’t support some feature we needed. With CF we signed up one day and have never had to have manual changes made by support, and the health checks, load balancing and failover have worked from day one. With the competitor the health check story did not live up to the reality and we had several outages related to the failover system not recovering properly. When they publish these blog posts, it shows to me that they are willing to consider bold moves and that they aren’t afraid to be different. Also, that their architecture has been designed with this agility in mind. I’ve visited their offices and while their corporate culture didn’t seem like the best fit for me, I liked the team and what they are doing.
- bmurphy1976 7y ago
  Come on, throw their competitor under the bus! I'm guessing it was Akamai. Every change we need to make to our Akamai distros is an up sell conversation with them. It's obnoxious. I much prefer Amazon CloudFront.
  - lukevp 7y ago
    Not Akamai but similar experience here. I've heard good things about CloudFront, but I believe it's mostly on the CDN side? Does it do things like authenticated origin pulls (client cert from provider presented to your origin), automatic SSL certs, health checks w/ failover, DDoS and WAF protection and unlimited bandwidth? The WAF was one of the reasons we switched, our existing provider wanted to charge us 10x more just for the WAF than the entire Cloudflare plan is now.
    - AznHisoka 7y ago
      Fastly?
ddevault 7y ago
SourceHut runs its performance critical infrastructure (git, builds) on AMD EPYC. These CPUs are sweeeet. The hard part is sourcing motherboards. I currently buy them from some dude in Germany on eBay because he's the only consistent source I've been able to find.
- fierarul 7y ago
  > SourceHut runs its performance critical infrastructure (git, builds) on AMD EPYC. These CPUs are sweeeet. The hard part is sourcing motherboards. I currently buy them from some dude in Germany on eBay because he's the only consistent source I've been able to find. Heh, you know that guy is with / getting a visit from the German BDN now.
  - sauerbraten 7y ago
    *BND, from Bundesnachrichtendienst
    - bithavoc 7y ago
      Why, is it illegal?
aloknnikhil 7y ago
I really wish AMD would focus on its tooling. The Intel VTune Amplifier is a fantastic multi-platform profiler that can help understand how effectively your software is using the hardware (Pipelining, micro-ops efficiency, cache usage, etc.). If AMD could come up with something similar, it'd make their offering a no-brainer. > https://software.intel.com/en-us/vtune https://software.intel.com/en-us/vtune
- wjnc 7y ago
  I think (without too much knowledge in this field) that AMD has had to play catch-up for so long in silicon that now the catch-up is complete, it will take some time to recognise organisationally that more focus on non-silicon is overdue. Silicon lives in an ecosphere, but without competitive products there's little use in investing in that ecosphere. If you suddenly get there, you need to change your game quite drastically.
  - stingraycharles 7y ago
    But wasn’t the same the case that last time they “disrupted” Intel with their amd64 and dual core CPUs? They caught up, but never really invested in the tooling. And then they lost market share again. Why would this time be different?
    - SkyMarshal 7y ago
      Different leadership this time, maybe the new crew understands this better.
    - c0nfused 7y ago
      There was the whole ftc sued Intel for anti competitive behavior thing last time where they locked oems out of buying Intel and AMD. Hopefully without that things will be more diverse in the market going forward. https://www.cnet.com/news/intel-to-pay-amd-1-25-billion-in-antitrust-settlement/ https://www.cnet.com/news/intel-to-pay-amd-1-25-billion-in-a... Lll https://www.ftc.gov/news-events/press-releases/2010/08/ftc-settles-charges-anticompetitive-conduct-against-intel https://www.ftc.gov/news-events/press-releases/2010/08/ftc-s...
gdm85 7y ago
> Notably, Intel is not inside. I can hear the loud silence of that full stop.
KaiserPro 7y ago
At FAAG its well known that some AMD procs are faster, cheaper and cooler(well higher performance per watt). None of that is in doubt, what they are having trouble with is supply. It appears that AMD are not able to provide either the support (microcode tweaks, firmware heads up, firmware testing and bug reporting) or the volume to be useful. The conclusion was that AMD want to nail consumer first.
abriosi 7y ago
Let's put it this way. I'd rather program single core applications on my i5-4690k than any other ryzen series chips including threadrippers with 12+ cores. I have used two threadrippers for more than 6 months. Linux still runs faster on i5-4690k which is 6 years old
- undersuit 7y ago
  Odd, I switched from that processor to the AMD 2700 last month and I'm enjoying the legroom.
iRobbery 7y ago
Now i wonder if these were considered/known https://www.servethehome.com/ampere-altra-80-arm-cores-for-cloud/ https://www.servethehome.com/ampere-altra-80-arm-cores-for-c...

284 points by jgrahamc 7y ago

bluedino 7y ago

Title is "Technical Details of Why Cloudflare Chose AMD EPYC for Gen X Servers"

justin66 7y ago
The intriguing thing is that the guy who submitted the story is the Cloudflare CTO...
PhantomGremlin 7y ago
Blog post is by Nitin Rao, Head of Global Infrastructure at Cloudflare. Submission and editorialized title change is by John Graham-Cumming, CTO at Cloudflare. Not sure what to make of that.
- sudhirj 7y ago
  jgrahamc is more active on HN, and he seems to keep tabs on all Cloudflare discussions and answers questions when asked. Don't see what's odd about it. Haven't noticed Nitin here.
  - Thorrez 7y ago
    It's not odd that he submitted it. It's odd that he changed the title, given the HN guidelines against changing titles. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
dang 7y ago
We've reverted the title to that, in accordance with the site guidelines. Submitted title was "Why Cloudflare abandoned Intel for AMD for its latest servers".

jasode 7y ago

Yes, there are lots of stories of individuals and companies choosing AMD over Intel. For me, what's more interesting would be any contrarian view listing the compelling reasons to pay the higher price for Intel instead of AMD. Off the top my head, some reasons would be: - Intel AVX-512 instruction set - Intel 4 and 8 sockets What realistic workloads have a better performance/cost advantage on Intel vs AMD EPYC? If you're a home user (gamer, programmer), is there any reason to buy Intel today? I'm about to buy another motherboard & CPU and considering AMD for the first time. In the past, I avoided it because AMD had an incompatibility (e.g. VMware) that kept me on Intel. (E.g. previous example of AMD showstopping bug affecting VMware: https://communities.vmware.com/thread/456094 https://communities.vmware.com/thread/456094)

pedrocr 7y ago
> If you're a home user (gamer, programmer), is there any reason to buy Intel today? There's definitely not much. I believe absolute peak single-core performance can still be slightly higher on Intel which is important for gaming, but maybe the last few AMD chips also killed that. And there are also some tools that are not well optimized or don't yet fully support AMD yet. For example: https://github.com/mozilla/rr/issues/2034 https://github.com/mozilla/rr/issues/2034 This year's release of Zen 3 looks very promising. Hopefully RAM manufacturers wake up and we also get a bunch more ECC options that aren't so overpriced and slow.
- MrStonedOne 7y ago
  > There's definitely not much. I believe absolute peak single-core performance can still be slightly higher on Intel https://www.cpubenchmark.net/singleThread.html https://www.cpubenchmark.net/singleThread.html It's still weird to see amd anywhere near the top of this list, a few years ago the first few screenfulls were intel.
  - pedrocr 7y ago
    >>but maybe the last few AMD chips also killed that. It's definitely quite close but PassMark just shows that it's a tie and it's not necessarily representative of game workloads. It all depends on how much overclocking headroom there is. The enthusiasts for this kind of thing will throw great cooling at these chips and get the best out of them. I wouldn't be surprised if the advantage AMD now has in fabrication meant it also won there though.

MikusR 7y ago

Two years ago they switched to ARM https://news.ycombinator.com/item?id=16646701 https://news.ycombinator.com/item?id=16646701

jgrahamc 7y ago
We tried to switch to ARM. We maintain our ARM port of all our code for the day there are servers we can use. However, we are in production with AMD now. From one of the posts: "Readers of our blog might remember our excitement around ARM processors. We even ported the entirety of our software stack to run on ARM, just as it does with x86, and have been maintaining that ever since even though it calls for slightly more work for our software engineering teams. We did this leading up to the launch of Qualcomm’s Centriq server CPU, which eventually got shuttered. While none of the off-the-shelf ARM CPUs available this moment are interesting to us, we remain optimistic about high core count offerings launching in 2020 and beyond, and look forward to a day when our servers are a mix of x86 (Intel and AMD) and ARM."
- MikusR 7y ago
  In two years followed by "Why we are switching from Intel to RISC-V" and two years after that "Why we are switching from Intel to POWER".
  - myself248 7y ago
    No love for itanium? Aww.
    - colejohnson66 7y ago
      I know this is sarcasm, but: I personally never used Itanium, but the features it had at the time were very compelling. Unfortunately, it was too ambitious, and now it’s faster to do OOO execution instead of making the compiler do all the heavy lifting.
      - 7y ago

lukevp 7y ago

I’m a big fan of CloudFlare. I own a little stock, and I moved our company over to it after running on a competitor for about a year. CloudFlare has been much better. Whenever I use the front end, it just feels like it’s made by people who actually need to use this stuff, and it is so straightforward. The api is great too. When we were with the competition it was ridiculous how often we had to have an engineer assist us with setup because the front end didn’t support some feature we needed. With CF we signed up one day and have never had to have manual changes made by support, and the health checks, load balancing and failover have worked from day one. With the competitor the health check story did not live up to the reality and we had several outages related to the failover system not recovering properly. When they publish these blog posts, it shows to me that they are willing to consider bold moves and that they aren’t afraid to be different. Also, that their architecture has been designed with this agility in mind. I’ve visited their offices and while their corporate culture didn’t seem like the best fit for me, I liked the team and what they are doing.

bmurphy1976 7y ago
Come on, throw their competitor under the bus! I'm guessing it was Akamai. Every change we need to make to our Akamai distros is an up sell conversation with them. It's obnoxious. I much prefer Amazon CloudFront.
- lukevp 7y ago
  Not Akamai but similar experience here. I've heard good things about CloudFront, but I believe it's mostly on the CDN side? Does it do things like authenticated origin pulls (client cert from provider presented to your origin), automatic SSL certs, health checks w/ failover, DDoS and WAF protection and unlimited bandwidth? The WAF was one of the reasons we switched, our existing provider wanted to charge us 10x more just for the WAF than the entire Cloudflare plan is now.
  - AznHisoka 7y ago
    Fastly?

ddevault 7y ago

SourceHut runs its performance critical infrastructure (git, builds) on AMD EPYC. These CPUs are sweeeet. The hard part is sourcing motherboards. I currently buy them from some dude in Germany on eBay because he's the only consistent source I've been able to find.

fierarul 7y ago
> SourceHut runs its performance critical infrastructure (git, builds) on AMD EPYC. These CPUs are sweeeet. The hard part is sourcing motherboards. I currently buy them from some dude in Germany on eBay because he's the only consistent source I've been able to find. Heh, you know that guy is with / getting a visit from the German BDN now.
- sauerbraten 7y ago
  *BND, from Bundesnachrichtendienst
  - bithavoc 7y ago
    Why, is it illegal?

aloknnikhil 7y ago

I really wish AMD would focus on its tooling. The Intel VTune Amplifier is a fantastic multi-platform profiler that can help understand how effectively your software is using the hardware (Pipelining, micro-ops efficiency, cache usage, etc.). If AMD could come up with something similar, it'd make their offering a no-brainer. > https://software.intel.com/en-us/vtune https://software.intel.com/en-us/vtune

wjnc 7y ago
I think (without too much knowledge in this field) that AMD has had to play catch-up for so long in silicon that now the catch-up is complete, it will take some time to recognise organisationally that more focus on non-silicon is overdue. Silicon lives in an ecosphere, but without competitive products there's little use in investing in that ecosphere. If you suddenly get there, you need to change your game quite drastically.
- stingraycharles 7y ago
  But wasn’t the same the case that last time they “disrupted” Intel with their amd64 and dual core CPUs? They caught up, but never really invested in the tooling. And then they lost market share again. Why would this time be different?
  - SkyMarshal 7y ago
    Different leadership this time, maybe the new crew understands this better.
  - c0nfused 7y ago
    There was the whole ftc sued Intel for anti competitive behavior thing last time where they locked oems out of buying Intel and AMD. Hopefully without that things will be more diverse in the market going forward. https://www.cnet.com/news/intel-to-pay-amd-1-25-billion-in-antitrust-settlement/ https://www.cnet.com/news/intel-to-pay-amd-1-25-billion-in-a... Lll https://www.ftc.gov/news-events/press-releases/2010/08/ftc-settles-charges-anticompetitive-conduct-against-intel https://www.ftc.gov/news-events/press-releases/2010/08/ftc-s...

gdm85 7y ago

> Notably, Intel is not inside. I can hear the loud silence of that full stop.

KaiserPro 7y ago

At FAAG its well known that some AMD procs are faster, cheaper and cooler(well higher performance per watt). None of that is in doubt, what they are having trouble with is supply. It appears that AMD are not able to provide either the support (microcode tweaks, firmware heads up, firmware testing and bug reporting) or the volume to be useful. The conclusion was that AMD want to nail consumer first.

abriosi 7y ago

Let's put it this way. I'd rather program single core applications on my i5-4690k than any other ryzen series chips including threadrippers with 12+ cores. I have used two threadrippers for more than 6 months. Linux still runs faster on i5-4690k which is 6 years old

undersuit 7y ago
Odd, I switched from that processor to the AMD 2700 last month and I'm enjoying the legroom.

iRobbery 7y ago

Now i wonder if these were considered/known https://www.servethehome.com/ampere-altra-80-arm-cores-for-cloud/ https://www.servethehome.com/ampere-altra-80-arm-cores-for-c...