9 ms·
Servers for an Accelerated Future
- rasengan 7y agoThis major shift is well received by the tech community and appears to be an industry wide movement [1]. Looks like meltdown [2] and spectre [3] have run their course. Bravo to Cloudflare! [1] https://www.amd.com/system/files/documents/linode-case-study.pdf https://www.amd.com/system/files/documents/linode-case-study... [2] https://en.m.wikipedia.org/wiki/Meltdown_(security_vulnerability) https://en.m.wikipedia.org/wiki/Meltdown_(security_vulnerabi... [3] https://en.m.wikipedia.org/wiki/Spectre_(security_vulnerability) https://en.m.wikipedia.org/wiki/Spectre_(security_vulnerabil...
- wbl 7y ago(Disclosure: I work at Cloudflare but wasn't involved in this decision) Spectre variant 1 still affects patched chips with any sort of speculation and the price of fixing via compiler is substantial. It's a multi vendor structural issue. The hardware industry is stuck with some very unpalatable decisions to fix this. In some cases there are tricks to protect particular arrays but it's very finicky especially if arrays aren't a power of two in length.
- britmob 7y agoI wonder when these issues will be far behind us... I can’t imagine it’s very near.
- ksec 7y ago>This major shift..... Intel is still selling as many Xeon as they can make. And AMD's EPYC 2 / Datacenter revenue hasn't move up much relatively speaking. Not only is it not major by any means, I wouldn't even agree on the word shift. And that is speaking as someone who really want AMD to do better.
- zoonosis 7y agoFrom 4Q2017 to 4Q2019 AMD's server market share increased from 0.8% to 4.5%[1]. [1] https://www.tomshardware.com/news/amd-vs-intel-cpu-market-share-q4-2019-epyc-and-ryzen-growth-decelerate-mobile-ryzen-up https://www.tomshardware.com/news/amd-vs-intel-cpu-market-sh...
- jagger27 7y ago> We selected the AMD EPYC 7642 processor in a single-socket configuration for Gen X. This CPU has 48-cores (96 threads), a base clock speed of 2.4 GHz, and an L3 cache of 256 MB. While the rated power (225W) may seem high, it is lower than the combined TDP in our Gen 9 servers and we preferred the performance of this CPU over lower power variants. Despite AMD offering a higher core count option with 64-cores, the performance gains for our software stack and usage weren’t compelling enough. I find this a bit puzzling for density reasons. I can definitely appreciate the clock speed benefits. One 64-core part (AMD EPYC 7742) has the same TDP of 225W, so power should be in the same ballpark. There's also lower clocked 64-core SKUs with 200W TDP. I can't imagine price would be major factor for a company of Cloudflare's size, but it's definitely true that the 48-core part is much cheaper. There's also the 7H12 with a higher base clock than the 48-core part, but its TDP is 280W. All of these EPYC chips have the same monstrous 256MB of L3, so maybe part of Cloudflare's workloads maxes out the cache before being able to feed all 64 cores, but that's a bit wishywashy. Maybe since they also all have the same PCIe lane capacity 48 cores is the sweet spot. The 64-core parts still seem like a nobrainer.
- o_x 7y agoYeah, I suspect it might be a memory throughput-per-core issue or one of the oldest ones in the books: they got a better deal for 48 cores as not all chiplet cores need to be operational...
- deleted 7y ago[deleted]
- jandrewrogers 7y agoAt least in my experience, for highly optimized code (which I imagine their's is), above certain core densities you run out of memory bandwidth, PCIe lanes, etc before you run out of cores unless your workload is particularly compute intensive. Power consumption also needs to be modeled into the hardware cost, so higher TDP does matter. It is a cost optimization problem, and those 64-core parts may offer little marginal gains for the added cost if bottlenecks are elsewhere in the system. My experience buying CPUs for data intensive servers has typically been that maximizing the performance-density-cost curve often recommends a mid-range number of cores at a lower-middle clock rate. These CPUs are inexpensive relative to their product families while still having enough horses to drive your memory, PCIe, etc to saturation with highly optimized code. Just enough resources, but no more.
- chadmeister 7y ago"Notably, for the first time, Intel is not inside. We are not using their hardware for any major server components such as the CPU, board, memory, storage, network interface card (or any type of accelerator)."
- Proven 7y agoWho cares? What matters is price performance. They need to deliver value. What's inside, I don't care.
- oh_sigh 7y agoIn the next paragraph from the article > This time, AMD is inside. We were particularly impressed by the 2nd Gen AMD EPYC processors because they proved to be far more efficient for our customers’ workloads. Since the pendulum of technology leadership swings back and forth between providers, we wouldn’t be surprised if that changes over time. However, we were happy to adapt quickly to the components that made the most sense for us.
- deleted 7y ago[deleted]
- tyingq 7y agoCloudflare's workers and kv store are pretty popular. I wonder if they ever plan to expand that to include more cloud compute. The article seems to indicate they have a lot of the needed unpinnings already.
- dx034 7y agoI'd guess they'd need to provision massively more to offer compute. They could offer it for some major data centres but I don't believe offering compute at all data centres would be feasible.
- a-ve 7y agohttps://blog-cloudflare-com-assets.storage.googleapis.com/2020/02/AMD_EPYC-35.jpg https://blog-cloudflare-com-assets.storage.googleapis.com/20... Looks like they're using Open Compute Project inspired designs, looking at those high-visibility green screws.
- Hardops 7y agohey, this is Rami .. I lead hw team in Cloudflare. Great observation! these chassis are designed for 19" racks, not OCP spec. But, who does not love green thumb screws!
- tbyehl 7y agoLooks like an OCP mezzanine interface tho. If a server can take OCP mezz cards and the BMC can be managed in the same manner and with the same tools as a fully OCP-spec system, does it really matter that it's not in an OCP-spec chassis and rack?
- Hardops 7y agoYes, the chassis form factor does not matter much to us at this point. Yes, we're using OCP Nic ... and many other components that are used by other servers/clouds. Nothing proprietary except our SW stack.
- jcdmacleod 7y agoCan I ask the chassis and motherboard combination? I know previous generations have been based on QCT chassis and motherboard components. I am curious what the non-hotswap, shallow depth, 1S is as it probably saves a fair amount of money dropping the HPC/density and hot swap premiums.
- Hardops 7y agoWe're still using ODM built servers that are available for the public. Yes, moving off the shared infra chassis helps us improve reliability & cost thru simplicity, and enables us to scale our infrastructure using single nodes vs. X nodes as the smallest scaling unit. Our machines are stateless as our globally interconnected infra is widely distributed and replicated. Hence we moved to no-hotplug.
- nodesocket 7y agoAre they running 2x 25Gbps NIC’s per server over standard cat 6? I thought 10Gbps over cat 6 was generally accepted as the fastest it can reliably push?
- etcet 7y agoIt looks like they're using DAC: https://blog-cloudflare-com-assets.storage.googleapis.com/2020/02/AMD2.png https://blog-cloudflare-com-assets.storage.googleapis.com/20...
- Hardops 7y agoThis is Rami from Cloudflare ... we actually use DAC Cables w/ SFP+ for all of our 25G nics.
- j-walker 7y agoThe orange cables in this pic (https://blog-cloudflare-com-assets.storage.googleapis.com/2020/02/AMD2.png https://blog-cloudflare-com-assets.storage.googleapis.com/20...) are probably Cat5/5e/6 and are for management.
- markroseman 7y agoImmense kudos for the Douglas Coupland reference.
- bluedino 7y ago>> For storage, we continue to have ~3TB, but moved to 3x1TB form factor using NVME flash Are they not using additional storage for caching? I also kind of expected some GPU's for some tasks.
- sudhirj 7y agoWhat cache tasks are possible on GPUs? Maybe computing a hash, but that doesn’t seem like a big problem.
- zoonosis 7y agoI think the question about caching was separate from the question about GPUs. I don't think it was meant to imply GPUs for caching.
- wmf 7y agoI could imagine CloudFlare using GPUs for image optimization or video encoding. They do a lot more than caching these days.
- elithrar 7y agoNeither is “great” on a GPU: • Image optimization is still mostly software: JPEG, PNG, WebP. GPUs do not bring much to the table here at all, and consume a lot of power, which is likely to upset the hosting ISP of that cache node. • Video encoding is better, but still average. NVENC (Nvidia’s encode Engine) has some quality challenges and limitations. Further, doing this at the edge makes less sense: if you have to fetch the source material from the origin, do the heavy lifting centrally and store the results at the edge. Storing 4-5 renditions is cheap compared to GPU running costs here.
- Hardops 7y agoNo GPUs, yet. Yes, the 3TB is 2/3 used for Caching, and 1/3 as a local storage for our SSL/DNS/Firewall/etc.
- 7y ago
- DenseComet 7y agoIn the last picture, the server seems to be 1U, but also squarish. Is there a benefit or a reason for not using the entire depth of the rack, or was there just nothing more to include?
- porker 7y agoWith the LUKS full disk encryption, how are you managing the encryption keys? I assume the keys are not password protected so the server automatically decrypts on boot?
- rajnathani 7y ago> For memory, we continued to use 256GB of RAM, as in our prior generation, but rated higher at 2933MHz. Is ECC memory being used? Also, great write up! The recent benchmark numbers from Netflix [0] comparing dual socket Intel Xeons with AMD EPYC Rome (64 cores) also showed really impressive results! [0] https://news.ycombinator.com/item?id=22229431 https://news.ycombinator.com/item?id=22229431