15 ms·
Intel vulnerabilities costing 25% CPU performance loss to a cloud provider
- samber 7y ago@waxzce: "FYI, as cloud provider we rawly loss around 25% of CPU performances the lasts 18 months due to different CVE and issues on CPU and mitigation limiting capacity using microcode, so we stuff more CPUs, but prices didn't go down at all... That's a kind of upselling. #IntelFail"
- aiCeivi9 7y agoIs it #IntelFail if Intel had supply issues in last year? It doesn't look like demand was affected by much.
- Traster 7y agoAny way you slice it it's an Intel Fail. They've basically had to lower the performance of their existing chips, they've failed on the road map for new chips, and they don't have capacity for more of the existing chips because they thought 14nm would be winding down by now rather than peak production.
- walrus01 7y agoDon't get married to Intel, if you need big beefy xen or kvm hypervisor machines, there's lots of good EPYC based motherboards.
- gcells 7y agoI am planning on making a home-server. Any suggestions for EPYC boards?
- cyphar 7y agoI'm setting up a home server and decided to just go with Ryzen. You get a bunch of server features (ECC RAM, lots of PCIE lanes, and lots of cores) for a lot less cash than the equivalent EPYC build. Obviously EPYC has a place but for the home usecase you could use a Ryzen as a substitution for Intel Xeons because of the baseline features of Ryzen.
- _JamesA_ 7y agoI just built my first AMD machine since the Athlon days with an ASRock Rack EPYCD8-2T [0] motherboard and an AMD EPYC 7281 CPU [1]. I upgraded a VMWare ESXi server hosting FreeNAS from a consumer motherboard with an Intel Core i7 3930K and it was completely painless. [0]: https://www.asrockrack.com/general/productdetail.asp?Model=EPYCD8-2T#Specifications https://www.asrockrack.com/general/productdetail.asp?Model=E... [1]: https://www.amd.com/en/products/cpu/amd-epyc-7281 https://www.amd.com/en/products/cpu/amd-epyc-7281
- gcells 7y agoI've used a lot of ASUS desktop boards and never have been disappointed. However I have heard that they stop pushing BIOS updates for older boards. Have you faces such an issue? My current HP home-server has been with me for ~10years
- mizzack 7y agoDepending on what you're using it for, I would take a look at Threadripper too. There are some crazy deals right now. If you don't IPMI and registered ECC, it's a great option for a ton of cores and PCIe lanes.
- kllrnohj 7y agoThreadripper and Ryzen both have ECC support.
- undersuit 7y ago'Registered ECC' is just slightly different than ECC memory, but it's definitely a feature of EPYC, and unsupported on Threadripper and Ryzen.
- kllrnohj 7y agoI cannot find anything at all that supports such a claim. Do you have any links or supporting spec sheets? All I can find are the same "Supports ECC: Yes" listing for Epyc, Threadripper, and Ryzen. The only spec differences I can find for Epyc vs. Threadripper on the memory side of things is 8-channel vs. 4-channel.
- mizzack 7y agoLook at specs for any X399 board and they will say something like "Supports Quad Channel DDR4 3600+(OC) & ECC UDIMM Memory" or "8 x DIMM, Max. 128GB, DDR4 3600(O.C.)... MHz ECC and non-ECC, Un-buffered Memory". Both omit buffered/registered DIMMs. AM4 boards have similar verbiage.
- undersuit 7y agoHere's a document from AMD about EPYC: https://developer.amd.com/wp-content/resources/56301_1.0.pdf https://developer.amd.com/wp-content/resources/56301_1.0.pdf They do not mention unregistered DIMMs being supported.
- kop316 7y agoIf you are running only open source software, you may want to consider this: https://www.raptorcs.com/ https://www.raptorcs.com/
- walrus01 7y agoStarting at $5800? For 5800 I can build one hell of a threadripper or epyc system with nvme storage. I've been seeing people try to make power architecture (ibm) servers a thing for 12+ years now, it never happens, because ordinary developers can't afford them. Compared to what you can put under your desk for a thousand bucks based on any Intel or amd, amd64 architecture system . Or this? https://www.raptorcs.com/content/TL2B01/intro.html https://www.raptorcs.com/content/TL2B01/intro.html $2700 and probably is outperformed by a $150 ryzen cpu on a $110 motherboard.
- nickpsecurity 7y ago"Starting at $5800? For 5800 I can build one hell of a threadripper or epyc system with nvme storage." Except that it won't have open firmware on a CPU hardly anyone targets. Whatever you build will probably be vulnerable. Unless you're running OpenBSD or something.
- TheOperator 7y agoI guess something being such terrible value nobody uses it is a sort of advantage...
- yjftsjthsd-h 7y agoLess used != terrible value.
- kop316 7y agohttps://www.phoronix.com/scan.php?page=article&item=power9-threadripper-core9&num=1 https://www.phoronix.com/scan.php?page=article&item=power9-t... In case you are curious to see benchmarks.
- oldjokes 7y agoIntel has been proven unreliable and untrustorthy time and time again, people have been complaining about how awful and unscalable x86 is since the 90s. The future is more likely ARM- I know a few companies are experimenting with moving their internal servers to ARM, and AWS started offering that platform a few months ago. But I'm not sure how long this takes, people have also been wrongly predicting the death of x86 since the 90s.
- jeltz 7y ago> If AMD was in a better shape, there is a real market momentup here. What does this comment refer to? As far as I know AMD is in pretty good shape.
- jasonvorhe 7y agoThey probably can't meet the demand for server-grade CPUs. But I'm just guessing.
- ksec 7y agoMost of the current AMD offering, APU, GPU, and CPU is still being fabbed with GF 14nm, which I believe to be very limited in capacity. Once the next generation GPU ( Navi ) and CPU ( Zen 2 ) along with EPYC 2 moves to TSMC 7nm they should be able to flex their muscles a lot more. I wished they have made the move earlier, like start of this year, but it is now all set for Q3.
- mtgx 7y agoAmd should probably keep production of whatever they think will sell +20% to account for whatever new Intel screwup is revealed.
- ksec 7y agoPure Speculation, judging from the slight delay ( 1Q ) in EPYC 2 introduction, my guess is that AMD wants the launch and fulfil all the demand of the big cloud provider ( AWS, Azure and Google ) while still have enough Chips for the rest of the market. The worst thing could happen is AMD announced their EPYC 2 and AWS getting all the stock with little left for other Channels. I have high expectation but I trust Dr Lisa Su and her executive team. And Intel is currently having a new low in my book.
- Someone1234 7y agoThey're a cloud provider, so cloud offerings. AMD has had some consumer success but remain rare in the date center.
- botto 7y agoIs this a chance for ARM providers to move in and outcompete Intel, especially if they can provide similar tech but with security? Basically an ARMs race?
- cyphar 7y agoI think it's more likely that AMD will pick up the slack.
- AnIdiotOnTheNet 7y agoYou can't stop this class of vulnerabilities without sacrificing performance because the reason for the performance is also the reason for the vulnerabilities. That's the choice you have, and no brand affiliation will save you.
- mfatica 7y agoIt's not about brand affiliation, AMD and Intel have vastly different implementations
- dfrage 7y agoWhich didn't save AMD from Spectre bugs. And everyone else with high performance speculative execution out-of-order designs also had Meltdown bugs, ARM and IBM, both POWER and mainframe/Z: https://en.wikipedia.org/wiki/Meltdown_(security_vulnerability) https://en.wikipedia.org/wiki/Meltdown_(security_vulnerabili...
- mda 7y agoYet, AMD looks way better considering all vulnerabilities reported so far. Why downplay this fact?
- Fnoord 7y agoNot only AMD. Also POWER, ARM. I don't know about RISC-V or MIPS. We heard the very same argument about Microsoft Windows back in the 90s and 00s. "Linux [1] just wasn't well tested." It is purely a hypothesis, without any proof whatsoever. Microsoft lost its dominance. Intel not yet, but this could lead to that. The question with losing dominance (e.g. being monopolist or market leader) is always a question of when, not if. And if it happens some people will lose a lot of money [2]. There's going to be a time where the USA is no longer #1 world power. The question isn't if, it is when. History teaches us this much. [1] (Whatever that means.) [2] (Or "money".)
- adrianN 7y agoWhy is nobody suing Intel? They sold defective chips.
- orliesaurus 7y agoNot voluntarily?
- adrianN 7y agoSo? If I buy a car that is promised to have 200HP but it actually just has 150HP after a software update wouldn't I be able to sue the manufacturer at least for the lost resale value?
- kllrnohj 7y agoPerhaps, but what performance did Intel actually promise that you'd be able to argue against?
- kazinator 7y agoIntel promises you better performance from a new processor than what you can get from one in the same class one that is several years old, also from Intel. You're paying for a new one, but it performs like something that is several years old that you can obtain much more cheaply. Another aspect of it is consumers who chose an Intel chip over a competitor's. Intel does promise competitive performance. Their pitch isn't "we are slower, but just buy us on the brand name alone".
- neogodless 7y agoHP is to Mhz what vehicle performance is to server performance. Intel's hardware still provides the the same MHz. So the analogy would be more like due to safety concerns, a software update to the car increases the braking used by traction control. That feels charitable to Intel, but unless they knew about the exploit, they used the tools in their arsenal to maximize performance. Now they are reducing it for safety. Maybe it's more like how auto manufacturers are now having to increase wind resistance by designing pedestrian friendly hoods. Obviously that's not "after purchase" but I can't think of a better analogy!
- sureaboutthis 7y agoPerhaps this is the reason for Intel's recent push on their self created version of Clear Linux?
- yjftsjthsd-h 7y agoHow does that help? You still take the perf hit?
- sureaboutthis 7y agoTests on Phoronix show Clear having significant wins in a number of tests. They have done so by rewriting libraries to perform well with Intel hardware. Those who complain about a 25% performance hit elsewhere can now be shown pretty good performance with Clear that isn't attainable with other distros.
- yjftsjthsd-h 7y agoOh, gotcha; boost from optimizations cancelling out against loss from patches. I would argue that that still constitutes a performance loss since before the patches you could run with just the improvements get better performance rather than neutral. But you're not wrong.
- exabrial 7y agoWhat are the possible pitfalls of _not_ runnning spectre et all mitigations, if you're _not_ hosting other people's code, but on your own hardware?
- Filligree 7y agoNot much. We've yet to see any live Spectre exploits (that I know of), and I'd be surprised if the first one is a remote exploit.
- londons_explore 7y agoGood point... I wonder why? The proof of concept code works so well for some variants.
- kllrnohj 7y agoBecause most channels for delivering untrusted code have already been fixed. There's really only 2 channels for delivering code these days to attack consumers - app stores & web browsers. Web browsers had crude mitigations immediately (disable SharedArrayBuffer), and have improved since then (full site isolation https://v8.dev/blog/spectre https://v8.dev/blog/spectre ) As for app stores there's typically easier malware attack vectors if you manage to get an app installed & past the scanners in place with that typical back & forth escalation game. And those run in process-level sandboxes with clamped down syscalls. You can't exploit a spectre BPF attack if you can't use BPF at all due to selinux rules, for example. Moving past consumer what else are you really going to attack? Sure you could rent some virtual hosting somewhere and hope you get lucky attacking a neighboring VM, but what's really your end goal there to justify the time & risk? It's not like you can reliably pick a target with that, you just have to go fishing and react to what you may or may not find.
- Symmetry 7y agoI can see how a local Meltdown exploit would be useful but what would be the point of a local Spectre exploit?
- 7y ago
- userbinator 7y agoIt's a little funny to see the "use AMD!" comments --- since from what I understand, Intel's optimisations that lead to these side-channels are specifically for performance, so using AMD instead of Intel might mean the same amount performance loss.
- mfatica 7y agoAMD is in a better position here
- scotty79 7y agoAmd was walways better wit bang for buck. Intel just had the fastest CPUs for rich who nwed best single core performance. If Intel doesn't have the fastest CPUs then AMD is just hands down cheaper and Intel has no advantage.
- Kye 7y agoAlternate perspective: AMD not doing these risky optimizations reflects a better engineering culture, or at least less risky priorities. Go with Intel and risk losing a huge chunk of your performance some day, or go with AMD and know what to expect. Different people and companies have different risk tolerances, so it's not an obvious choice one way or the other.
- PedroBatista 7y agoCurrent AMD offerings are generally beating Intel, and in the price-to-performance game it's a complete slaughter.
- mda 7y agoThey have completely different architectures, some optimizations can be done in safer ways.
- dfrage 7y agoIsn't one of the "secrets" to AMD's post-486 success adopting the general approach the Pentium Pro took along with others in the 1990s, which in turn are based on IBM's 1960s Tomasulo out-of-order algorithm adding speculative execution? This general approach made all out-of-order speculative designs, including ARM, POWER and IBM Z vulnerable to Spectre, so I submit they're not "completely different".
- morrbo 7y agoDoes anyone actually have some benchmarks of the latest gen AMD vs the latest gen intel processors with all mitigations for spectre, meltdown, and the 10 other sidechannel/speculative execution vulnerabilities applied? I'd genuinely be curious to find out what the eventual results are because as i understand it AMD is not too far away from Intel as a standalone processor, surely in a "real world" scenario they'd be significantly faster?
- Someone1234 7y agoIt is a moving target. A lot of "day 1" patches have absolutely tanked performance only to regain performance later via smarter mitigations. Both Linux and Windows as well as microcode updates have all seen some of the performance loss regained. And while it would be nice to call any point "the end" and measure then, as of two weeks ago they were still finding additional vulnerabilities and patching them. So if there is an end we aren't there yet. That all being said, I am also curious on what the results would show. Just very hard to pin down.
- rincebrain 7y agoTo be fair, at least Microsoft explicitly went with finding perf optimizations elsewhere to reduce the impact, not smarter mitigations (though they also eventually did that when they decided to go for retpoline).
- noir_lord 7y agoOn multithreaded stuff AMD is killing it, they where slightly behind on IPC about 5% but the clocks are typically a bit lower. I'd expect with both machines patched that clock for clock AMD are slightly ahead. I've got a 2700X at home and it's a monster and I'd Zen2 is on the conservative end of the leaks/rumours it's going to be a total killer.
- bitL 7y agoSeems like Threadripper 3 might be delayed until 2020, so no 64c HEDT for a while...
- mtgx 7y agoI bet Google isn't so giddy now about being FIRST!! [1] with Skylake in the data center a couple of years ago, or "going on all-in" with Intel on the Chromebooks (it didn't even give AMD a chance until very recently...), despite Chrome OS being one of the very few operating systems that are truly architecture agnostic. Now it's paying dearly for that mistake, with up to 40% performance loss on Chromebooks due to the disabling of HT: https://www.techrepublic.com/article/mds-vulnerabilities-lead-chrome-os-74-to-disable-hyper-threading/ https://www.techrepublic.com/article/mds-vulnerabilities-lea... Google broke one of the most basic business rules: never rely on a single supplier. You're always worse off in the end, even if the exclusivity deals seem very tempting in the short-term. [1] https://cloud.google.com/blog/products/gcp/compute-engine-updates-bring-skylake-ga-extended-memory-and-more-vm-flexibility https://cloud.google.com/blog/products/gcp/compute-engine-up...
- yjftsjthsd-h 7y ago> "going on all-in" with Intel on the Chromebooks Weren't Chromebooks on ARM before x86? > Google broke one of the most basic business rules: never rely on a single supplier. You're always worse off in the end, even if the exclusivity deals seem very tempting in the short-term. I don't think this is universally true, although I agree that it's probably prudent. Diverse options/suppliers are a risk mitigation, but they do have a cost.
- qaq 7y agoIntel vulnerabilities are providing additional 25% revenue to Cloud Providers.
- bufferoverflow 7y agoTo Intel maybe. It's a loss to the cloud providers and their customers.
- iopq 7y agoI pay by the hour, if I need to provision more servers AWS doesn't take a hit, I do
- sokoloff 7y agoOn EC2 and Lambda and similar, yes. On S3 and other managed services, AWS takes the hit.
- rdsubhas 7y agoThey could very well optimize only for their own service costs, rather than for the customer.
- srrr 7y agoDo they really need to apply all mitigation patches if they only execute their own code on machines running S3 and other managed services? (For EC2/... of course they do because they execute customer code.) I wonder if I can skip the performance penalty for my own (hardware) server too... But I am not smart enough to answer this question myself.
- msbarnett 7y agoDo you run everything on the machines you only execute your own code on as root? Most people don’t. It’s a defence in depth thing — if someone hits an exploit in a service you want it to be running the lowest privileged user it can get away with so that it can’t rapidly pull all the data out of other subsystems, roam out into the local network and dump DBs, etc. With these vulnerabilities unpatched, an attacker running code as the low privilege user account can sniff credentials for other accounts on the machine, or underlying host, private keys that authenticate against this machine and therefore probably others in the network, etc. I’d say it increases risk quite a bit.
- imhoguy 7y agoNow how much this 25% is in terms of CO2 emissions?
- the-dude 7y agoAbout 25%.
- kierank 7y agoThis is an argument for organisations to go back to being on-prem, running their non-public facing workloads privately.
- nixgeek 7y agoHow's that an improvement over using 'Dedicated Instances' or the 'Bare Metal' offerings which exist with cloud providers, where you're not colocated with other entities but still enjoy all the benefits of how the infrastructure is managed, flexibility in upscale/downscale (instead of acquiring capital assets which depreciate over 36-48 months).
- kierank 7y agoExternal access to the bare metal is still possible and thus information can be leaked via Meltdown and Spectre
- LinuxBender 7y agoWe are doing perf testing of the AMD versions of HP/Dell servers. We may tech refresh about 40k servers with epyc-2 if performance is better than Intel. It's about time for some new servers anyway.
- tgtweak 7y agoIt seems that they're on track to get epyc into OEMs and cloud providers in quantity. They've been successfully growing the ecosystem in the server space over the last 2 years (Supermicro, dell, HPE, Lenovo all pushing epyc sku's now) so they're very well poised to capitalize on it if they can keep up with supply. The power efficiency gains on 7nm will be pretty tempting to cloud providers and colocation clients alike. If you can put 256 cores (4 epyc2 sockets) into a 1U server and keep it under 1200W that's going to be a landslide win for density and running costs. Factor in pcie 4.0 and things are looking pretty sexy. I've read that wafer yeilds on zen2 are 70%+ which gives a huge advantage on cost and production efficiency to AMD. I think Intel's skylake yield for an equivalent 28 core die is sub-40%. If anything the limiting factor is going to be TSMCs ability to give AMD capacity on it's very crowded and in-demand 7nm fab.
- LinuxBender 7y agoDell and HP already have them in quantity. We just have to perform due diligence to ensure all of our workloads will remain the same or better performance. Thus far, testing is looking good.
- bArray 7y agoQuick question: Why put all of your eggs in one basket? I know you want to maximize profits, but you also want to minimize risk? For all anybody knows the next big CPU CVE could be on AMD architecture, especially as people will now being taking the processor much more seriously.
- LinuxBender 7y agoOur eggs are already all in one basket across two vendors and that did already bite us. Cost wise, it doesn't make sense to load balance across Intel and AMD. We have to pick one or spares cost are doubled and tooling has more overhead. In a few years, if Intel have their act together and AMD start having more vulns, then we may swing back the other direction. Only time and vendor actions will tell. This assumes we even move to AMD.