14 ms·
Linux Kernel Developer Criticizes Intel for Meltdown, Spectre Response
- cthalupa 8y ago>"The majority of the world runs Debian or they run their own kernel," Kroah-Hartman said. "Debian was not allowed to be part of the disclosure, so the majority of the world was caught with their pants down, and that's not good." Is there any actual statistics to back this up? I feel like RHEL and to a lesser extent CentOS have a stranglehold on the big enterprise-y environments, and I see Ubuntu basically everywhere else, and Canonical does their own kernels. Edit: To be clear, I am aware Ubunbtu is a Debian derivative, but since we're talking specifically about who was informed for kernel level mitigations, and Canonical does their own kernels, it seems weird to talk about how Debian wasn't informed and thus people were affected, when Ubuntu being updated wasn't reliant on Debian being updated.
- sounds 8y agoPerhaps Greg-KH was implicitly including Ubuntu under Debian?
- 2trill2spill 8y agoYea I don't use Linux too often but how different is an Ubuntu kernel compared to a Debian kernel? I know Ubuntu is downstream from Debian or at least they were. So how much extra work/code does Canonical add to its kernels.
- cthalupa 8y agoCanonical had Spectre and Meltdown mitigations out for their kernels before Debian did, and he also specifically mentions Canonical as getting alerted, so apparently enough.
- _arvin 8y agoI think there should be a distinction between Canonical and debian, but Ubuntu is certainly based on Debian, and the latter should be used when talking big scale like he did. Many cloud services and providers run on Ubuntu, and as long as Ubuntu uses apt and .deb, it's Debian (along with Linux Mint, elementaryOS, etc)
- craftyguy 8y ago> and as long as Ubuntu uses apt and .deb, it's Debian No, there is more to a distro than them sharing a common package manager/format. The kernel, which is heavily patched by canonical, differs from what Debian ships. Canonical also has a bad habit of carrying a lot of non-upstream patches for other core system components. Ubuntu may have been derived from Debian at one point, but they are fairly different at this point. So much so that you cannot take a system running Debian and 'upgrade' (or downgrade, depending on your point of view) it to one running Ubuntu.
- StudentStuff 8y agoActually, you can smash Debian & ubuntu together, you just get a terrible monster called frankendebian. As a youngster I tried this, it didn't pan out.
- int_19h 8y agoA distro, for the most part is 1) package manager + repos, and 2) system organization (where things get installed, where configs are, how services are managed etc). In that respect, Ubuntu is still Debian - most packages are vanilla Debian packages, and they both use systemd.
- geezerjay 8y ago> So much so that you cannot take a system running Debian and 'upgrade' (or downgrade, depending on your point of view) it to one running Ubuntu. That doesn't say much, as sometimes ubuntu and debian upgrades are far from smooth. As a long Debian & Ubuntu user, I believe you're exagerating the work invested by Ubuntu to custonize Debian. Betond the default desktop environment and attitude regarding proprietary drivers, the work done by Ubuntu is rather negligible.
- Twirrim 8y agoAs mentioned in the article, Canonical were brought in much earlier by Intel. They appear to have been among the "chosen few": RedHat, SuSE and Canonical.
- stefan_ 8y agoThe majority of Linux installations is probably Android smartphones, and well Google was informed alright, but still 95% of those phones were caught with their pants down because well their pants had fallen down a long long time ago given they last received security updates years ago.
- joe_the_user 8y agoHow many android installations use Intel processors? Edit: never mind, Android was apparently indeed effected. https://www.androidcentral.com/meltdown-spectre https://www.androidcentral.com/meltdown-spectre
- gsnedders 8y agoSpectre affected virtually every out-of-order execution CPU in existence; Meltdown was much more limited in scope (and very few non-Intel CPUs were affected, but some were).
- DCKing 8y agoSpectre and Meltdown are a problem for Android but it's a smaller problem than on PCs. The vast majority of Android phones use processors that are in-order, and lack the out-of-order execution necessary to exploit. High-end Android phones with out-of-order processors (the minority) are only vulnerable to Spectre, but it's a little complicated [1]. Then there is only one ARM core (Cortex A75) that is vulnerable to Meltdown, but that was released after the vulnerability was published, so software could ship with mitigations in place. [1]: https://github.com/lgeek/spec_poc_arm https://github.com/lgeek/spec_poc_arm
- gsnedders 8y agoSome of Apple's A-series processors were affected by Meltdown too, AFAIK, though obviously that doesn't affect Android. (But is another ARM core vulnerable to it.)
- dijit 8y agoI can speak to the companies I've worked for: US companies (and, this includes the UK because apparently we love to be like america) use CentOS predominantly, but mainland europe (Sweden, Germany, Finland) seem to prefer Debian. I can say that I've personally administered roughly a 1:1 ratio of CentOS:Debian despite coming from a country that's servers tend to be CentOS. Of course, this is anecdotal, but don't undersell debian.
- deleted 8y ago[deleted]
- bluedino 8y agoIs SuSE still popular in Europe? (Apologies if I got the capitalization wrong)
- pjmlp 8y agoBesides SAP deployments I guess not much. Here in Germany it is mostly Red-Hat/CentOS or Ubuntu, on the projects I have been involved. So anecdote data.
- gaius 8y agoSince 2008 working for US/UK companies I have only seen RHEL and CentOS (prior to that, mostly SunOS/Solaris). I prefer Debian personally but for the last few years I have used CentOS for my own projects too - too much effort to maintain two parallel sets of skills now I am not really into OS level stuff anymore.
- xchaotic 8y agoI'm sure he cares a lot about Debian and a lot of my early learning was on Debian derivatives but I feel that the production server side world has largely standardised on RedHat. Non RedHat/CentOS, non-Ubuntu debian distros are probably a distant fifth in terms of business critical production server so no wonder they were not included in the disclosure early on...
- jessaustin 8y agoDoes CentOS get immediate updates like this? My overwhelming impression of that platform is that everything is years out of date... of course maybe that's just the fault of the organizations I've encountered that have used CentOS.
- bonzini 8y agoYes it does. Both RHEL and CentOS have a kernel that is "officially" 3.10 but feature-wise is probably something like 4.10-ish for RHEL 7.5 (which was released last April). RHEL and CentOS 7 had updates for all of Spectre, Meltdown and Foreshadow on the day of unembargo-ing.
- nineteen999 8y agoI'm pretty sure it's being kept fairly up to date since Redhat acquired it. Bear in mind that it's a RHEL clone so it's usually about as out of date as RHEL is - which is what we generally want in the infrastructure/telco/banking space because stability is preferred over shiny new features.
- fl0wenol 8y agoCentOS gets patches as soon as Red Hat publishes the source RPMs upstream or pushes changes to git.centos.org. What can be confusing about CentOS/RHEL is that many shops using them are on a release that came out years ago. But that's an issue because Red Hat supports 10 year lifecycles for their releases, and will go on supporting them indefinitely if you pay them enough. All the software bundled gets feature and security updates through it's lifecycle, just without changing any interfaces, ABIs, or version numbers. This is great for shops that aren't looking to operate on the bleeding edge and have expensive COTS or bespoke software stacks sitting on top. Of course, you still have to patch, and organizations can't be forced to do that. Hopefully what you were encountering was just the former and not the latter at those organizations.
- l0b0 8y agoTranslation: s/The majority of the world runs/My fellow Debian kernel developers run/
- bogomipz 8y agoUbuntu is a Debian derivative[1], I imagine that is what he means: [1] https://www.debian.org/derivatives/ https://www.debian.org/derivatives/
- cthalupa 8y agoYes, but the problem is we're talking specifically about kernel level mitigations, and Canonical rolls their own kernels. They do NOT use debian kernels.
- bogomipz 8y agoPlease re-read the context. I was simply explaining to the OP why this individual may have said this.
- kelnos 8y agoThat doesn't make sense, though. Debian and Ubuntu each roll their own kernels. Canonical was one of the chosen few that had advance warning from Intel. Debian did not, so it wouldn't make sense for Greg K-H to be talking about Ubuntu at all when he referenced Debian.
- pera 8y ago> The Linux distribution can be positively identified in around 30% of cases, and of these 1.39 million Linux computers, just over half are running Ubuntu Linux, nearly a quarter are running CentOS, and around fifth are running Debian Linux. https://news.netcraft.com/archives/2017/09/11/september-2017-web-server-survey.html https://news.netcraft.com/archives/2017/09/11/september-2017...
- craftyguy 8y agoThere are a massive number of systems running Linux that are not publicly facing (e.g. compute/service nodes in supercomputers, systems for providing internal infrastructure for R&D, manufacturing, etc).
- proginthebox 8y agoSuch systems also most likely are not vulnerable to Meltdown/Spectre.
- vokep 8y agoWhy would you begin to think that at all? Because they're offline doesn't mean that they're safe, just safe from more than an online system.
- darpa_escapee 8y agoThose systems might be immune to attacks that networked machines are vulnerable to, but often networked attacks are used in order to leverage another exploit, such as Meltdown/Spectre. A vector other than the network could be exploited to leverage an attack that utilizes Meltdown/Spectre.
- fulafel 8y agoMost the biggest software vulnerability catastrophies actually involve just such systems. "Secure internal network" is a 90s mirage. See eg https://www.wired.com/story/notpetya-cyberattack-ukraine-russia-code-crashed-the-world/ https://www.wired.com/story/notpetya-cyberattack-ukraine-rus...
- rurban 8y agoDesktop and personal PC has Ubuntu dominant, but servers are 70% CentOS. Nobody trusts Debian's security record and gcc/kernel abilities. He is right with the affected number in this case because this time bugs are affected by simple webpage drive-by Javascript attacks, like a common virus. So it affects Debian and Android more than servers and cloud services. And those are the ones which are not regularly updated so the security impact is bigger than on a server.
- curveship 8y agoBy "majority" he may have meant majority of Linux distributions, not majority of Linux users. There are a ton of distributions derived from Debian: https://wiki.debian.org/Derivatives/Census https://wiki.debian.org/Derivatives/Census . All of them would have to update their distros to respond to Meltdown.
- jfaioif3 8y agoIt's the typical arrogance of Debian developers and users. Debian is becoming completely irrelevant but they still keep their heads buried in the sand.
- shakna 8y agoWhy would an off-the-record statement, which is what he said this was, need to be 100% without exaggeration or hyperbole?
- cthalupa 8y agoThe article doesn't mention it was off the record, and I wasn't there, so how would I know it's off the record? But it seems likely that this isn't just exaggerated or hyperbolic, but just not close to accurate, in which case even if it's off the record it's silly.
- jack12 8y agohttps://youtu.be/lQZzm9z8g_U?t=1565 https://youtu.be/lQZzm9z8g_U?t=1565 seems to be the same talk delivered at another instance of this conference. He does claim one statistic that an unnamed "top 3" cloud provider told him that fewer than 10% of their customers install company-based kernels and over 90% install community-based kernels like debian or kernel.org code. He is not limiting "the majority" to just Debian, though. He is comparing installs of distributions backed by companies (like Red Hat+Canonical+SuSE) as "the minority" vs. every possible non-corporate-backed distribution or source.
- crankylinuxuser 8y agoI run centos in production. That's because of the following: 1. SELinux 2. OpenSCAP 3. 99% fit with RedHat None of the other Linux variants have those tools, or the time put in for compliance goodies. My time == $$$
- 2bluesc 8y agoAny videos of the presentation?
- deleted 8y ago[deleted]
- mikeyouse 8y agoNo video as far as I know, but he did post the slides with some backup materials on his GitHub: https://github.com/gregkh/presentation-spectre https://github.com/gregkh/presentation-spectre One note from his talk though, his slides say that Foreshadow was fixed in April -- but it's clearly supposed to be August. He mentioned he was fixing the updates even on the flight out to OSSNA.
- 2trill2spill 8y ago> "That's a long time, and we only heard rumors because another very large operating system vendor told Intel to get off their tails and tell us about it." I wonder which operating system vendor pressured Intel to tell the Linux dev community, especially because it sounds like it was a non Linux OS vendor. Whomever it was, good job! But it seems like Intel has angered the Linux community as well as the various BSD operating systems. You would think Intel would be doing whatever it can to please all operating system vendors especially now that AMD is getting competitive again.
- judge2020 8y agoMy guess is Microsoft seeing how recently they're pouring a lot of resources into the Linux subsystem and how they're trying to seem more "developer-oriented" overall in recent months.
- 2trill2spill 8y agoThat was my guess as well, also doesn't Microsoft run some Linux in their Azure cloud? So that adds yet another incentive for Microsoft to help Linux folks out. Edit: I found a link confirming that Microsoft uses Linux for networking in their Azure data centers [1]. [1]: https://www.datacenterdynamics.com/news/microsoft-runs-azure-networking-on-linux/ https://www.datacenterdynamics.com/news/microsoft-runs-azure...
- notimetorelax 8y agoI work for Microsoft. We build customer facing business applications on Linux, Java, Go, etc. Microsoft is Big!
- 2trill2spill 8y agoI'm sure Microsoft is doing this now, but wasn't that frowned upon before Satya Nadella took over as CEO or was building on non Microsoft products okay before?
- snaky 8y agoBTW > Experts called for a new generation of secure-by-design computers at the Hot Chips conference here. In small steps in that direction, Microsoft and Google described their separate but similar hardware security architectures. https://www.eetimes.com/document.asp?doc_id=1333616 https://www.eetimes.com/document.asp?doc_id=1333616
- y-c-o-m-b 8y agoI wonder how many NSA back-doors will go into those.
- deleted 8y ago[deleted]
- snaky 8y agoThe vast majority of regular users would voluntarely install a dozen of NSA backdoors on their computers if it guarantees them full protection from script kiddies, phishing, malware and ransomware.
- TheJoYo 8y agoI feel like you just described Google.
- tmalsburg2 8y agoUsers in the US perhaps but elsewhere? I doubt it.
- rrix2 8y ago> "Intel has gotten better at this," he said. Someone should let the BSD folks know and see what they think...
- 2trill2spill 8y agoThey are getting better, even if they didn't set a very high bar to begin with. I believe they let the FreeBSD folks know about the last round of Intel bugs as opposed to trying to keep them in the dark like the first few rounds of Intel bugs.
- dshep 8y agoI get the impression OpenBSD isn't so happy with Intel: Disable SMT/Hyperthreading in all Intel BIOSes https://news.ycombinator.com/item?id=17829790 https://news.ycombinator.com/item?id=17829790
- taborj 8y agoAnd also: "I'm going to spend my money at a more trustworthy vendor in the future." - Theo de Raadt
- cperciva 8y agoMy understanding is that they've gotten better. FreeBSD has had advance notice of some issues. Last I heard they offered to let OpenBSD in too, but hadn't found anyone willing to sign an NDA.
- sverige 8y agoI'm reasonably certain that OpenBSD has never agreed to any NDA from anyone as a matter of principle. It's one of the things that makes me love the project so much.
- cperciva 8y agoI don't know if "never" is accurate, but certainly they are very NDA-averse. That's their right, but it means they're going to get left out of things like this; it's simply not possible to organize coordinated disclosure of issues if the participants don't agree to not blab ahead of the agreed disclosure date.
- bigDeal 8y agoDoes anyone honestly buy into the idea that this is something other than a back door, carefully plotted by certain nation-state actors? For real.
- froogie 8y agoWhy would Intel be complicit in it? Suppose they did this. Now, that the vulnerabilities are public, Intel takes the fall. Oopsie, huh?
- deleted 8y ago[deleted]
- bArray 8y agoThe creators aren't always aware of the monsters they create. I think Intel was caught with their trousers down - AMD and ARM in many respects too. Hopefully this will lead to automated testing procedures for Intel, other processor manufacturers and security researchers. The reason these flaws were found in the first place were because of security researchers, hopefully they can now make their cases to get large research grants and bring about a more secure future. On the other hand, if you want to talk about tin-foil hat "CPUs working against users", only look so far as Intel ME and the equivalents on other processors. Closed-source and highly privileged code running all the time inside the CPU - it's not unthinkable that Intel would bend the knee to some state actors, especially if that leads to higher profits (i.e. a tax break or entry into a new market (cough China)). It's enough of a concern that a lot of reverse engineering effort has gone into preventing it from running.
- acdha 8y ago> Hopefully this will lead to automated testing procedures for Intel, other processor manufacturers and security researchers. They have tons of testing already. The problem is that this is the kind of problem which is easy to miss with tests, especially automated ones, because everything worked correctly and no real code would ever have been affected by the side effects.
- dschuetz 8y agoBecause Intel is that good in security and hardening, why not use Intel's newly minted own special secure Linux distribution? https://01.org/blogs/imad/2018/letter-industry https://01.org/blogs/imad/2018/letter-industry This is almost like satire.
- smhost 8y agoNot sure what you mean. This just looks to me like they're admitting they're awful at security and they're pleading/threatening the global community to help secure their ecosystem or else you're all going to be in a ton of trouble because the robots/cars are going to kill you and your workers and what'll you do then?
- benchaney 8y ago> This just looks to me like they're admitting they're awful at security and they're pleading/threatening the global community to help secure their ecosystem That post seems to make it pretty clear that they are trying to position themselves as the experts rather than the people asking for help. That is the part that is absurd.
- hornetblack 8y agoI think even a Intel contractor was indirectly responsible for GRSecurity going behind a paywall. (tl;dr intel contractor were shipping an os that was using grsec tradmark in marketing for a non-grsec approved kernel)
- docker_up 8y agoWhat is the current status of this? The last time I heard, the OS fixes would impact CPU performance by 30%. Is this still the case? Will new iterations of Intel CPUs be immune to this, or is this an ongoing issue going forward because it's inherent with the architecture?
- craftyguy 8y agoMeanwhile, on HN frontpage: https://news.ycombinator.com/item?id=17876476 https://news.ycombinator.com/item?id=17876476
- simcop2387 8y agoBased on my own experience with things and seeing other benchmarks, it looks like there's some synthetic benchmarks and very specific loads that will see up to 30% losses, but most things are really around 10-15% if they've got any significant losses at all. That's still a big impact but not nearly as "sky is falling" as it seemed like it might be. No idea how this'll be going forward with hardware changes since those haven't happened yet, and even when they do, how will we benchmark them properly?
- hansendc 8y agoDisclaimer: I work on Linux at Intel. Future hardware will be mitigated against side-channel issues. There's a nice table showing how things are mitigated on future processors here: https://www.tomshardware.com/news/intel-cascade-lake-details-spectre-meltdown,37674.html https://www.tomshardware.com/news/intel-cascade-lake-details... But, not everything is mitigated in the silicon or microcode. The mitigation for Spectre / Variant 1 / Bounds Check Bypass, for instance, will continue to be in software.
- tracker1 8y agoEverything I've read has said real workload differences can be as little as 3% or as much as 30%, but mostly somewhere in between (closer to the 3% side). Depends on what you're doing I guess.
- cperciva 8y agoWhile I agree that Intel's response was far from ideal, I find it a bit rich for Linux kernel developers to be criticizing them. Remember, the completely uncoordinated disclosure happened because Linux kernel developers started discussing the vulnerability -- while under NDA -- on a public mailing list.
- NullPrefix 8y agoYou sure those devs were under NDA?
- 0xFFC 8y agoI was in the room. Greg specifically said “off the record”.
- TillE 8y agoThat's not a magic phrase, it's something you negotiate with reporters in advance. You can't really make "off the record" comments in front of an audience at a conference.
- 0xFFC 8y agoDid I say it is a magic phrase? It used to say when you want something not reported. You should definitely get familiar with journalism terminology.
- wtallis 8y ago> Did I say it is a magic phrase? Pretty much, yeah. You did. Pretending that declaring "off the record" during a presentation like that falls anywhere near normal journalistic practice is essentially the same thing as imputing magical properties to that phrase. Confidentiality needs to be negotiated ahead of time, usually with an NDA or embargo agreement before journalists are given any detail. When presenting at a conference that lets journalists attend for free, it's unreasonable to expect any confidentiality for the content of your presentation unless your presentation is part of a closed session that the free media passes don't grant access to.
- hansendc 8y agoI worked on the Meltdown mitigations, at Intel, during the embargo. I still work on Linux at Intel. Intel told quite a few members of the Linux community about this well before late December. Some got told because they work for traditional distributions and others were contacted directly because they are community maintainers or subject matter experts. Unfortunately, Greg was not one of those folks that was told early. It's pretty clear at this point that it would have been a lot better had he and folks like him been involved earlier. This is especially true for Greg since he plays such a crucial role in stable kernel maintenance, which is how a lot of the world consumes their kernels (including distros like Debian). I'm glad Greg thinks "Intel has gotten better at this." I'd like to think so too.
- 2trill2spill 8y agoShouldn't Greq and Linus be some of the very first people to be informed considering the amount of impact they both have on the entire Linux ecosystem? I find it alarming that they were not contacted immediately considering how much of the world/internet relies on Linux.
- deleted 8y ago[deleted]
- ibotty 8y agoThat does not address the stupid embargo on involved linux developers to speak about it to each other.
- mjevans 8y agoIt shows that legal and marketing (PR spin) were prominent on the security decision tree rather than security minded individuals.
- MaxBarraclough 8y agoIt's a real pity that Company caught putting PR before customers' interests so rarely causes a backlash, even when it's plain as day.
- AdmiralAsshat 8y agoGreg's response is understandably frustrated, though seemingly less so than the OpenBSD devs. Why are they being repeatedly left out of the loop?
- amaranth 8y agoOpenBSD has been accused of breaking embargoes in the past. They are pretty open about their policy of pushing their fix as soon as it's ready and not doing anything to obfuscate what they're fixing and why.
- adamrt 8y agoCan you provide some links for that? I feel like this is a thing people repeat, specifically about KRACK and its been shown to be false multiple times (like here: https://lobste.rs/s/dwzplh/krack_attacks_breaking_wpa2#c_pbhnfz https://lobste.rs/s/dwzplh/krack_attacks_breaking_wpa2#c_pbh...).
- benchaney 8y ago> They are pretty open about their policy of pushing their fix as soon as it's ready and not doing anything to obfuscate what they're fixing and why. This isn't true at all. Please stop spreading misinformation.
- occams_chainsaw 8y agoNeither of you have provided citation or reasoning. You basically just said "nah uh"
- pmontra 8y ago> While there have been many patches made in Linux, he strongly advised users to update with Intel's microcode fixes as well, as they provide an additional layer of protection beyond what an operating system can provide. I found this page explaining how to do it https://www.cyberciti.biz/faq/install-update-intel-microcode-firmware-linux/ https://www.cyberciti.biz/faq/install-update-intel-microcode... I checked and my microcode is from 2018-01-21. Being on a 2014 laptop it means it got updated by the package manager.