14 ms·
Linus Torvalds: “Do No Harm”
- carlsborg 9y ago"without users, your program is pointless, and all the development work you've done over decades is pointless. .. and (then) security is pointless too, in the end." He tends to get really mad when kernels dev inconvenience user space devs. Perhaps one of the reasons Linux succeeded was because of this fanatical customer focus - if linux is the platform, user space developers are the customers.
- babarock 9y agoThis. Not breaking user space is fundamental to the success of most operating systems today. I read this article I cannot seem to find about Microsoft employees spending months replicating "wrong" behavior in <oldwindows.. win95?> so that applications still ran on <newerwindows... win98? win2000?>. Users don't want to see their applications crashing. I wish I could find this article again, it's very relevant to your comment.
- aurelianito 9y agoIt reminds me of the hack Microsoft did to make simcity work in Windows 95. https://news.ycombinator.com/item?id=2281932 https://news.ycombinator.com/item?id=2281932
- digi_owl 9y agoAnd then userspace goes on to piss on that by breaking their own stuff left right and center...
- chris_wot 9y agoSo tl;dr - fixing a security bug is rarely the end of the story, fixing the root cause is far more important. And don’t piss off the users.
- _nalply 9y agoIf your security patch kills users' buggy processes or even crash their systems then you are a «bad security person». Please report the bad access first so users and developers of their software have time to fix the bug. Upgrades disabling users' software are a big no-no. After all security is meaningless for a non-working system.
- theWatcher37 9y agoI’m so glad that backwards thinking concepts like this are dominant, otherwise we might actually have secure software! Think about it, an open-source OS is choosing backwards compatibility over security. This would have caused quite the stir in the 90’s Linux community.
- _nalply 9y agoSecurity is meaningless if your system doesn't work. Makes sense for me. Or your car is forbidden from starting after an upgrade because you might run over someone tonight. It's a trade-off.
- iforgotpassword 9y agoThey're choosing backwards compatibility over bad security (kill everything instead of reporting and fixing bugs). Applying duct tape everywhere instead of fixing things can hardly be desired. And as a side note, backwards compatibility is what made windows succeed.
- matwood 9y agoDo you want your system to kernel panic every time the security code gets a false positive about a problem? The security code is just as likely to have bugs as the software it is securing.
- topspin 9y agoLinus has been pounding on the bench saying "Don't Break Userspace" for over two decades now. This is really just another manifestation of that same policy. Some 'security' folk apparently believe they occupy a special place in which they are exempt from this policy. Linus is showing them they're wrong; evolving the kernel into a minefield in the name of security is not the way to world domination.
- lima 9y agoBackground: the "kernel self protection project" (KSSP) recently upstreamed the Grsecurity/PAX reference counting implementation which prevents a certain class of security bugs from being exploited. Grsecurity is a security hardening patchset for Linux that makes deliberate trade-offs in favor of security, sacrificing availability if necessary. This, aside from the political issue, is the main reasons why it's hard to upstream it. Linus has called some of their mitigations "insane" before precisely for that reason. Grsecurity will rather terminate userland programs or, in some rare cases, panic the kernel if it finds itself in an undefined state. This is exactly what you want if you care about security, but it's not a trade-off everyone is happy with (including Linus). Unfortunately, Grsecurity/PAX is not (and probably won't ever be) involved in the KSSP project, and the KSSP developers do not understand the code nearly as well as the Grsecurity team does. This lead to a situation where the new code caused a crash that they weren't able to fix in time, so they disabled the feature in the last minute. I've been using Grsecurity for years until they stopped making it publicly available, and I remember many bugs that were uncovered by PAX_REFCOUNT and yes, occasionally panicked the kernel where a vanilla kernel would run just fine. They usually found and fixed those within hours. Grsecurity/PAX have invented many of the modern exploitation mitigations, probably second to none. Some have even been implemented in hardware. Their expertise in building modern defenses is astonishing (their latest invention, the control flow integrity mechanism RAP, is a work of art). Linux could be the most secure kernel, instead, it's fallen way behind Windows - which has much better defenses than Linux nowadays thanks to Microsoft's ongoing battle with rootkit writers. Go figure. If the large companies who use Linux really want to improve kernel security, they need to work with Grsecurity and not against them. It's beyond me how this isn't happening already.
- amelius 9y ago> Grsecurity will rather terminate userland programs or, in some rare cases, panic the kernel if it finds itself in an undefined state. This is exactly what you want if you care about security, but it's not a trade-off everyone is happy with (including Linus). I'd also like my kernel to halt whenever an assertion does not hold, for the sake of keeping my sanity; not just for security. Why would you not want this?
- Santosh83 9y agoVery pragmatic. He sees software in the overall context of getting a job done with a computer, imperfect though it may be, instead of dying because it was not perfect. Unlike a segfault from a user space program that indeed merits a 'kill', the kernel should strive at all costs to keep running, since kernel panics are so much more inconvenient.
- theWatcher37 9y agoAbsolutely not. If “do no harm” is a principle, then the kernel should ensure that no harm is taking place. If flaws within the kernel allow harm to occur while otherwise normal transactions are occurring then it is absolutely preferable to panic and shut down over allowing that potential harm to occur. To suggest otherwise, that detected errors that allow harm should be allowed, is pure insanity. Linus is unquestionably wrong in the regaurd.
- iforgotpassword 9y agoNo. First, you never know if an invalid access or integer overflow can actually be exploited, and even if, you don't know if it can remotely be exploited. If you run a server hosting sensitive data by all means, use grsecurity but on my home PC where I browse Facebook and send emails, fuck off with your kernel panic.
- anameaname 9y agoA thought experiment that comes up in Kernel design classes is what should happen if the OS was running the flight-control software for an Airplane you are on? If there was a bug in the kernel, perhaps a double free or a memory leak, what should happen? A panic would result in the airplane falling to certain doom. But if it were to keep running, it may be a security vulnerability. Being absolutist in either direction of the discussion will lead to absurd scenarios where you would make the wrong decision.
- Slavius 9y agoKernel is modular. Literally everything can be enabled/disabled. Aviation has strict regulations and that's why most critical systems have redundant parts. Putting a sigle critical component into plane is stupid in and of itself. Think of simple freezing in high altitude or overheating otherwise. On the other hand I would rather fly in a plane whos altitude meter shuts down and switches to redundant circuit other than letting it report incorrect values...
- margorczynski 9y agoQuick question - doesn't Linux driver model (they run in kernel-space) create a giant attack vector because of that?
- TheChaplain 9y agoAFAIK you must be root to load kernel modules.
- mschuster91 9y agoLots of drivers on Windows, OS X and Linux run in kernel space simply because kernel-to-user-and-back context switches are expensive and so kill performance. I believe the exceptions are printer and scanner drivers (these run in user-space CUPS in OS X/Linux), some filesystem drivers (basically, FUSE-backed) and cheap-ish USB drivers.
- margorczynski 9y agoThe logic behind why it is done like that I get. Just wondering as You said is it possible to push at least the most bug-prone and exploitable ones to user-space
- darklajid 9y agoThis would be something like the Hurd for example (or Mach based systems), as far as I understand. Over there drivers and whole subsystems are running as separate processes and just pass messages. Is that what you're interested in?
- Santosh83 9y agoI don't see how you can convert a kernel-space driver to a user-space one without significant rewriting, and in some cases it may not be possible at all.
- margorczynski 9y agoWhat about some abstraction/interfacing layer/driver that would take care of exposing some kernel functionality an average driver needs and provide additional validation?
- alanfranzoni 9y agoThat's a consequence of an "old" issue in the IT security field - security researchers and developers sit at opposite sides of the table, they've got different concerns and agendas. Pick some security researchers; now tell them to build any nontrivial piece of software; I doubt they'd be able to do it, and if they succeed their software will be full of bugs, including security ones. Security is part of the correctness and of proper building of the software, so it should be integrated into software development. Security experts can (and should) still exist, but the current state, where the infosec people appear to rule, is pointless - exactly because the same infosec people wouldn't be able to deliver better software than current developers. I highly regard somebody who can write a software without security bugs; I don't regard as highly somebody who shows me the bugs, but would be unable to write that software at all. Let's turn the infosec objective: not to undiscover security bugs, but to write software without security bugs. Then we're at the same side of the table.
- lima 9y agoThat's not a fair (or useful) assessment. Obviously, the narrow-minded security people you describe exist, but they're a minority. Many security people are developers who specialized in security, and are very much capable of building software. The kernel code is question is exactly what you ask for - instead of finding and fixing single bugs, it's a mitigation that prevents all occurrences of a particular class of bugs.
- alanfranzoni 9y agoOf course it's "anecdata" because I've never personally conducted a test of all security consultants for their programming skills. But the people you tell about (developers who specialized in security) maybe exist in large shops (Google, MS, FB), but many, many security consultants will work for specialized firms that offer security services, but not software development, and vice versa. Take a look at most Defcon/BlackHat talks where a vuln is explained/uncovered/exploited: most such researchers don't pertain to a software development firm, but to independent security firms. Why should my assessment not be useful? I proposed a very clear solution to what is the problem. Source: I worked for almost ~10 years in a firm who had both a software development and a security services branch, met tens of security consultants and worked in remediation activities for software issues where the security consultants weren't able to do it. EDIT: about the kernel code, I agree with you that such code is a step in the right direction, but I agree with Linus that the "warn" should come before the "kill".
- lifeisstillgood 9y agodo you want to know why your (random big company) does not do software development that well. when was the last time you saw an email like that from the chairman of the board to all employees? with Please in it. and long winded explanations? companies need to chnage
- corpMaverick 9y agoThis is what I was thinking when I was reading. Linus has been thinking carefully about this for a long time. He has taken this piece of software and he has nurtured it for years. He has protected it. He made sure that it is coherent and maintainable. He has put forward as set of guiding principles. I haven't seen this in the companies that I have worked for, middle and upper management don't have any insight over the products they are building. They just care about dates and getting projects done. There are not insights. There is no long term thinking. There is no awareness of the technical debt that is building up. Their only solution is to throw more money (and people) to the projects.
- ksk 9y agoI 100% agree with Linus here, buuuut Linus doesn't have to make money for his company, and isn't personally responsible for the salaries of people in the company. He doesn't have to sit in meetings and explain why sales are down 10% this quarter, or keep assuring investors that their money is safe. Those upper-management people do solve problems, its just that they're solving problems in an environment that isn't based on logic, and is fundamentally unfair.
- KasianFranks 9y agoA dumpster is not a bug, it's a social issue - http://www.mit.edu/hacker/hacker.html http://www.mit.edu/hacker/hacker.html
- sgentle 9y agoIt's interesting to see this laser focus on a particular kind of user. If you're running Linux on a server, you're a user, but unless you're very irresponsible you would probably rather your programs crash than give away private information. Your interface is to a cluster of machines where individual crashes are probably not that big a deal. If you're running Linux via Android, you're a user, but mostly you're a user of actively developed apps on top of an actively developed OS, usually pegged to specific kernel versions. Your interface is to that layer on top, and given that its code is written by app developers and hardware vendors who will ship anything that doesn't crash, you probably want security bugs to crash. It seems to me that the kind of user Linus means when he talks about "the new kernel didn't work for me" is a user of Linux without any substantial layers on top, where kernel updates happen more often than userland software updates, and where individual crashes have a significant impact. In other words, users of desktop Linux. But I wonder if that focus on desktop Linux really reflects the majority of users. And, if not, perhaps it might make sense to have "hardening the Linux kernel" as the first step if it makes "raise the standard for the layers built on top" the endpoint.
- lima 9y agoSadly, fault-tolerant clusters where you can tolerate the loss of a single machine aren't the norm. There are many (MANY) server applications or industrial use cases that do not handle random kernel panics very well. I still prefer "crashing" over "silently ignoring critical errors", but you cannot generalize it like that.
- gjulianm 9y agoOne of the point of Linus is that most of the crashes will not be because of an active attack but because of a possibly latent bug that sometimes appear and might very well be non exploitable. So probably people running server would like to have everything working instead of having random crashes on processes or drivers that are not the core of your service but can affect it. And given the size and complexity of the kernel it would not be strange to have these crashes appear only on certain setups and not necessarily on the ones of the people that are testing it first.
- jbangert 9y ago
- kyberias 9y agoI think the earlier message drives the points home in more familiar Linus style: https://lkml.org/lkml/2017/11/17/767 https://lkml.org/lkml/2017/11/17/767 "Some security people have scoffed at me when I say that security problems are primarily "just bugs". Those security people are f*cking morons." Gotta love the guy. :)
- smnrchrds 9y agoThe person at the other end of the conversation would disagree with this sentiment: "Thanks. Still, I'd prefer Linus yell at me than other folks trying to do similar work. If I can shield anyone from this abuse, then maybe they won't give up on kernel security development. Digging Linus's actionable feedback out of the ad-hominem attack can be challenging." [1] [1] https://twitter.com/kees_cook/status/932694978366619648 https://twitter.com/kees_cook/status/932694978366619648
- rimliu 9y agoI am happy Linus shields Linux kernel from those golden-hearted individuals whose feel-good attitude would allow questionable solutions designed by committee to slip in.
- kadenshep 9y ago>Digging Linus's actionable feedback out of the ad-hominem attack can be challenging. They're not really that separate. He's being totally disingenuous and still letting his own fragile ego get involved.
- jbangert 9y agoA few points: 1) failing loudly is better than failing silently. A memory corruption issue (or a bad refcount, etc.) is not a benign issue that only becomes relevant under carefully crafted exploit conditions. You need the carefully crafted exploit to get the system back into an attacker controlled state (I.e. code execution); by itself (with non-malicious inputs, usually something random or slightly atypical — enough to not have been noticed yet, but typical enough that some program does it) the system is likely to either panic immediately (same result as with pax) or to corrupt some memory, in which case you will have a lot of strange behaviour to track down later (users will probably blame them on hardware or on their user space, so you might never see them. for example a recent OSDI paper showed that ext3/4 had several real world data corruption bugs. If these aren’t as frequent as the recent bcache issues, no one notices). 2) When I was doing research projects (into memory defenses on the kernel) about 3 years ago, there was no (commonly used, that I saw) automated testing infrastructure in the kernel. This makes catching regressions, especially in drivers for rare hardware, hard to catch. While tests aren’t a panacea, i think Linux overestimates what fraction of problems Code reviews will catch. 3) the “don’t break user space” strategy is already failing. Every mainstream distribution and embedded vendor stays on an old kernel branch. Big deployments do staged rollouts and extensive burn in tests. This isn’t just because the kernel, but because of extensive abreaking changes everywhere (compilers, standard libraries, etc. all need to change sometimes).the last time this happened, IIRC it was some audio bug in a strange configuration. In my experience, running a non standard Linux audio confit causes countless breakages, so an additional one in the kernel that might save my personal data from being exfiltrated is worth it. Most users have average (and therefore well tested) setups, which means thy won’t see breakages as often. Perfect software doesn’t exist, and even MSFT backed off maintaining religious backwards compatibility (note that Microsoft’s approach was not to flame at developers and hinder new development, but through extensively building compatibility shims. Often, these came with trade offs strongly in favours or security, e.g. UAC). Breaking user space is ok; users already expect breakage, and the cost of the additional breakages is low (to users and to society as a whole) compared to the cost of security breaches [citation needed, but Linux kernel security is relied on in a lot of places].
- blaisio 9y agoSo one of Linus' main points in this series of posts is that failing loudly is actually not always better than failing silently or quietly, and it's really annoying when people come in making that assumption without thinking. This is also something that he is constantly repeating and ranting about, and it's arguably one of the reasons why Linux is so successful. Think about a smartphone - do most users want it to crash and reboot, even if some error (which could end up being a security issue) occurred? The answer is no, absolutely not. The crashing and rebooting itself isn't really that helpful. Reporting the bug to the Linux developers _would_ be helpful. Some people do want the frequent crashing behavior and that's okay, but it's not okay to make that decision for everyone. Also, users might expect minor breakage if someone somewhere makes a mistake, but that doesn't mean it's okay. That's like saying if someone always washes their hands before eating, it's okay if they get sick, because they were expecting that they might get sick.
- unixhero 9y agoThis is the most insightful Linus writeup yet. His others are good too, but this just hits the spot. Great! Funny note, this post could have been a textbook sort of material. At the end he even says please. The only thing that breaks it is the reference to touching oneself :)
- quink 9y agoI'm reminded of a recent security fix for IE11 that Microsoft pushed out earlier this year: https://developer.microsoft.com/en-us/microsoft-edge/platform/issues/12349663/ https://developer.microsoft.com/en-us/microsoft-edge/platfor... It killed printing from iframes completely. Great that they solved the security problem, whatever it may have been. They also broke a major piece of browser functionality that a lot of enterprises rely on fundamentally. Hell, even printing shipping labels from eBay was broken, and so was hilariously Microsoft Dynamics 365. And our own product. And the first response by Microsoft on this very link? "Won't Fix" and keep a major bit of fundamental browser functionality - printing a document - not working. "Do No Harm".
- pjmlp 9y agoBecause there is a workaround, applications just need to be updated. "Either use Print Preview instead of print button or if currently using window.print() in javascript change this to document.execCommand('print’, false, null)"
- AnIdiotOnTheNet 9y ago> Because there is a workaround, applications just need to be updated. What actually happened: people reverted the patch. In the real world, you can't expect timely or even correct response from vendors you rely on. It sucks, but it's how it is.
- torpcoms 9y agoThen after you get compromised in some way, you will start to look for better vendors.
- AnIdiotOnTheNet 9y agoOnly if the cost*risk of a future compromise is > cost of replacing the product. This is rarely going to be the case for business critical software.
- 9y ago
- zaarn 9y ago>It's that the code has been RUN BY USERS for months. If it's been [...] in grsecurity for five years [...] It only means that hardly anybody actually ever ran it. Subtle burn towards Grsec, I laughed a bit. In all seriousness, I think Linus is somewhat on the right track. Security Patches should foremostly not break anyone's workflow (maybe except the evil haxor's workflow) and rather print a warning until the exact implications of a full Terminator-mode patch is understood. Because people won't upgrade to kernels that break their workflow and a warning in the kernel log is better than a vulnerable kernel.
- zx2c4 9y agoI wrote the email that prompted this quite civil response. I'm very pleased with the outcome, because I think this clear statement of his position is a lot more useful for people to work with, rather than just assuming Linus hates security or something. I interpreted his response in practical terms as essentially being the following. Patch set merge 1 has "report" as default and "kill" as a non-default option. Patch set merge 2 has "kill" as default and "report" as a non-default option. Patch set merge 3 removes support for "report". This way we have the best of both worlds: we eventually reach the thing that actually adds real security benefit, which makes security folks happy. And we don't break everybody's computers immediately, allowing time for the more obvious bugs to surface via "report", which makes users and developers happy. Seems like a reasonable process to me.
- zaarn 9y agoThough, I think that the time between PSM1 and PSM2 will be significant. Usually default options are changed once basically all distros compile with another option without widespread breakage. And once no LTS kernel with PSM1 is supported, you merge PSM3. Might take years but atleast the airplanes keep flying instead of crashing their computers and consequently themselves.
- zx2c4 9y ago> Though, I think that the time between PSM1 and PSM2 will be significant. Indeed you're probably right there. Fortunately security-focused distributions and individuals would be able to change the defaults in the interim.
- shandor 9y agoI think the most important point here is that with those different patch sets, the more security-conscious users/companies get to have the properly hardened version immediately into use, rather than running the "only report" versions for what might be years, as you say. Granted, I'm looking at this from a perspective where our company compiles our own kernel for use in embedded devices, so we can have whatever patch sets supported we want. But I think that's much better than everyone having to use the "report only" patches for years, or even worse, the features never getting into the kernel in the first place.
- Annatar 9y agoBecause the primary focus should be "debugging". The primary focus should be "let's make sure the kernel released in a year is better than the one released today". Is he starting to sound like illumos engineers or what? Better late than never, but it took him long enough.
- mtgx 9y agoLinus: Do no harm! Except against security people. I'd like to harm them a lot!
- luckydude 9y agoWow, Linus still has it. That was an _excellent_ email. Go Linus, that's some full on management chops on display.
- dang 9y agoRelated to this from a couple days ago: https://news.ycombinator.com/item?id=15738392 https://news.ycombinator.com/item?id=15738392
- throw2016 9y agoSome security folks tend to have tunnel vision. They are focused on their own little patch and the first instinct is to control and shutdown with zero concern for usability. job done. bye. But that's too self serving and lazy. And if that's what they want fine, release your own app or distro and let people who need/desire that level of security choose security over everything else without trade-offs. The worst thing is a kind of entryism and trying to impose yourself on people operating under different constraints. That's why you always need a strong manager to balance interests and this is the role Linus is playing. More often that not with this kind of imposition something comes out of the blue, and you are left squandering hours to fix things and get back up and running.
- SAI_Peregrinus 9y agoNo security person is likely to argue that DOS attacks aren't a security issue. They get CVEs all the time! A patch that denies the user the ability to use the service introduces a security issue. Bad usability IS a DOS attack.
- leothekim 9y ago"Without those users, your system may be "secure", but all your security work was still just masturbation. You didn't do anything useful at all in the end." Amen. Linus has been consistent with his views on security, and this is just an elucidation of a process that reflects those views. [1][2][3] [1] https://lkml.org/lkml/2008/7/15/296 https://lkml.org/lkml/2008/7/15/296 [2] https://www.cio.com/article/2973995/linux/linus-torvalds-security-is-never-going-to-be-perfect.html https://www.cio.com/article/2973995/linux/linus-torvalds-sec... [3] https://www.youtube.com/watch?v=C1EJvDU2Ek4 https://www.youtube.com/watch?v=C1EJvDU2Ek4
- kauegimenes 9y agoBody for this message unavailable Cache: https://web.archive.org/web/20171122005241/https://lkml.org/lkml/2017/11/21/356 https://web.archive.org/web/20171122005241/https://lkml.org/...
- fb03 9y agoThe first thing I did when starting to read Linus' answer was to Ctrl-F for badwords. I am really happy he's blossomed from that teenage angst colorful name-calling era. Way to go, man!
- kseifried 9y agoWith respect to https://lkml.org/lkml/2017/11/21/356 https://lkml.org/lkml/2017/11/21/356 and this thread I just want to get #CVEs on all the things so we can figure out the scope of the problem, and then look at what we need to do to "fix" "it" (assuming "it" is a real problem, dunno without data).
- Shorel 9y agoI wish Wayland developers would share Linus' mentality. Right now it restarts once every 36 hours in average, killing all my windows and programs. X11 didn't behave as bad.
- torpcoms 9y agoIs there no way to make this an argument at boot time? Something like, `ksecerr=log+kill`