6 ms·

Reading this, the question that immediately pops in my head is: Could a hack like this one go undetected for so long in a widely used free/open-source project

by cs702 6y ago

Reading this, the question that immediately pops in my head is: Could a hack like this one go undetected for so long in a widely used free/open-source project developed in the open, such as the Linux kernel? While I have no doubt that something like this could happen to the Linux kernel source code (because security is Capital-H Hard), my perception is that something like this is less likely to happen to the Linux kernel -- and, were it to happen, it would likely be detected sooner, due to the inherent transparency of widely used open-source code.

xen2xen1 6y ago
Code was added once to Debian (IIRC) and it was detected almost immediately due to code signing.
- newacct583 6y ago
  The exploit in this case had access to the build (and presumably signing) system. That wouldn't have helped. The protection against this would have been the comparatively new efforts at reproducible builds. A modified binary, in theory, could be detected by current Fedora and Ubuntu releases (not sure about Debian or other distros). I don't think we've had an attack in practice though.
- AnIdiotOnTheNet 6y ago
  On the other hand, Debian broke OpenSSL generation and didn't detect it for almost 2 years. That appears to have been a legitimate mistake, but it is quite conceivable that a malicious actor gets a change merged that contains a backdoor that looks like an innocent mistake and goes undetected for a long time.
aquaticsunset 6y ago
As others (and Microsoft) mentioned, it was read only access. The only points of concern here would be if that statement somehow was not true and they were able to add undetected changes, or if their security audit process was severely lacking. But yeah, to your point - being able to read and analyze the Linux kernel source is considered a feature, not a liability :)
- neodymiumphish 6y ago
  I think you're connecting two points he made that weren't connected. On the one hand, open source projects make for an environment where bad actors could propose changes to the software that include these bug/backdoors. The benefit to the open source arena is that these changes can easily be analyzed and tested. In Microsoft's case, the source being visible but not editable is still a real risk (assuming the bad actor is able to extract the data they're viewing for further analysis), because they can use the source to determine avenues for attack. The fact that is was read-only does help ensure that no new attack vectors were created, but it still increases the chance of new attack vectors being found/used in the future.
deleted 6y ago
[deleted]
staticassertion 6y ago
I suspect adding bugdoors to Linux is far easier than it is than for Windows, but there are already so many bugs it's easier and more viable to just look for them than to try to insert them.
kerng 6y ago
I do security research and bug bounties on side sometimes and had read/write access to a couple of large open source projects in the past, incl. being able to impersonate employees from well known companies that work on open source stuff. Most common issue was access tokens found in public places. Would be interesting to know what happens when code is updated - which I obviously wouldn't do. Wonder how long it would take until caught. Since open source projects probably dont do "red teaming" (to use a fancy buzz word) I wonder how they could practice this?
wil421 6y ago
Why would you need to back door Linux when you can find a company like Solarwinds that is already in most networks with greater access to the network as a whole than a Linux server.
joe_the_user 6y ago
This hack wasn't really a failure of code construction but a failure of institutional practices. The same thing could have happened if SolarWinds had a garbagy sys admin tool that happened to also be open source but still otherwise followed the procedures of SolarWinds. Giant bureaucracies have a bunch of tasks they need to accomplish. Giant bureaucracies hire poorly trained people to accomplish those tasks and buy software to aid it's those people in accomplishing those tasks. The software is sold "by the feature" so it is colloquially "garbage" that is itself produced as cheaply as necessary to achieve these features. Naturally, such garbage is constantly updated and all these giant bureaucracies are sieves with these updates running through them. Sure, if these bureaucracies hired competent people, downloaded open source tools, tested the tools themselves and essentially had their own quality control in-house, this might not have happened. But that wouldn't be the out-sourcing-based, cut costs and skills to the bone, neoliberal paradigm that's near and dear to the high level managers' heart, now would it? Now, you would think that an event like this would create a realization "what we do is too important for outsourcing, for bargain-basement, neoliberal style operations". But the Office of Personnel Management hack [1] was what should have created this realization and didn't. [1] https://en.wikipedia.org/wiki/Office_of_Personnel_Management_data_breach https://en.wikipedia.org/wiki/Office_of_Personnel_Management...
AnIdiotOnTheNet 6y ago
Considering how long bugs can go unfixed and undetected even in large open source projects, I think it can totally happen. Just create a backdoor that looks like an honest mistake, submit it in a PR that adds some feature or fix, and exploit it at will as people update. Heartbleed took over 2 years to find and fix.
hsbauauvhabzb 6y ago
A kernel breach happened on iirc an SVN server in 2013 but was detected almost immediately. If I were a nation state I wouldn’t try to poison mainline kernel - there would be far easier sources along the stack for both local and remote attacks which would more easily go unnoticed. Tools that come to mind are systemd, openssh, http/ftp services, GNU tools and common non-gnu shell utilities. Failing that, distribution level kernels would be my next bet purely because any commits would be less scrutinised.
- amelius 6y ago
  The attack surface is so large, I'm surprised we're not dealing with backdoors much more often (not just nation states, but also commercial hacker groups).
  - hsbauauvhabzb 6y ago
    Yeah, a tweet said this is the beginning of a decade of those sorts of attacks which I agree with. It’ll start with pip, npm, etc, then move to bigger targets. For that reason I’ve moved away from those managers and stuff like react (I trust facebook but dependency trees are huge) - the worst part is you can’t not patch, but you might be doomed by any upgrade. I think eventually it’ll snuff out innovation in medium sized businesses and government - large businesses can afford the cost of manual review and startups will ignore the risk, but middle-tier will be screwed. I’d love to see a crowdsourced review model, but I just don’t think it can be viable without getting abused.

by cs702 6y ago

xen2xen1 6y ago

Code was added once to Debian (IIRC) and it was detected almost immediately due to code signing.

newacct583 6y ago
The exploit in this case had access to the build (and presumably signing) system. That wouldn't have helped. The protection against this would have been the comparatively new efforts at reproducible builds. A modified binary, in theory, could be detected by current Fedora and Ubuntu releases (not sure about Debian or other distros). I don't think we've had an attack in practice though.
AnIdiotOnTheNet 6y ago
On the other hand, Debian broke OpenSSL generation and didn't detect it for almost 2 years. That appears to have been a legitimate mistake, but it is quite conceivable that a malicious actor gets a change merged that contains a backdoor that looks like an innocent mistake and goes undetected for a long time.

aquaticsunset 6y ago

As others (and Microsoft) mentioned, it was read only access. The only points of concern here would be if that statement somehow was not true and they were able to add undetected changes, or if their security audit process was severely lacking. But yeah, to your point - being able to read and analyze the Linux kernel source is considered a feature, not a liability :)

neodymiumphish 6y ago
I think you're connecting two points he made that weren't connected. On the one hand, open source projects make for an environment where bad actors could propose changes to the software that include these bug/backdoors. The benefit to the open source arena is that these changes can easily be analyzed and tested. In Microsoft's case, the source being visible but not editable is still a real risk (assuming the bad actor is able to extract the data they're viewing for further analysis), because they can use the source to determine avenues for attack. The fact that is was read-only does help ensure that no new attack vectors were created, but it still increases the chance of new attack vectors being found/used in the future.

deleted 6y ago

[deleted]

staticassertion 6y ago

I suspect adding bugdoors to Linux is far easier than it is than for Windows, but there are already so many bugs it's easier and more viable to just look for them than to try to insert them.

kerng 6y ago

I do security research and bug bounties on side sometimes and had read/write access to a couple of large open source projects in the past, incl. being able to impersonate employees from well known companies that work on open source stuff. Most common issue was access tokens found in public places. Would be interesting to know what happens when code is updated - which I obviously wouldn't do. Wonder how long it would take until caught. Since open source projects probably dont do "red teaming" (to use a fancy buzz word) I wonder how they could practice this?

wil421 6y ago

Why would you need to back door Linux when you can find a company like Solarwinds that is already in most networks with greater access to the network as a whole than a Linux server.

joe_the_user 6y ago

This hack wasn't really a failure of code construction but a failure of institutional practices. The same thing could have happened if SolarWinds had a garbagy sys admin tool that happened to also be open source but still otherwise followed the procedures of SolarWinds. Giant bureaucracies have a bunch of tasks they need to accomplish. Giant bureaucracies hire poorly trained people to accomplish those tasks and buy software to aid it's those people in accomplishing those tasks. The software is sold "by the feature" so it is colloquially "garbage" that is itself produced as cheaply as necessary to achieve these features. Naturally, such garbage is constantly updated and all these giant bureaucracies are sieves with these updates running through them. Sure, if these bureaucracies hired competent people, downloaded open source tools, tested the tools themselves and essentially had their own quality control in-house, this might not have happened. But that wouldn't be the out-sourcing-based, cut costs and skills to the bone, neoliberal paradigm that's near and dear to the high level managers' heart, now would it? Now, you would think that an event like this would create a realization "what we do is too important for outsourcing, for bargain-basement, neoliberal style operations". But the Office of Personnel Management hack [1] was what should have created this realization and didn't. [1] https://en.wikipedia.org/wiki/Office_of_Personnel_Management_data_breach https://en.wikipedia.org/wiki/Office_of_Personnel_Management...

AnIdiotOnTheNet 6y ago

Considering how long bugs can go unfixed and undetected even in large open source projects, I think it can totally happen. Just create a backdoor that looks like an honest mistake, submit it in a PR that adds some feature or fix, and exploit it at will as people update. Heartbleed took over 2 years to find and fix.

hsbauauvhabzb 6y ago

A kernel breach happened on iirc an SVN server in 2013 but was detected almost immediately. If I were a nation state I wouldn’t try to poison mainline kernel - there would be far easier sources along the stack for both local and remote attacks which would more easily go unnoticed. Tools that come to mind are systemd, openssh, http/ftp services, GNU tools and common non-gnu shell utilities. Failing that, distribution level kernels would be my next bet purely because any commits would be less scrutinised.

amelius 6y ago
The attack surface is so large, I'm surprised we're not dealing with backdoors much more often (not just nation states, but also commercial hacker groups).
- hsbauauvhabzb 6y ago
  Yeah, a tweet said this is the beginning of a decade of those sorts of attacks which I agree with. It’ll start with pip, npm, etc, then move to bigger targets. For that reason I’ve moved away from those managers and stuff like react (I trust facebook but dependency trees are huge) - the worst part is you can’t not patch, but you might be doomed by any upgrade. I think eventually it’ll snuff out innovation in medium sized businesses and government - large businesses can afford the cost of manual review and startups will ignore the risk, but middle-tier will be screwed. I’d love to see a crowdsourced review model, but I just don’t think it can be viable without getting abused.