7 ms·

"If you can't see anything because you can't get the source code... well, who knows what they might be hiding?" Actually, it's worse than that. Even if you ca

by cs702 13y ago

"If you can't see anything because you can't get the source code... well, who knows what they might be hiding?" Actually, it's worse than that. Even if you can review the source code, you STILL won't know for sure. As Ken Thompson put it three decades ago, "no amount of source-level verification or scrutiny will protect you from using untrusted code."[1] This is because the tools we all use to process or transform source code (e.g., already-compiled-to-binary compilers, assemblers, loaders, etc.) may already contain secret back-doors. Ditto for even lower-level tools like hardware microcode. While open-source software tools -- and open hardware too -- are much less likely to have secret back-doors embedded in them, there are no 100% guarantees. Ultimately we have little choice but to trust the people who create, package, and distribute all the software and hardware we use but didn't create ourselves from scratch. -- EDITS: Added last paragraph. Made minor edits to second paragraph so it more accurately conveys my thoughts. -- [1] http://cm.bell-labs.com/who/ken/trust.html http://cm.bell-labs.com/who/ken/trust.html

cperciva 13y ago
It's not quite as bad as it sounds: I don't think even the NSA has people smart enough to insert code into a compiler which will add backdoors to software which didn't yet exist when the compiler was written. If you're paranoid, use software written in the past 5 years and compile it on a system running FreeBSD 4.1 off a CD which you've had in your closet for the past decade.
- bvttf 13y ago
  I knew I was keeping all those discs for a reason!
- AnIrishDuck 13y ago
  > I don't think even the NSA has people smart enough to insert code into a compiler which will add backdoors to software which didn't yet exist when the compiler was written. I doubt such a thing is possible given the current state of the art in program analysis. Even if it were possible, it's hard to imagine it being undetectable (e.g. it would probably cause massive slowdowns in compilation). > If you're paranoid, use software written in the past 5 years and compile it on a system running FreeBSD 4.1 off a CD which you've had in your closet for the past decade. If you have a trusted compiler, you can bootstrap any non-trusted one [1]. That said, I suspect implementing diverse double-compiling to build any nontrivial system would take significant work. 1. http://www.dwheeler.com/trusting-trust/ http://www.dwheeler.com/trusting-trust/
  - russellsprouts 13y ago
    It's better than that -- you don't have to trust either compiler, just that they both don't have the same backdoor.
- sillysaurus2 13y ago
  I don't think even the NSA has people smart enough to insert code into a compiler which will add backdoors to software Really? Why wouldn't they? That's one of the most valuable attack vectors, precisely because it's so difficult. I wouldn't have guessed the NSA had people smart enough to break Windows Update's encryption using a brand-new cryptographic technique that also required several hundred thousand dollars of machine time to execute, but it happened nonetheless. We have to start thinking ahead, e.g. by making deterministic builds a standard procedure: https://blog.torproject.org/blog/deterministic-builds-part-one-cyberwar-and-global-compromise https://blog.torproject.org/blog/deterministic-builds-part-o...
  - ajross 13y ago
    > Why wouldn't they? That's one of the most valuable attack vectors, precisely because it's so difficult. If this logic made any sense then we should just give up, as their megabit-scale quantum computers have already cracked all keys on the planet. My guess is you forgot the bit of the sentence you clipped.
    - sillysaurus2 13y ago
      Deterministic builds do more than merely defend against the Trusting Trust attack (what you and cperciva are dismissing as a flight of fancy equivalent to worrying about megabit-scale quantum computers in 2013) --- they also prevent compromised open-source binaries, a much more serious and realistic attack vector. So we need to start implementing deterministic builds into every major open source project if we're even pretending like we care about putting up resistance to what's going on. If Tor browser can do it in just a few weekends, then so can we for mainstream Firefox, and hopefully eventually Ubuntu.
  - quasque 13y ago
    > I wouldn't have guessed the NSA had people smart enough to break Windows Update's encryption using a brand-new cryptographic technique that also required several hundred thousand dollars of machine time to execute, but it happened nonetheless. That's rather unsettling. Do you have any further information on this?
- _quasimodo 13y ago
  I dont think it is as hard as it it sounds. Consider, for example, hooking all fclose function calls and testing on every call whether * you have write permission on the file, * its an object file, * its in an architecture your exploit supports, * it uses the fclose function (or the corresponding system call, if it is linked statically) * and your exploit is not already present. If those conditions are true, hook the fclose calls in the object file before actually closing it, otherwise just close it normally.
  - PeterisP 13y ago
    If I understand your proposal correctly, it would be found out almost immediately, as file checksums would be altered and quite a few places do check them.
    - _quasimodo 13y ago
      My idea was that, if you have an object file open with write permissions, you can assume that it has been created/modified anyway, and people will expect the checksum to change. Of course this does not work if the checksum is calculated before the file is closed by your backdoored compiler.
- atmosx 13y ago
  Good, so don't give them ideas.
- MacsHeadroom 13y ago
  >don't think even the NSA has people smart enough to insert code into a compiler which will add backdoors to software which didn't yet exist when the compiler was written. Oddly enough, that is almost exactly how the Flame malware worked. It injected itself into a compiler, causing the compiler to silently compile malicious code into custom firmware being compiled for nuclear reactor control equipment. So at least one state actor has done just that at least once.
- harrytuttle 13y ago
  GCC's optimiser does some crazy shit sometimes including removing necessary code in -O3. Perhaps some of that crazy could be related? As for using FreeBSD 4.1 off a CD in your closet, I recently set up an NT4 machine with SP2 (requirement!) that was to be used entirely offline. Surprisngly it booted and works fine on a brand new Intel chipset H61 based machine with an SSD in it (graphics are stuck at 800x600x16 but that's a requirement too).
edwintorok 13y ago
If you have two compilers (and you trust one) you can counter the 'trusting trust' type attack: http://www.dwheeler.com/trusting-trust/ http://www.dwheeler.com/trusting-trust/ Even if you don't fully trust both compilers, the 'diverse-double compiling' is still useful: "Finally, note that the “trusted” compiler(s) could be malicious and still work well well for DDC. We just need justified confidence that any triggers or payloads in a trusted compiler do not affect the DDC process when applied to the compiler-under-test. That is much, much easier to justify."
malandrew 13y ago
What I've always wondered is how hard it is to have both a production system running some software on some hardware both of which may or may not have a backdoor and have a sentry system running next to it on different hardware that is somehow watching the first system for traffic anomalies not aligned with how the first system should be functioning. On top of that the first production system also watches the sentry for signs of disabling or tampering. With two machines watching each other, I would imagine that it's much harder to use any backdoor without being detected and shut down. In a similar vein, would it not be possible to spot hardware backdoors in PRNG by running many identical encryption tests across many different types of hardware looking for one that doesn't behave like the others? More generally, what tools exist for the automatic detection and mitigation of backdoors?
- konstruktor 13y ago
  On the Space Shuttle, four computers with identical software ran redundantly, and a fifth with independent software was used to detect errors. http://en.wikipedia.org/wiki/Space_Shuttle http://en.wikipedia.org/wiki/Space_Shuttle http://www.hq.nasa.gov/pao/History/computers/Ch4-4.html http://www.hq.nasa.gov/pao/History/computers/Ch4-4.html This was, of course, very expensive, and made software changes a very difficult and slow process.
- mje__ 13y ago
  It doesn't mitigate untrusted software, but an interesting approach to tamper resistance can be found in Infineon SLE78 security controllers (for smart cards, etc). They run dual CPUs out-of-sync and compare results after each instruction. http://www.infineon.com/cms/en/product/chip-card-and-security-ics/security-controller-contact-based/sle-78-solid-flash-tm-family-digital-security-controller/channel.html?channel=db3a30431ce5fb52011d47b166342af0 http://www.infineon.com/cms/en/product/chip-card-and-securit...
Spooky23 13y ago
Be careful about magical thinking here. They aren't omniscient supermen. The materials on the Guardian implied that these exploits were "fragile". Given that statement and the insinuations that the NSA had folks participating on standards discussions basically gumming up the works for things like IPSec, NIST standards, etc (effectively crowding the people who know what they are doing out of the room), it's likely that they're created circumstances where accepting defaults or not really understanding how to implement various technologies creates attack surfaces or weaknesses that the NSA has known means to exploit. Like anything else, you need to think about the risks and controls for whatever your doing. If you're protecting the interests of a government or company likely to be spied upon, you probably want to factor in the ability of a nation-state actor to potentially intercept data into your risk calculus and operational strategy. For the rest of us, you need to think about what the risks really are -- if you're a politician or other visible individual, you need to be very mindful of your communications privacy and practices. (Google "Anthony Weiner"). If you're posting on Hacker News and buying stuff from Amazon, the NSA decrypting that is a low-risk threat.
mpyne 13y ago
Open source software represents what is IMHO the best combination between ease of self-verification and total security. In other words if you have only a little bit of time/resources/etc. to put into verification so that you don't have to rely on trust, the software that you're running is probably the best place to put that time and energy. It's probably possible to verify closed-source binaries by decompilation and a lot of effort, if you wish. But on the other hand between compilers/hardware/OS/etc. the part of the computer that is most susceptible to being attacked (the application software) is the part that's easiest to audit with open source. If your chip's built-in RNG is cracked you can simply refuse to use it. But if your libopenssl.so or CRYPTAPI.DLL is cracked you're probably screwed. It's at least somewhat straightforward to build your own libopenssl.so (or at the very least, verify that your distro's signed package RPM builds the same sha1sum from the known-good source tarball + patches). There's nothing you can do to independently verify the source used for CRYPTAPI.DLL, and even the binary of that DLL requires trusting Microsoft (and figuring out which Service Pack, KB patchset, etc. are installed). However in the end you do have to trust people no matter what you do, so if you're willing to trust FooCorp's binaries then just go into decision with eyes open. Even most open source users are effectively no different in that they simply trust what Fedora or Debian are pushing out there. But even one alert user is enough to catch a hacked .deb or .rpm, so you don't need most of the users to be paranoid. It's kind of a reverse herd immunity, if you will.
- danielweber 13y ago
  I've seen security problems sit in open-source code for years because no one was interested enough in going through it. If you are talking emacs, or the kernel, or gcc, there is probably enough interest there (and strong personalities in charge of the projects) to keep things safe. I don't know where the cutoff is. At thousands of users you probably can't count on someone else having looked through it. At millions you can.
  - mpyne 13y ago
    That's certainty true. I can tell you as an OSS dev that that are simply not enough eyeballs for all bugs to be shallow. However you don't have even the possibility with closed-source products. That means you must instead rely on the process used to create the software. So something by MS might actually be pretty trustworthy from a "security issues not accidentally introduced" standpoint at this point after years of improvement on secure coding practices. But they still operate by a profit motive, and there's still the possibility of deliberate introduction of security issues when it suits business purposes or for legal compliance. Unfortunately there's no great way to tell that an OSS project from RANDOM_DEV is well-coded without looking at the code and it may be that a well-intentioned but junior dev introduces a relative swath of security bugs by accident. I don't know where the cutoff is either. It may depend more on your threat model than anything else.
maaku 13y ago
Look into Gitian, which is used by the bitcoin developers to create deterministic, verifiable builds that can be compared by multiple developers on multiple machines before doing a binary release. In theory, the process could be extended to an entire *nix distribution (and there is interest in doing so).

by cs702 13y ago

cperciva 13y ago

It's not quite as bad as it sounds: I don't think even the NSA has people smart enough to insert code into a compiler which will add backdoors to software which didn't yet exist when the compiler was written. If you're paranoid, use software written in the past 5 years and compile it on a system running FreeBSD 4.1 off a CD which you've had in your closet for the past decade.

bvttf 13y ago
I knew I was keeping all those discs for a reason!
AnIrishDuck 13y ago
> I don't think even the NSA has people smart enough to insert code into a compiler which will add backdoors to software which didn't yet exist when the compiler was written. I doubt such a thing is possible given the current state of the art in program analysis. Even if it were possible, it's hard to imagine it being undetectable (e.g. it would probably cause massive slowdowns in compilation). > If you're paranoid, use software written in the past 5 years and compile it on a system running FreeBSD 4.1 off a CD which you've had in your closet for the past decade. If you have a trusted compiler, you can bootstrap any non-trusted one [1]. That said, I suspect implementing diverse double-compiling to build any nontrivial system would take significant work. 1. http://www.dwheeler.com/trusting-trust/ http://www.dwheeler.com/trusting-trust/
- russellsprouts 13y ago
  It's better than that -- you don't have to trust either compiler, just that they both don't have the same backdoor.
sillysaurus2 13y ago
I don't think even the NSA has people smart enough to insert code into a compiler which will add backdoors to software Really? Why wouldn't they? That's one of the most valuable attack vectors, precisely because it's so difficult. I wouldn't have guessed the NSA had people smart enough to break Windows Update's encryption using a brand-new cryptographic technique that also required several hundred thousand dollars of machine time to execute, but it happened nonetheless. We have to start thinking ahead, e.g. by making deterministic builds a standard procedure: https://blog.torproject.org/blog/deterministic-builds-part-one-cyberwar-and-global-compromise https://blog.torproject.org/blog/deterministic-builds-part-o...
- ajross 13y ago
  > Why wouldn't they? That's one of the most valuable attack vectors, precisely because it's so difficult. If this logic made any sense then we should just give up, as their megabit-scale quantum computers have already cracked all keys on the planet. My guess is you forgot the bit of the sentence you clipped.
  - sillysaurus2 13y ago
    Deterministic builds do more than merely defend against the Trusting Trust attack (what you and cperciva are dismissing as a flight of fancy equivalent to worrying about megabit-scale quantum computers in 2013) --- they also prevent compromised open-source binaries, a much more serious and realistic attack vector. So we need to start implementing deterministic builds into every major open source project if we're even pretending like we care about putting up resistance to what's going on. If Tor browser can do it in just a few weekends, then so can we for mainstream Firefox, and hopefully eventually Ubuntu.
- quasque 13y ago
  > I wouldn't have guessed the NSA had people smart enough to break Windows Update's encryption using a brand-new cryptographic technique that also required several hundred thousand dollars of machine time to execute, but it happened nonetheless. That's rather unsettling. Do you have any further information on this?
_quasimodo 13y ago
I dont think it is as hard as it it sounds. Consider, for example, hooking all fclose function calls and testing on every call whether * you have write permission on the file, * its an object file, * its in an architecture your exploit supports, * it uses the fclose function (or the corresponding system call, if it is linked statically) * and your exploit is not already present. If those conditions are true, hook the fclose calls in the object file before actually closing it, otherwise just close it normally.
- PeterisP 13y ago
  If I understand your proposal correctly, it would be found out almost immediately, as file checksums would be altered and quite a few places do check them.
  - _quasimodo 13y ago
    My idea was that, if you have an object file open with write permissions, you can assume that it has been created/modified anyway, and people will expect the checksum to change. Of course this does not work if the checksum is calculated before the file is closed by your backdoored compiler.
atmosx 13y ago
Good, so don't give them ideas.
MacsHeadroom 13y ago
>don't think even the NSA has people smart enough to insert code into a compiler which will add backdoors to software which didn't yet exist when the compiler was written. Oddly enough, that is almost exactly how the Flame malware worked. It injected itself into a compiler, causing the compiler to silently compile malicious code into custom firmware being compiled for nuclear reactor control equipment. So at least one state actor has done just that at least once.
harrytuttle 13y ago
GCC's optimiser does some crazy shit sometimes including removing necessary code in -O3. Perhaps some of that crazy could be related? As for using FreeBSD 4.1 off a CD in your closet, I recently set up an NT4 machine with SP2 (requirement!) that was to be used entirely offline. Surprisngly it booted and works fine on a brand new Intel chipset H61 based machine with an SSD in it (graphics are stuck at 800x600x16 but that's a requirement too).

edwintorok 13y ago

If you have two compilers (and you trust one) you can counter the 'trusting trust' type attack: http://www.dwheeler.com/trusting-trust/ http://www.dwheeler.com/trusting-trust/ Even if you don't fully trust both compilers, the 'diverse-double compiling' is still useful: "Finally, note that the “trusted” compiler(s) could be malicious and still work well well for DDC. We just need justified confidence that any triggers or payloads in a trusted compiler do not affect the DDC process when applied to the compiler-under-test. That is much, much easier to justify."

malandrew 13y ago

What I've always wondered is how hard it is to have both a production system running some software on some hardware both of which may or may not have a backdoor and have a sentry system running next to it on different hardware that is somehow watching the first system for traffic anomalies not aligned with how the first system should be functioning. On top of that the first production system also watches the sentry for signs of disabling or tampering. With two machines watching each other, I would imagine that it's much harder to use any backdoor without being detected and shut down. In a similar vein, would it not be possible to spot hardware backdoors in PRNG by running many identical encryption tests across many different types of hardware looking for one that doesn't behave like the others? More generally, what tools exist for the automatic detection and mitigation of backdoors?

konstruktor 13y ago
On the Space Shuttle, four computers with identical software ran redundantly, and a fifth with independent software was used to detect errors. http://en.wikipedia.org/wiki/Space_Shuttle http://en.wikipedia.org/wiki/Space_Shuttle http://www.hq.nasa.gov/pao/History/computers/Ch4-4.html http://www.hq.nasa.gov/pao/History/computers/Ch4-4.html This was, of course, very expensive, and made software changes a very difficult and slow process.
mje__ 13y ago
It doesn't mitigate untrusted software, but an interesting approach to tamper resistance can be found in Infineon SLE78 security controllers (for smart cards, etc). They run dual CPUs out-of-sync and compare results after each instruction. http://www.infineon.com/cms/en/product/chip-card-and-security-ics/security-controller-contact-based/sle-78-solid-flash-tm-family-digital-security-controller/channel.html?channel=db3a30431ce5fb52011d47b166342af0 http://www.infineon.com/cms/en/product/chip-card-and-securit...

Spooky23 13y ago

Be careful about magical thinking here. They aren't omniscient supermen. The materials on the Guardian implied that these exploits were "fragile". Given that statement and the insinuations that the NSA had folks participating on standards discussions basically gumming up the works for things like IPSec, NIST standards, etc (effectively crowding the people who know what they are doing out of the room), it's likely that they're created circumstances where accepting defaults or not really understanding how to implement various technologies creates attack surfaces or weaknesses that the NSA has known means to exploit. Like anything else, you need to think about the risks and controls for whatever your doing. If you're protecting the interests of a government or company likely to be spied upon, you probably want to factor in the ability of a nation-state actor to potentially intercept data into your risk calculus and operational strategy. For the rest of us, you need to think about what the risks really are -- if you're a politician or other visible individual, you need to be very mindful of your communications privacy and practices. (Google "Anthony Weiner"). If you're posting on Hacker News and buying stuff from Amazon, the NSA decrypting that is a low-risk threat.

mpyne 13y ago

Open source software represents what is IMHO the best combination between ease of self-verification and total security. In other words if you have only a little bit of time/resources/etc. to put into verification so that you don't have to rely on trust, the software that you're running is probably the best place to put that time and energy. It's probably possible to verify closed-source binaries by decompilation and a lot of effort, if you wish. But on the other hand between compilers/hardware/OS/etc. the part of the computer that is most susceptible to being attacked (the application software) is the part that's easiest to audit with open source. If your chip's built-in RNG is cracked you can simply refuse to use it. But if your libopenssl.so or CRYPTAPI.DLL is cracked you're probably screwed. It's at least somewhat straightforward to build your own libopenssl.so (or at the very least, verify that your distro's signed package RPM builds the same sha1sum from the known-good source tarball + patches). There's nothing you can do to independently verify the source used for CRYPTAPI.DLL, and even the binary of that DLL requires trusting Microsoft (and figuring out which Service Pack, KB patchset, etc. are installed). However in the end you do have to trust people no matter what you do, so if you're willing to trust FooCorp's binaries then just go into decision with eyes open. Even most open source users are effectively no different in that they simply trust what Fedora or Debian are pushing out there. But even one alert user is enough to catch a hacked .deb or .rpm, so you don't need most of the users to be paranoid. It's kind of a reverse herd immunity, if you will.

danielweber 13y ago
I've seen security problems sit in open-source code for years because no one was interested enough in going through it. If you are talking emacs, or the kernel, or gcc, there is probably enough interest there (and strong personalities in charge of the projects) to keep things safe. I don't know where the cutoff is. At thousands of users you probably can't count on someone else having looked through it. At millions you can.
- mpyne 13y ago
  That's certainty true. I can tell you as an OSS dev that that are simply not enough eyeballs for all bugs to be shallow. However you don't have even the possibility with closed-source products. That means you must instead rely on the process used to create the software. So something by MS might actually be pretty trustworthy from a "security issues not accidentally introduced" standpoint at this point after years of improvement on secure coding practices. But they still operate by a profit motive, and there's still the possibility of deliberate introduction of security issues when it suits business purposes or for legal compliance. Unfortunately there's no great way to tell that an OSS project from RANDOM_DEV is well-coded without looking at the code and it may be that a well-intentioned but junior dev introduces a relative swath of security bugs by accident. I don't know where the cutoff is either. It may depend more on your threat model than anything else.

maaku 13y ago

Look into Gitian, which is used by the bitcoin developers to create deterministic, verifiable builds that can be compared by multiple developers on multiple machines before doing a binary release. In theory, the process could be extended to an entire *nix distribution (and there is interest in doing so).