6 ms·

The lesson isn’t that any particular victim sucks at security, it’s that well-resourced targeted attacks are generally unstoppable.

jakeva 6y ago
May I propose a corollary: we all suck at security when confronted by a well sourced adversary.
varjag 6y ago
Very much so. The description of the staged attack suggests months of work by a sizeable development team.
joe_the_user 6y ago
I think you can put things as modern baroque software stacks with their effectively vast "attack surfaces" are not going to stand-up to a well-financed, patient, skilled attacker. I recall that many companies have switched from a perimeter model of defense where systems are secured from the outside to a "defense in depth" model where each system is secured on it's own (plus the perimeter). Perhaps folks should think about tightening the in-depth model and avoiding the consumer model of constant updates from a zillion providers. Or perhaps a single lab could verify updates of the zillion providers rather than leaving them on their own.
- xvector 6y ago
  > I recall that many companies have switched from a perimeter model of defense where systems are secured from the outside to a "defense in depth" model where each system is secured on it's own (plus the perimeter). Yes, this is the current bleeding edge of cloud security, known as “zero trust.” Amongst other things, it usually involves provisioning mTLS identities in a secured manner to each service, with connections restricted to a whitelisted set of identities. I found Evan Gilman and Doug Barth’s “Zero Trust Networks: Building Secure Systems in Untrusted Networks” [1] a pretty helpful read in understanding what modern/next-gen cloud security looks like. Some modern implementations of varying depth and scale include SPIRE [2], Tailscale [3], and BeyondCorp [4]. -—- [1]: https://www.amazon.com/Zero-Trust-Networks-Building-Untrusted/dp/1491962194 https://www.amazon.com/Zero-Trust-Networks-Building-Untruste... [2]: https://spiffe.io/ https://spiffe.io/ [3]: https://tailscale.com/ https://tailscale.com/ [4]: https://beyondcorp.com/ https://beyondcorp.com/
  - joe_the_user 6y ago
    Yeah, but if you're blindly installing a third party's binary blob, it's hard to call that "zero trust". Edit: It seems like a serious extension of the zero trust concept would involve something like "only allow in source code into the system from sources we trust and then we compile it ourselves". Limit trust to trusted identities and don't allow binaries in any more than people.
    - xvector 6y ago
      Of course, SDLC security and binary verification is also part of zero-trust systems. But you have to keep in mind - the whole point of zero-trust systems is that even if a malicious third-party blob is present on some nodes, it's blast radius is limited by the whitelist of clients that it can communicate with, which makes exfiltrating information particularly difficult unless you have additional exploits that let you move laterally.
      - joe_the_user 6y ago
        I am suggesting you could limit "binary verification" and just allow "sources you compile yourself", 'cause how much can a binary be verified? (enough to know your blob was hacked at the source and not along the way, yeah but as we see, that's not enough). What describing just involves "we make sure our vendors are sort of trying to be secure" and that's about it 'cause that's all you can be when someone hands you a binary. And sure, there exist things like Linux drivers that are binary blobs in source code form but at least directs one's eyes to such things. Edit: Also, even in these environments, some people and some software has to run the network itself, like a sys admin tool. Which makes sys admin tools something everyone should look at. And brings us to the present hack.
        bargle0 6y ago
        How much do you trust your compiler?
        6y ago
- Pokepokalypse 6y ago
  This is the real issue. We can hire all the "security experts" in the world, but if software and architecture are fatally flawed from the getgo, you can check all the checkboxes, and scan and produce all the pretty reports you want. The end result is still paper mache
- im3w1l 6y ago
  The only way out of this mess I see is throwing everything away and starting from scratch, using provably (machine verifiable proofs) correct software. I'm thinking of properties like "for any api input the reply is independent of $SECRET".
  - tsimionescu 6y ago
    If you started today, 50 years from now you may be able to run vim on such a system, but not sooner.
    - d3ckard 6y ago
      Original Unix was created in a couple of years. While your point has merit, you’re overestimating by much.
      - tsimionescu 6y ago
        Original Unix wasn't formally verified. seL4 took ~8 years from initial design to release (though apparently the verification was done after 3 years?), and it is a code base smaller than TypeScript. CompCert started in 2005 and it still can't compile all of C, and it doesn't go past the level of Gcc's -O1 optimizations. Formal verification is an extreme cost amplifier, and the cost is super-linear with the size of the code base. I don't think my estimate is unrealistic, depending on exactly what you would expect to have running before vi (I was assuming a complete OS including a kernel, device drivers, a C stdlib or equivalent, and a userspace similar to the GNU tools).
- indymike 6y ago
  The consumer model got where it is because leaving software proven vulnerable in place is worse than the risk of a potential supply chain attack.
- phendrenad2 6y ago
  I'm always amazed that the top-tier security firms use... Linux. And the Militaries of the world use Windows (like come on). You'd think there would be some crazy million-dollar-per-license impenetrable operating system out there. The race to the bottom is palpable.
Veserv 6y ago
True, but it is important to quantify that inadequacy. SolarWinds was actually attacked by an actual threat and their defenses were comically outmatched by that real threat who found real value in attacking them. We are not talking 10%-20% or even 100%, we are talking systems that need to improve by 1,000%-10,000% to provide credible defense against real foes. And this is not just SolarWinds, FireEye, a major cybersecurity company, needs to improve their security by a factor of 100x to protect their customers against people who actually wish to attack them. The security is not merely inadequate, it is inadequate to a degree that is almost mind-boggling. Systems are being deployed that are not even 1% of the necessary level of effectiveness. These organizations are completely incapable of developing and deploying adequate defenses. This ignores the secondary problem which is that if the attacks being deployed are 100x stronger than the defenses, how hard is it to develop an attack that is merely 2x stronger than the defenses. If we lazily extrapolate this linearly, that would be 1/50th the resources to develop an attack that still outmatches the defenders. How many people do you think were on the SolarWinds attack? 10, 100, 1000? Even at 1000 that means you would only need 20 fulltime people for a year to develop a credible attack against nearly every Fortune 500 company in the world and most of the US government. That should be a terrifying concept. Obviously, this is lazy extrapolation, but it is not so off as to be non-illustrative of this problem. Given this immense difference between real problems and available solutions, the only reasonable assumption for companies and people is to assume that they are completely defenseless against such actors and likely even largely defenseless against even poorly-resourced attacks as demonstrated time and time again. It is imperative that they act accordingly and only make systems accessible where the harm of guaranteed breach is less than the benefit of making those systems accessible.
- jrochkind1 6y ago
  I don't understand how you are measuring "security" quantitatively to say "1,000% - 10,000%" and "100x". But yeah, security is in a terrible spot, throughout the software industry. It is not clear to me what the steps would be to improve it. It is not clear to me whether these steps would increase software costs (something that is quantitative) by "1,000% - 10,000%", and if so how society as a whole could possibly afford it, in this society where we have put (insecure) software everywhere.
x0x0 6y ago
Or, this is a desperate ploy by SolarWinds to cover up and make excuses for their incompetence in the hopes that the market doesn't give their company the death penalty. Like sure, securing yourself against a government is hard. But when a security researcher told solarwinds an update server was accessible with the password 'solarwinds123' and they sat on that report for 3 days [1], my confidence levels that they were generally competent are not high. > Security researcher Vinoth Kumar told Reuters that, last year, he alerted the company that anyone could access SolarWinds’ update server by using the password “solarwinds123” [1] and > When reached for comment by Newsweek, Kumar forwarded his email correspondence with SolarWinds. He first notified the company of the issue on November 19, 2019. SolarWinds' information security team responded a few days later on November 22, 2019. [2] and > Others - including Kyle Hanslovan, the cofounder of Maryland-based cybersecurity company Huntress - noticed that, days after SolarWinds realized their software had been compromised, the malicious updates were still available for download. [1] Did the company with the above security practices get more important stuff right? Like sure... maybe? But probably not. [1] https://www.reuters.com/article/global-cyber-solarwinds/hackers-at-center-of-sprawling-spy-campaign-turned-solarwinds-dominance-against-it-idUSKBN28P2N8 https://www.reuters.com/article/global-cyber-solarwinds/hack... [2] https://www.newsweek.com/solarwinds-update-server-could-accessed-2019-using-password-solarwinds123-report-1554986 https://www.newsweek.com/solarwinds-update-server-could-acce...
GSGBen 6y ago
No mention of using an "air gap" in any of the comments here. Air-gapped networks are a pain to work with but are incredibly effective. I still haven't read anything on how the attackers actually reached the build server though, so it's hard to say if it would have worked in this specific case. I wonder if the lack of a clear statement of how they made the initial breach is due to - bad searching on my part - they still don't know - they're still trying to mitigate it - they're still trying to word it in a PR-approved way
- dublin 6y ago
  An alternative to full air-gapping is gateways that pass only the traffic they're supposed to. One example: For many years before they were bought by HP, Compaq's mail system had a regular Internet-connected SMTP MTA that fed two back-to-back hardwired uucp MTAs that connected to another SMTP MTA that fed their internal network. This was the early days of the public Internet, and they didn't want to have to trust that their SMTP MTA was secure, so they just made sure nothing but mail could ever get through their mail pipeline. It's awkward and a bit of a pain to set up and manage, but enterprise companies have the resources to do things like that. And it did actually provide state-of-the-art email security for its time. (FWIW, I didn't work with Compaq, but compared notes with their network architect on an occasional basis...)
hanselot 6y ago
Why do you think OMB is so relaxed right now? Yes the implication is, if you think the things that have been made public so far are bad, the show is really only about to start. Soon OMB will be on the screen of every device in the world revealing the things we have all convinced ourselves are too fringe and schizophrenic to be real. Is this attack related? Who can say, but for sure it is conveniently timed, no? Just 7 more days. Screencap this.

jakeva 6y ago

May I propose a corollary: we all suck at security when confronted by a well sourced adversary.

varjag 6y ago

Very much so. The description of the staged attack suggests months of work by a sizeable development team.

joe_the_user 6y ago

I think you can put things as modern baroque software stacks with their effectively vast "attack surfaces" are not going to stand-up to a well-financed, patient, skilled attacker. I recall that many companies have switched from a perimeter model of defense where systems are secured from the outside to a "defense in depth" model where each system is secured on it's own (plus the perimeter). Perhaps folks should think about tightening the in-depth model and avoiding the consumer model of constant updates from a zillion providers. Or perhaps a single lab could verify updates of the zillion providers rather than leaving them on their own.

xvector 6y ago
> I recall that many companies have switched from a perimeter model of defense where systems are secured from the outside to a "defense in depth" model where each system is secured on it's own (plus the perimeter). Yes, this is the current bleeding edge of cloud security, known as “zero trust.” Amongst other things, it usually involves provisioning mTLS identities in a secured manner to each service, with connections restricted to a whitelisted set of identities. I found Evan Gilman and Doug Barth’s “Zero Trust Networks: Building Secure Systems in Untrusted Networks” [1] a pretty helpful read in understanding what modern/next-gen cloud security looks like. Some modern implementations of varying depth and scale include SPIRE [2], Tailscale [3], and BeyondCorp [4]. -—- [1]: https://www.amazon.com/Zero-Trust-Networks-Building-Untrusted/dp/1491962194 https://www.amazon.com/Zero-Trust-Networks-Building-Untruste... [2]: https://spiffe.io/ https://spiffe.io/ [3]: https://tailscale.com/ https://tailscale.com/ [4]: https://beyondcorp.com/ https://beyondcorp.com/
- joe_the_user 6y ago
  Yeah, but if you're blindly installing a third party's binary blob, it's hard to call that "zero trust". Edit: It seems like a serious extension of the zero trust concept would involve something like "only allow in source code into the system from sources we trust and then we compile it ourselves". Limit trust to trusted identities and don't allow binaries in any more than people.
  - xvector 6y ago
    Of course, SDLC security and binary verification is also part of zero-trust systems. But you have to keep in mind - the whole point of zero-trust systems is that even if a malicious third-party blob is present on some nodes, it's blast radius is limited by the whitelist of clients that it can communicate with, which makes exfiltrating information particularly difficult unless you have additional exploits that let you move laterally.
    - joe_the_user 6y ago
      I am suggesting you could limit "binary verification" and just allow "sources you compile yourself", 'cause how much can a binary be verified? (enough to know your blob was hacked at the source and not along the way, yeah but as we see, that's not enough). What describing just involves "we make sure our vendors are sort of trying to be secure" and that's about it 'cause that's all you can be when someone hands you a binary. And sure, there exist things like Linux drivers that are binary blobs in source code form but at least directs one's eyes to such things. Edit: Also, even in these environments, some people and some software has to run the network itself, like a sys admin tool. Which makes sys admin tools something everyone should look at. And brings us to the present hack.
      - bargle0 6y ago
        How much do you trust your compiler?
        6y ago
Pokepokalypse 6y ago
This is the real issue. We can hire all the "security experts" in the world, but if software and architecture are fatally flawed from the getgo, you can check all the checkboxes, and scan and produce all the pretty reports you want. The end result is still paper mache
im3w1l 6y ago
The only way out of this mess I see is throwing everything away and starting from scratch, using provably (machine verifiable proofs) correct software. I'm thinking of properties like "for any api input the reply is independent of $SECRET".
- tsimionescu 6y ago
  If you started today, 50 years from now you may be able to run vim on such a system, but not sooner.
  - d3ckard 6y ago
    Original Unix was created in a couple of years. While your point has merit, you’re overestimating by much.
    - tsimionescu 6y ago
      Original Unix wasn't formally verified. seL4 took ~8 years from initial design to release (though apparently the verification was done after 3 years?), and it is a code base smaller than TypeScript. CompCert started in 2005 and it still can't compile all of C, and it doesn't go past the level of Gcc's -O1 optimizations. Formal verification is an extreme cost amplifier, and the cost is super-linear with the size of the code base. I don't think my estimate is unrealistic, depending on exactly what you would expect to have running before vi (I was assuming a complete OS including a kernel, device drivers, a C stdlib or equivalent, and a userspace similar to the GNU tools).
indymike 6y ago
The consumer model got where it is because leaving software proven vulnerable in place is worse than the risk of a potential supply chain attack.
phendrenad2 6y ago
I'm always amazed that the top-tier security firms use... Linux. And the Militaries of the world use Windows (like come on). You'd think there would be some crazy million-dollar-per-license impenetrable operating system out there. The race to the bottom is palpable.

Veserv 6y ago

True, but it is important to quantify that inadequacy. SolarWinds was actually attacked by an actual threat and their defenses were comically outmatched by that real threat who found real value in attacking them. We are not talking 10%-20% or even 100%, we are talking systems that need to improve by 1,000%-10,000% to provide credible defense against real foes. And this is not just SolarWinds, FireEye, a major cybersecurity company, needs to improve their security by a factor of 100x to protect their customers against people who actually wish to attack them. The security is not merely inadequate, it is inadequate to a degree that is almost mind-boggling. Systems are being deployed that are not even 1% of the necessary level of effectiveness. These organizations are completely incapable of developing and deploying adequate defenses. This ignores the secondary problem which is that if the attacks being deployed are 100x stronger than the defenses, how hard is it to develop an attack that is merely 2x stronger than the defenses. If we lazily extrapolate this linearly, that would be 1/50th the resources to develop an attack that still outmatches the defenders. How many people do you think were on the SolarWinds attack? 10, 100, 1000? Even at 1000 that means you would only need 20 fulltime people for a year to develop a credible attack against nearly every Fortune 500 company in the world and most of the US government. That should be a terrifying concept. Obviously, this is lazy extrapolation, but it is not so off as to be non-illustrative of this problem. Given this immense difference between real problems and available solutions, the only reasonable assumption for companies and people is to assume that they are completely defenseless against such actors and likely even largely defenseless against even poorly-resourced attacks as demonstrated time and time again. It is imperative that they act accordingly and only make systems accessible where the harm of guaranteed breach is less than the benefit of making those systems accessible.

jrochkind1 6y ago
I don't understand how you are measuring "security" quantitatively to say "1,000% - 10,000%" and "100x". But yeah, security is in a terrible spot, throughout the software industry. It is not clear to me what the steps would be to improve it. It is not clear to me whether these steps would increase software costs (something that is quantitative) by "1,000% - 10,000%", and if so how society as a whole could possibly afford it, in this society where we have put (insecure) software everywhere.

x0x0 6y ago

Or, this is a desperate ploy by SolarWinds to cover up and make excuses for their incompetence in the hopes that the market doesn't give their company the death penalty. Like sure, securing yourself against a government is hard. But when a security researcher told solarwinds an update server was accessible with the password 'solarwinds123' and they sat on that report for 3 days [1], my confidence levels that they were generally competent are not high. > Security researcher Vinoth Kumar told Reuters that, last year, he alerted the company that anyone could access SolarWinds’ update server by using the password “solarwinds123” [1] and > When reached for comment by Newsweek, Kumar forwarded his email correspondence with SolarWinds. He first notified the company of the issue on November 19, 2019. SolarWinds' information security team responded a few days later on November 22, 2019. [2] and > Others - including Kyle Hanslovan, the cofounder of Maryland-based cybersecurity company Huntress - noticed that, days after SolarWinds realized their software had been compromised, the malicious updates were still available for download. [1] Did the company with the above security practices get more important stuff right? Like sure... maybe? But probably not. [1] https://www.reuters.com/article/global-cyber-solarwinds/hackers-at-center-of-sprawling-spy-campaign-turned-solarwinds-dominance-against-it-idUSKBN28P2N8 https://www.reuters.com/article/global-cyber-solarwinds/hack... [2] https://www.newsweek.com/solarwinds-update-server-could-accessed-2019-using-password-solarwinds123-report-1554986 https://www.newsweek.com/solarwinds-update-server-could-acce...

GSGBen 6y ago

No mention of using an "air gap" in any of the comments here. Air-gapped networks are a pain to work with but are incredibly effective. I still haven't read anything on how the attackers actually reached the build server though, so it's hard to say if it would have worked in this specific case. I wonder if the lack of a clear statement of how they made the initial breach is due to - bad searching on my part - they still don't know - they're still trying to mitigate it - they're still trying to word it in a PR-approved way

dublin 6y ago
An alternative to full air-gapping is gateways that pass only the traffic they're supposed to. One example: For many years before they were bought by HP, Compaq's mail system had a regular Internet-connected SMTP MTA that fed two back-to-back hardwired uucp MTAs that connected to another SMTP MTA that fed their internal network. This was the early days of the public Internet, and they didn't want to have to trust that their SMTP MTA was secure, so they just made sure nothing but mail could ever get through their mail pipeline. It's awkward and a bit of a pain to set up and manage, but enterprise companies have the resources to do things like that. And it did actually provide state-of-the-art email security for its time. (FWIW, I didn't work with Compaq, but compared notes with their network architect on an occasional basis...)

hanselot 6y ago

Why do you think OMB is so relaxed right now? Yes the implication is, if you think the things that have been made public so far are bad, the show is really only about to start. Soon OMB will be on the screen of every device in the world revealing the things we have all convinced ourselves are too fringe and schizophrenic to be real. Is this attack related? Who can say, but for sure it is conveniently timed, no? Just 7 more days. Screencap this.