8 ms·

Oh good. The attackers were only in their systems since 9/4/19 before being detected on 12/12/20, so only 15 months of infiltration into SolarWinds' systems bef

by Veserv 6y ago

Oh good. The attackers were only in their systems since 9/4/19 before being detected on 12/12/20, so only 15 months of infiltration into SolarWinds' systems before detection. At least the payload was only deployed 2/20/20, so their customers were only completely infiltrated without detection for 8 months. Assuming the attackers could only get a 10 MB/s channel total per target even though they probably infected thousands to tens of thousands of machines per target, at ~20 million seconds that would constitute ~200 TB exfiltrated per customer or ~19 years of 1080p video. If an attacker has just one day to root around and exfiltrate they can easily get valuable information. If they are given 8 months they have already gotten everything of value for months and are just waiting around for any new data to come in. Think how inadequate your systems must be to let an attacker sit around in your systems for 8 months, it is mind-boggling how unqualified their systems are for their problems. And this is not just an indictment of SolarWinds. Just in case anybody forgets, it was the top-flight security company FireEye who discovered this breach after realizing they themselves were breached. A "best of the best" security company took 8 months before realizing that they or any of their customers had been breached. This is what "top-class" security buys you.

vasco 6y ago
Whenever I read a post as strongly worded as yours I wonder that its author must really be the best security minded engineer out there, and has never compromised on anything for a deadline or missed something because they simply didn't know it should be configured in a particular safe way. This definitely doesn't look good and there's probably many failures along the way, but jeez.
- spijdar 6y ago
  I think it's fair to say that these problems shouldn't be blamed on individuals, but systemic failure. In my imagination, someone should have seen this at some point. The system should have had safeguards to ensure that, regardless of any individual's personal failures. That said, I agree it's a little harsh, since there's no evidence that anyone else could/has done better in this incident.
  - 0xy 6y ago
    Why shouldn't it be blamed on the person directly responsible? In this case, the VP of Security and/or the CEO?
    - Pokepokalypse 6y ago
      I'm not sure who would be to blame; but in my case, my former employer is still digging themselves out. 2 years ago, I was shown their Solarwinds console, and I thought it had neat pretty graphs. They asked me to set it up for an additional production database, and I told them how risky this was. As usual; I'm deemed the paranoid one, and they said to do it anyway, and being the good worker I am, and NOT a security specialist, I did it. I left there 2 months later for a different job, and TBH - this was one of the many reasons I didn't want to work there anymore, and I stated as such in my exit interview. Now at the end of the day, the IT director signed-off on our HIPAA certification. I'm not exactly sure which of the 800+ security controls Solar Winds violates off the top of my head. But for him to sign a document that says; "we comply with all of these controls" - when it wasn't true, makes him LEGALLY culpable. And believe it or not - penalties for negligence under HIPAA are fucking hardcore. I doubt they'll be charged though.
- Veserv 6y ago
  You do not need to be the best civil engineer in the world to recognize that a new bridge design falling on its first day is a totally inadequate bridge. Similarly, you do not need to be a security expert to recognize that 15 months of active undetected infiltration demonstrates a nearly comical inadequacy. To provide real material mitigation you would likely need to be in the hour to day range at most to actually stop a double-digit percentage of value-weighted exfiltration. This even ignores the cases where they just want to damage your systems which would generally only take minutes. Now maybe nobody can do that, maybe these systems are the best available, but even if that is true that does not suddenly make them adequate. Adequacy is an objective evaluation of reaching a standard, if nobody can reach the standard, then nobody is adequate. We would not let somebody use a new untested material in a bridge if it can not support the bridge just because "nobody knows how to make a bridge that works with the new material". And, by any reasonable metric, an inability to recognize active infiltration for months indicates that against a threat actor similar to what attacked them, they are about as adequate as a piece of paper against a gun.
  - Jestar342 6y ago
    Terrible and dishonest analogy. The very reason this went undetected for 15 months is because the bridge _didn't_ fall down. There were no signs of a "break in" and it's wholly improper to compare a virtual system to a physical entity like that in the first place.
    - josephg 6y ago
      It was a catastrophic failure by the infosec engineers involved. (And by extension, their / our common processes.) Sure - the bridge metaphor doesn't fully land because a collapsed bridge is visible immediately. But the security failure is worse because of how long it took to become visible. I think the people defending the engineers involved have a mistaken idea of what the responsibility of the security team is. Their job description is not "follow industry best practices" or "look for signs of a break in using their tools". Their job is to keep their company and customer's data secure. At this job they failed. I probably would have failed too, so I have some sympathy for everyone involved. There's an open question of how we engineer our systems to make sure this never happens again. But none of that changes the fact on the ground that the security teams involved failed their responsibility to their businesses.
- hobofan 6y ago
  I think there ought to be a bit of a difference between e.g. an engineer that customizes your local car dealerships WP instance and one that works on a semi-security-related product that can become a prime hacking target and is deployed across nearly all Fortune 500 companies.
SheinhardtWigCo 6y ago
The lesson isn’t that any particular victim sucks at security, it’s that well-resourced targeted attacks are generally unstoppable.
- jakeva 6y ago
  May I propose a corollary: we all suck at security when confronted by a well sourced adversary.
- varjag 6y ago
  Very much so. The description of the staged attack suggests months of work by a sizeable development team.
- joe_the_user 6y ago
  I think you can put things as modern baroque software stacks with their effectively vast "attack surfaces" are not going to stand-up to a well-financed, patient, skilled attacker. I recall that many companies have switched from a perimeter model of defense where systems are secured from the outside to a "defense in depth" model where each system is secured on it's own (plus the perimeter). Perhaps folks should think about tightening the in-depth model and avoiding the consumer model of constant updates from a zillion providers. Or perhaps a single lab could verify updates of the zillion providers rather than leaving them on their own.
  - xvector 6y ago
    > I recall that many companies have switched from a perimeter model of defense where systems are secured from the outside to a "defense in depth" model where each system is secured on it's own (plus the perimeter). Yes, this is the current bleeding edge of cloud security, known as “zero trust.” Amongst other things, it usually involves provisioning mTLS identities in a secured manner to each service, with connections restricted to a whitelisted set of identities. I found Evan Gilman and Doug Barth’s “Zero Trust Networks: Building Secure Systems in Untrusted Networks” [1] a pretty helpful read in understanding what modern/next-gen cloud security looks like. Some modern implementations of varying depth and scale include SPIRE [2], Tailscale [3], and BeyondCorp [4]. -—- [1]: https://www.amazon.com/Zero-Trust-Networks-Building-Untrusted/dp/1491962194 https://www.amazon.com/Zero-Trust-Networks-Building-Untruste... [2]: https://spiffe.io/ https://spiffe.io/ [3]: https://tailscale.com/ https://tailscale.com/ [4]: https://beyondcorp.com/ https://beyondcorp.com/
0xy 6y ago
Their VP of Security posted a blog post entitled "Does your vendor take security seriously?" one month before they announced the breach.
dylan604 6y ago
> that would constitute ~200 TB exfiltrated per customer or ~19 years of 1080p video. Nitpick. That's a very strange metric you're trying to use. What codec/bitrate are you using to determine that amount of spaced used? 19 years = 599184000 seconds 200TB = 200 * 1000GB * 1000MB * 8bits = 1600000000 bits roughly 375kbps for the bit rate That's some crap video quality even for h.265. I for one would not keep a hard drive full of video that poorly encoded. I'd have been more impressed with how many times the Library of Congress could be stored in that 200TB
- Veserv 6y ago
  Here is my math. 200TB = 200,000,000,000,000 B * 8 bits / B = 1,600,000,000,000,000 bits. 1,600,000,000,000,000 / 599,184,000 = 2,670,298 bits/s = 2.67 Mbps which I think is in the normal range, though maybe a little low. I think I originally computed from 20 MB/min which I just found by searching on the internet randomly. Unfortunately, there does not appear to be a consistent number on the amount of data held by the Library of Congress, so how about Wikipedia instead? Wikipedia says Wikipedia is 18.9 GB compressed [1], so 200 TB / 18.9 GB = 10582 Wikipedias. Wikipedia says a Wikipedia would be ~2821 Encyclopedia Britannica volumes which are ~1000 pages each, so this would in total constitute ~30,000,000 volumes of Encyclopedia Britannica exfiltrated before discovery. Hardly a sizeable amount of data. [1] https://en.wikipedia.org/wiki/Wikipedia:Size_of_Wikipedia https://en.wikipedia.org/wiki/Wikipedia:Size_of_Wikipedia
- itsdrewmiller 6y ago
  You forgot KB and mixed up your numerator and denominator.
ransom1538 6y ago
"This is what "top-class" security buys you" Having lived in SF for 15 years, don't leave things in your car. Just don't. If you do, people will break into it. If you don't people might break into it - but! much less of a target. Same goes for systems. Don't keep sensitive data. Don't keep your users private data. Delete your logs, delete your VMs, destroy everything you can.
Jedd 6y ago
> ... since 9/4/19 before being detected on 12/12/20. Somewhat tangentially, I can highly recommend ISO 8601 to make life easier for all us non-Americans. https://en.wikipedia.org/wiki/ISO_8601 https://en.wikipedia.org/wiki/ISO_8601
- geitir 6y ago
  I used to dislike the American format then I realized it's ordered in max(m) < max(d) < max(y) which made me like it more
  - Jedd 6y ago
    , it or but how the hell does that make more usable likeable? Interesting (For your convenience I've sorted that question by len(word))
- artursapek 6y ago
  The format OP used is how we naturally pronounce dates - “September 4th, 2019”. Nobody says “2019 September 4th”.
  - Jedd 6y ago
    Which we do you mean? In Australia and the UK (at least) the standard spoken form would be '4th of September, 2019'. This aligns with traditional short-date formats in those countries (dd/mm/yy[yy]). But that just emphasises why adoption of an international standard is needed.
forgingahead 6y ago
What are the insurance protections for a security company? As in, there's professional liability available, but what are the consequences for SolarWinds/FireEye (if any) for this kind of breach? Can their clients take any meaningful action?

by Veserv 6y ago

vasco 6y ago

Whenever I read a post as strongly worded as yours I wonder that its author must really be the best security minded engineer out there, and has never compromised on anything for a deadline or missed something because they simply didn't know it should be configured in a particular safe way. This definitely doesn't look good and there's probably many failures along the way, but jeez.

spijdar 6y ago
I think it's fair to say that these problems shouldn't be blamed on individuals, but systemic failure. In my imagination, someone should have seen this at some point. The system should have had safeguards to ensure that, regardless of any individual's personal failures. That said, I agree it's a little harsh, since there's no evidence that anyone else could/has done better in this incident.
- 0xy 6y ago
  Why shouldn't it be blamed on the person directly responsible? In this case, the VP of Security and/or the CEO?
  - Pokepokalypse 6y ago
    I'm not sure who would be to blame; but in my case, my former employer is still digging themselves out. 2 years ago, I was shown their Solarwinds console, and I thought it had neat pretty graphs. They asked me to set it up for an additional production database, and I told them how risky this was. As usual; I'm deemed the paranoid one, and they said to do it anyway, and being the good worker I am, and NOT a security specialist, I did it. I left there 2 months later for a different job, and TBH - this was one of the many reasons I didn't want to work there anymore, and I stated as such in my exit interview. Now at the end of the day, the IT director signed-off on our HIPAA certification. I'm not exactly sure which of the 800+ security controls Solar Winds violates off the top of my head. But for him to sign a document that says; "we comply with all of these controls" - when it wasn't true, makes him LEGALLY culpable. And believe it or not - penalties for negligence under HIPAA are fucking hardcore. I doubt they'll be charged though.
Veserv 6y ago
You do not need to be the best civil engineer in the world to recognize that a new bridge design falling on its first day is a totally inadequate bridge. Similarly, you do not need to be a security expert to recognize that 15 months of active undetected infiltration demonstrates a nearly comical inadequacy. To provide real material mitigation you would likely need to be in the hour to day range at most to actually stop a double-digit percentage of value-weighted exfiltration. This even ignores the cases where they just want to damage your systems which would generally only take minutes. Now maybe nobody can do that, maybe these systems are the best available, but even if that is true that does not suddenly make them adequate. Adequacy is an objective evaluation of reaching a standard, if nobody can reach the standard, then nobody is adequate. We would not let somebody use a new untested material in a bridge if it can not support the bridge just because "nobody knows how to make a bridge that works with the new material". And, by any reasonable metric, an inability to recognize active infiltration for months indicates that against a threat actor similar to what attacked them, they are about as adequate as a piece of paper against a gun.
- Jestar342 6y ago
  Terrible and dishonest analogy. The very reason this went undetected for 15 months is because the bridge _didn't_ fall down. There were no signs of a "break in" and it's wholly improper to compare a virtual system to a physical entity like that in the first place.
  - josephg 6y ago
    It was a catastrophic failure by the infosec engineers involved. (And by extension, their / our common processes.) Sure - the bridge metaphor doesn't fully land because a collapsed bridge is visible immediately. But the security failure is worse because of how long it took to become visible. I think the people defending the engineers involved have a mistaken idea of what the responsibility of the security team is. Their job description is not "follow industry best practices" or "look for signs of a break in using their tools". Their job is to keep their company and customer's data secure. At this job they failed. I probably would have failed too, so I have some sympathy for everyone involved. There's an open question of how we engineer our systems to make sure this never happens again. But none of that changes the fact on the ground that the security teams involved failed their responsibility to their businesses.
hobofan 6y ago
I think there ought to be a bit of a difference between e.g. an engineer that customizes your local car dealerships WP instance and one that works on a semi-security-related product that can become a prime hacking target and is deployed across nearly all Fortune 500 companies.

SheinhardtWigCo 6y ago

The lesson isn’t that any particular victim sucks at security, it’s that well-resourced targeted attacks are generally unstoppable.

jakeva 6y ago
May I propose a corollary: we all suck at security when confronted by a well sourced adversary.
varjag 6y ago
Very much so. The description of the staged attack suggests months of work by a sizeable development team.
joe_the_user 6y ago
I think you can put things as modern baroque software stacks with their effectively vast "attack surfaces" are not going to stand-up to a well-financed, patient, skilled attacker. I recall that many companies have switched from a perimeter model of defense where systems are secured from the outside to a "defense in depth" model where each system is secured on it's own (plus the perimeter). Perhaps folks should think about tightening the in-depth model and avoiding the consumer model of constant updates from a zillion providers. Or perhaps a single lab could verify updates of the zillion providers rather than leaving them on their own.
- xvector 6y ago
  > I recall that many companies have switched from a perimeter model of defense where systems are secured from the outside to a "defense in depth" model where each system is secured on it's own (plus the perimeter). Yes, this is the current bleeding edge of cloud security, known as “zero trust.” Amongst other things, it usually involves provisioning mTLS identities in a secured manner to each service, with connections restricted to a whitelisted set of identities. I found Evan Gilman and Doug Barth’s “Zero Trust Networks: Building Secure Systems in Untrusted Networks” [1] a pretty helpful read in understanding what modern/next-gen cloud security looks like. Some modern implementations of varying depth and scale include SPIRE [2], Tailscale [3], and BeyondCorp [4]. -—- [1]: https://www.amazon.com/Zero-Trust-Networks-Building-Untrusted/dp/1491962194 https://www.amazon.com/Zero-Trust-Networks-Building-Untruste... [2]: https://spiffe.io/ https://spiffe.io/ [3]: https://tailscale.com/ https://tailscale.com/ [4]: https://beyondcorp.com/ https://beyondcorp.com/

0xy 6y ago

Their VP of Security posted a blog post entitled "Does your vendor take security seriously?" one month before they announced the breach.

dylan604 6y ago

> that would constitute ~200 TB exfiltrated per customer or ~19 years of 1080p video. Nitpick. That's a very strange metric you're trying to use. What codec/bitrate are you using to determine that amount of spaced used? 19 years = 599184000 seconds 200TB = 200 * 1000GB * 1000MB * 8bits = 1600000000 bits roughly 375kbps for the bit rate That's some crap video quality even for h.265. I for one would not keep a hard drive full of video that poorly encoded. I'd have been more impressed with how many times the Library of Congress could be stored in that 200TB

Veserv 6y ago
Here is my math. 200TB = 200,000,000,000,000 B * 8 bits / B = 1,600,000,000,000,000 bits. 1,600,000,000,000,000 / 599,184,000 = 2,670,298 bits/s = 2.67 Mbps which I think is in the normal range, though maybe a little low. I think I originally computed from 20 MB/min which I just found by searching on the internet randomly. Unfortunately, there does not appear to be a consistent number on the amount of data held by the Library of Congress, so how about Wikipedia instead? Wikipedia says Wikipedia is 18.9 GB compressed [1], so 200 TB / 18.9 GB = 10582 Wikipedias. Wikipedia says a Wikipedia would be ~2821 Encyclopedia Britannica volumes which are ~1000 pages each, so this would in total constitute ~30,000,000 volumes of Encyclopedia Britannica exfiltrated before discovery. Hardly a sizeable amount of data. [1] https://en.wikipedia.org/wiki/Wikipedia:Size_of_Wikipedia https://en.wikipedia.org/wiki/Wikipedia:Size_of_Wikipedia
itsdrewmiller 6y ago
You forgot KB and mixed up your numerator and denominator.

ransom1538 6y ago

"This is what "top-class" security buys you" Having lived in SF for 15 years, don't leave things in your car. Just don't. If you do, people will break into it. If you don't people might break into it - but! much less of a target. Same goes for systems. Don't keep sensitive data. Don't keep your users private data. Delete your logs, delete your VMs, destroy everything you can.

Jedd 6y ago

> ... since 9/4/19 before being detected on 12/12/20. Somewhat tangentially, I can highly recommend ISO 8601 to make life easier for all us non-Americans. https://en.wikipedia.org/wiki/ISO_8601 https://en.wikipedia.org/wiki/ISO_8601

geitir 6y ago
I used to dislike the American format then I realized it's ordered in max(m) < max(d) < max(y) which made me like it more
- Jedd 6y ago
  , it or but how the hell does that make more usable likeable? Interesting (For your convenience I've sorted that question by len(word))
artursapek 6y ago
The format OP used is how we naturally pronounce dates - “September 4th, 2019”. Nobody says “2019 September 4th”.
- Jedd 6y ago
  Which we do you mean? In Australia and the UK (at least) the standard spoken form would be '4th of September, 2019'. This aligns with traditional short-date formats in those countries (dd/mm/yy[yy]). But that just emphasises why adoption of an international standard is needed.

forgingahead 6y ago

What are the insurance protections for a security company? As in, there's professional liability available, but what are the consequences for SolarWinds/FireEye (if any) for this kind of breach? Can their clients take any meaningful action?