6 ms·

Whenever I read a post as strongly worded as yours I wonder that its author must really be the best security minded engineer out there, and has never compromise

by vasco 6y ago

Whenever I read a post as strongly worded as yours I wonder that its author must really be the best security minded engineer out there, and has never compromised on anything for a deadline or missed something because they simply didn't know it should be configured in a particular safe way. This definitely doesn't look good and there's probably many failures along the way, but jeez.

spijdar 6y ago
I think it's fair to say that these problems shouldn't be blamed on individuals, but systemic failure. In my imagination, someone should have seen this at some point. The system should have had safeguards to ensure that, regardless of any individual's personal failures. That said, I agree it's a little harsh, since there's no evidence that anyone else could/has done better in this incident.
- 0xy 6y ago
  Why shouldn't it be blamed on the person directly responsible? In this case, the VP of Security and/or the CEO?
  - Pokepokalypse 6y ago
    I'm not sure who would be to blame; but in my case, my former employer is still digging themselves out. 2 years ago, I was shown their Solarwinds console, and I thought it had neat pretty graphs. They asked me to set it up for an additional production database, and I told them how risky this was. As usual; I'm deemed the paranoid one, and they said to do it anyway, and being the good worker I am, and NOT a security specialist, I did it. I left there 2 months later for a different job, and TBH - this was one of the many reasons I didn't want to work there anymore, and I stated as such in my exit interview. Now at the end of the day, the IT director signed-off on our HIPAA certification. I'm not exactly sure which of the 800+ security controls Solar Winds violates off the top of my head. But for him to sign a document that says; "we comply with all of these controls" - when it wasn't true, makes him LEGALLY culpable. And believe it or not - penalties for negligence under HIPAA are fucking hardcore. I doubt they'll be charged though.
Veserv 6y ago
You do not need to be the best civil engineer in the world to recognize that a new bridge design falling on its first day is a totally inadequate bridge. Similarly, you do not need to be a security expert to recognize that 15 months of active undetected infiltration demonstrates a nearly comical inadequacy. To provide real material mitigation you would likely need to be in the hour to day range at most to actually stop a double-digit percentage of value-weighted exfiltration. This even ignores the cases where they just want to damage your systems which would generally only take minutes. Now maybe nobody can do that, maybe these systems are the best available, but even if that is true that does not suddenly make them adequate. Adequacy is an objective evaluation of reaching a standard, if nobody can reach the standard, then nobody is adequate. We would not let somebody use a new untested material in a bridge if it can not support the bridge just because "nobody knows how to make a bridge that works with the new material". And, by any reasonable metric, an inability to recognize active infiltration for months indicates that against a threat actor similar to what attacked them, they are about as adequate as a piece of paper against a gun.
- Jestar342 6y ago
  Terrible and dishonest analogy. The very reason this went undetected for 15 months is because the bridge _didn't_ fall down. There were no signs of a "break in" and it's wholly improper to compare a virtual system to a physical entity like that in the first place.
  - josephg 6y ago
    It was a catastrophic failure by the infosec engineers involved. (And by extension, their / our common processes.) Sure - the bridge metaphor doesn't fully land because a collapsed bridge is visible immediately. But the security failure is worse because of how long it took to become visible. I think the people defending the engineers involved have a mistaken idea of what the responsibility of the security team is. Their job description is not "follow industry best practices" or "look for signs of a break in using their tools". Their job is to keep their company and customer's data secure. At this job they failed. I probably would have failed too, so I have some sympathy for everyone involved. There's an open question of how we engineer our systems to make sure this never happens again. But none of that changes the fact on the ground that the security teams involved failed their responsibility to their businesses.
hobofan 6y ago
I think there ought to be a bit of a difference between e.g. an engineer that customizes your local car dealerships WP instance and one that works on a semi-security-related product that can become a prime hacking target and is deployed across nearly all Fortune 500 companies.

by vasco 6y ago

spijdar 6y ago

I think it's fair to say that these problems shouldn't be blamed on individuals, but systemic failure. In my imagination, someone should have seen this at some point. The system should have had safeguards to ensure that, regardless of any individual's personal failures. That said, I agree it's a little harsh, since there's no evidence that anyone else could/has done better in this incident.

0xy 6y ago
Why shouldn't it be blamed on the person directly responsible? In this case, the VP of Security and/or the CEO?
- Pokepokalypse 6y ago
  I'm not sure who would be to blame; but in my case, my former employer is still digging themselves out. 2 years ago, I was shown their Solarwinds console, and I thought it had neat pretty graphs. They asked me to set it up for an additional production database, and I told them how risky this was. As usual; I'm deemed the paranoid one, and they said to do it anyway, and being the good worker I am, and NOT a security specialist, I did it. I left there 2 months later for a different job, and TBH - this was one of the many reasons I didn't want to work there anymore, and I stated as such in my exit interview. Now at the end of the day, the IT director signed-off on our HIPAA certification. I'm not exactly sure which of the 800+ security controls Solar Winds violates off the top of my head. But for him to sign a document that says; "we comply with all of these controls" - when it wasn't true, makes him LEGALLY culpable. And believe it or not - penalties for negligence under HIPAA are fucking hardcore. I doubt they'll be charged though.

Veserv 6y ago

You do not need to be the best civil engineer in the world to recognize that a new bridge design falling on its first day is a totally inadequate bridge. Similarly, you do not need to be a security expert to recognize that 15 months of active undetected infiltration demonstrates a nearly comical inadequacy. To provide real material mitigation you would likely need to be in the hour to day range at most to actually stop a double-digit percentage of value-weighted exfiltration. This even ignores the cases where they just want to damage your systems which would generally only take minutes. Now maybe nobody can do that, maybe these systems are the best available, but even if that is true that does not suddenly make them adequate. Adequacy is an objective evaluation of reaching a standard, if nobody can reach the standard, then nobody is adequate. We would not let somebody use a new untested material in a bridge if it can not support the bridge just because "nobody knows how to make a bridge that works with the new material". And, by any reasonable metric, an inability to recognize active infiltration for months indicates that against a threat actor similar to what attacked them, they are about as adequate as a piece of paper against a gun.

Jestar342 6y ago
Terrible and dishonest analogy. The very reason this went undetected for 15 months is because the bridge _didn't_ fall down. There were no signs of a "break in" and it's wholly improper to compare a virtual system to a physical entity like that in the first place.
- josephg 6y ago
  It was a catastrophic failure by the infosec engineers involved. (And by extension, their / our common processes.) Sure - the bridge metaphor doesn't fully land because a collapsed bridge is visible immediately. But the security failure is worse because of how long it took to become visible. I think the people defending the engineers involved have a mistaken idea of what the responsibility of the security team is. Their job description is not "follow industry best practices" or "look for signs of a break in using their tools". Their job is to keep their company and customer's data secure. At this job they failed. I probably would have failed too, so I have some sympathy for everyone involved. There's an open question of how we engineer our systems to make sure this never happens again. But none of that changes the fact on the ground that the security teams involved failed their responsibility to their businesses.

hobofan 6y ago

I think there ought to be a bit of a difference between e.g. an engineer that customizes your local car dealerships WP instance and one that works on a semi-security-related product that can become a prime hacking target and is deployed across nearly all Fortune 500 companies.