4 ms·

You do not need to be the best civil engineer in the world to recognize that a new bridge design falling on its first day is a totally inadequate bridge. Simila

by Veserv 6y ago

You do not need to be the best civil engineer in the world to recognize that a new bridge design falling on its first day is a totally inadequate bridge. Similarly, you do not need to be a security expert to recognize that 15 months of active undetected infiltration demonstrates a nearly comical inadequacy. To provide real material mitigation you would likely need to be in the hour to day range at most to actually stop a double-digit percentage of value-weighted exfiltration. This even ignores the cases where they just want to damage your systems which would generally only take minutes. Now maybe nobody can do that, maybe these systems are the best available, but even if that is true that does not suddenly make them adequate. Adequacy is an objective evaluation of reaching a standard, if nobody can reach the standard, then nobody is adequate. We would not let somebody use a new untested material in a bridge if it can not support the bridge just because "nobody knows how to make a bridge that works with the new material". And, by any reasonable metric, an inability to recognize active infiltration for months indicates that against a threat actor similar to what attacked them, they are about as adequate as a piece of paper against a gun.

Jestar342 6y ago
Terrible and dishonest analogy. The very reason this went undetected for 15 months is because the bridge _didn't_ fall down. There were no signs of a "break in" and it's wholly improper to compare a virtual system to a physical entity like that in the first place.
- josephg 6y ago
  It was a catastrophic failure by the infosec engineers involved. (And by extension, their / our common processes.) Sure - the bridge metaphor doesn't fully land because a collapsed bridge is visible immediately. But the security failure is worse because of how long it took to become visible. I think the people defending the engineers involved have a mistaken idea of what the responsibility of the security team is. Their job description is not "follow industry best practices" or "look for signs of a break in using their tools". Their job is to keep their company and customer's data secure. At this job they failed. I probably would have failed too, so I have some sympathy for everyone involved. There's an open question of how we engineer our systems to make sure this never happens again. But none of that changes the fact on the ground that the security teams involved failed their responsibility to their businesses.
  - marcus0x62 6y ago
    > Their job description is not "follow industry best practices" or "look for signs of a break in using their tools". Their job is to keep their company and customer's data secure. At this job they failed. No. Their job is to use the resources they’ve been allocated to manage risk within the organization to the risk level senior management has agreed to accept. Maybe that shouldn’t be their job, but it is. There isn’t a security team on the planet that gets to dictate security to the rest of the organization or unilaterally make decisions that influence the running of the business to achieve the level of risk mitigation that they themselves would prefer to attain. That is putting aside the fact that defending against a determined nation-state adversary is a nigh-on impossible task that would require countermeasures like ‘don’t connect to the Internet at all’, ‘don’t hire anyone you haven’t personally known since childhood’, ‘hand-deliver your product to your customers’, and other equally impractical mitigations. Nobody outside of Solarwinds knows if their security team succeeded or failed in the mission they were given.
    - josephg 6y ago
      I hear what you're saying and I appreciate that perspective. I agree, and there's also something slippery in that argument taken to its extreme that we need to be careful of. Some people here on HN made the same argument when Equifax leaked personal information about millions of americans. They said it was ultimately management's fault and not the engineers' fault for not allocating enough resources to security. And the same argument was used by the engineers who made the Therac-25 radiology machine. In that case, software bugs resulted in a handful of deaths due to lethal radiation. Upper management can't be responsible for everything that happens in their business. Engineering isn't their job or their expertise. Thats why they hire software engineers and security engineers - to be the local experts. We need to bear responsibility for the decisions we make in our field. And engineers have a duty not just to the companies we work for, but also to society at large. If we leave our personal judgement at the door in the morning, we fail in our duty to society. To go back to the bridge metaphor, if a bridge falls down, its not good enough for the civil engineers involved to blame management for not giving them enough time / budget / whatever. They also bear some responsibility for the disaster. This has been enshrined in case law too, at the Nuremberg trials. "I was just doing my job" wasn't considered a good enough excuse for the guards in WW2 concentration camps. These are big examples, but I think the principle is fractally true. And the inverse also holds. Praise and blame go together. The biomedical engineers in the labs also deserve praise for the covid19 vaccines they've invented, even if upper management told them to do it. We aren't management's slaves.
viraptor 6y ago
Apart from what the sibling comment mentioned, it's not a great analogy because we have a very limited things that can go wrong with a bridge. This is knowledge shared between all engineers - you analyse the design for multiple known forces and you're done. Compare this to the CI systems designed to take unknown code and run it in a safeish way. In a bridge analogy it would be something like "one of the screws turned out to be a remote controlled drilling device which hollowed out parts of steel without visible changes" - of course nobody would notice that for some time.
- joe_the_user 6y ago
  ...it's not a great analogy because we have a very limited things that can go wrong with a bridge. Security engineering is still evolving. But it seems reasonable to say that if you engineer a thing that may be subjected to unlimited, unknown stress, you engineered it wrong. These many large companies giving themselves a single point of failure by installing SolarWinds binary, not knowing the contents of home-phoning and so-forth is a terrible security engineering solution. Sure, maybe they "couldn't have done better under the circumstance" but someone allowed the circumstances to happen too.
  - viraptor 6y ago
    > But it seems reasonable to say that if you engineer a thing that may be subjected to unlimited, unknown stress, you engineered it wrong. A computer is that by design. Anyone using computers is using a system subjected to unknown and unlimited inputs, especially when connected to the internet. (But it can be exposed via employees if it isn't) Any controls we have at the moment are "this will not happen (under the assumptions we're currently aware of)". If those companies didn't run this monitoring solution, they would use a different one. Unless the system is perfectly covered with access policies (none are), the monitoring solution is a global access to enterprises. You can mitigate the impact, shard things, etc. but there are still many cross-cutting concerns which will (suddenly?) turn out to be a single point of failure.
  - Pokepokalypse 6y ago
    The problem is that Security Engineering is still treated as if it's a separate thing that is evolving. It's systems engineering, where security is a necessary feature that's being grafted-on after-the-fact.
lifeisstillgood 6y ago
I think a better analogy is the human analog of what this is - spying. Not noticing a spy in your company, university, army headquarters for 15 months is hardly an indictment of one's counter-espionage - especially if as it seems they did very very little till Feb, and did it very carefully.
- adolph 6y ago
  seems is working hard here.
jodrellblank 6y ago
> "We would not let somebody use a new untested material in a bridge if it can not support the bridge just because "nobody knows how to make a bridge that works with the new material"" You might be interested to read about the Morandi Bridge collapse in Italy: https://www.engineering.com/story/italys-morandi-bridge-collapsewhat-do-we-know https://www.engineering.com/story/italys-morandi-bridge-coll... "When the Morandi Bridge was built, encasing a cable in concrete was innovative.", "the decision to pre-stress it was debatable", the concrete meant they couldn't check for cable rusting underneath, the concrete could have been supplied by the Mafia and under specification, "“We have used materials that are destined to deteriorate quickly, like those of the bridge in Genoa,” Bercich said in a post-collapse analysis". "Settimo Martinello, director of bridge inspection company 4EMME Service, told CNN that there are “about 15 to 20 bridges collapsing every year” in Italy." From another source: https://www.theguardian.com/cities/2019/feb/26/what-caused-the-genoa-morandi-bridge-collapse-and-the-end-of-an-italian-national-myth https://www.theguardian.com/cities/2019/feb/26/what-caused-t... "In the 1960s little was known about the interaction of materials, or the effects of pollution and climate on corrosion.", "Morandi himself was surprised to see the structure age faster than he had anticipated. In 1979 he issued a report detailing a number of interventions to protect the structure against pollution from nearby factories" It's not a collapse on its first day, but it undermines your suggestion that bridges are a solved problem, designed once with traditional methods, are well understood inside and out, and therefore never collapse. Or if that's too old, how about the celebratory Millennium Bridge in London, a footbridge opened in June 2000 and closed two days later because it couldn't withstand the footsteps? https://en.wikipedia.org/wiki/Millennium_Bridge,_London#Opening https://en.wikipedia.org/wiki/Millennium_Bridge,_London#Open...
- tialaramex 6y ago
  The Millennium Bridge was altered because the swaying was "alarming" but there's no particular reason so far as I know to think it would have caused structural failure, it's just not useful to have a pedestrian bridge that some fraction of pedestrians won't cross because they're scared, it isn't a theme park attraction.

by Veserv 6y ago

Jestar342 6y ago

Terrible and dishonest analogy. The very reason this went undetected for 15 months is because the bridge _didn't_ fall down. There were no signs of a "break in" and it's wholly improper to compare a virtual system to a physical entity like that in the first place.

josephg 6y ago
It was a catastrophic failure by the infosec engineers involved. (And by extension, their / our common processes.) Sure - the bridge metaphor doesn't fully land because a collapsed bridge is visible immediately. But the security failure is worse because of how long it took to become visible. I think the people defending the engineers involved have a mistaken idea of what the responsibility of the security team is. Their job description is not "follow industry best practices" or "look for signs of a break in using their tools". Their job is to keep their company and customer's data secure. At this job they failed. I probably would have failed too, so I have some sympathy for everyone involved. There's an open question of how we engineer our systems to make sure this never happens again. But none of that changes the fact on the ground that the security teams involved failed their responsibility to their businesses.
- marcus0x62 6y ago
  > Their job description is not "follow industry best practices" or "look for signs of a break in using their tools". Their job is to keep their company and customer's data secure. At this job they failed. No. Their job is to use the resources they’ve been allocated to manage risk within the organization to the risk level senior management has agreed to accept. Maybe that shouldn’t be their job, but it is. There isn’t a security team on the planet that gets to dictate security to the rest of the organization or unilaterally make decisions that influence the running of the business to achieve the level of risk mitigation that they themselves would prefer to attain. That is putting aside the fact that defending against a determined nation-state adversary is a nigh-on impossible task that would require countermeasures like ‘don’t connect to the Internet at all’, ‘don’t hire anyone you haven’t personally known since childhood’, ‘hand-deliver your product to your customers’, and other equally impractical mitigations. Nobody outside of Solarwinds knows if their security team succeeded or failed in the mission they were given.
  - josephg 6y ago
    I hear what you're saying and I appreciate that perspective. I agree, and there's also something slippery in that argument taken to its extreme that we need to be careful of. Some people here on HN made the same argument when Equifax leaked personal information about millions of americans. They said it was ultimately management's fault and not the engineers' fault for not allocating enough resources to security. And the same argument was used by the engineers who made the Therac-25 radiology machine. In that case, software bugs resulted in a handful of deaths due to lethal radiation. Upper management can't be responsible for everything that happens in their business. Engineering isn't their job or their expertise. Thats why they hire software engineers and security engineers - to be the local experts. We need to bear responsibility for the decisions we make in our field. And engineers have a duty not just to the companies we work for, but also to society at large. If we leave our personal judgement at the door in the morning, we fail in our duty to society. To go back to the bridge metaphor, if a bridge falls down, its not good enough for the civil engineers involved to blame management for not giving them enough time / budget / whatever. They also bear some responsibility for the disaster. This has been enshrined in case law too, at the Nuremberg trials. "I was just doing my job" wasn't considered a good enough excuse for the guards in WW2 concentration camps. These are big examples, but I think the principle is fractally true. And the inverse also holds. Praise and blame go together. The biomedical engineers in the labs also deserve praise for the covid19 vaccines they've invented, even if upper management told them to do it. We aren't management's slaves.

viraptor 6y ago

Apart from what the sibling comment mentioned, it's not a great analogy because we have a very limited things that can go wrong with a bridge. This is knowledge shared between all engineers - you analyse the design for multiple known forces and you're done. Compare this to the CI systems designed to take unknown code and run it in a safeish way. In a bridge analogy it would be something like "one of the screws turned out to be a remote controlled drilling device which hollowed out parts of steel without visible changes" - of course nobody would notice that for some time.

joe_the_user 6y ago
...it's not a great analogy because we have a very limited things that can go wrong with a bridge. Security engineering is still evolving. But it seems reasonable to say that if you engineer a thing that may be subjected to unlimited, unknown stress, you engineered it wrong. These many large companies giving themselves a single point of failure by installing SolarWinds binary, not knowing the contents of home-phoning and so-forth is a terrible security engineering solution. Sure, maybe they "couldn't have done better under the circumstance" but someone allowed the circumstances to happen too.
- viraptor 6y ago
  > But it seems reasonable to say that if you engineer a thing that may be subjected to unlimited, unknown stress, you engineered it wrong. A computer is that by design. Anyone using computers is using a system subjected to unknown and unlimited inputs, especially when connected to the internet. (But it can be exposed via employees if it isn't) Any controls we have at the moment are "this will not happen (under the assumptions we're currently aware of)". If those companies didn't run this monitoring solution, they would use a different one. Unless the system is perfectly covered with access policies (none are), the monitoring solution is a global access to enterprises. You can mitigate the impact, shard things, etc. but there are still many cross-cutting concerns which will (suddenly?) turn out to be a single point of failure.
- Pokepokalypse 6y ago
  The problem is that Security Engineering is still treated as if it's a separate thing that is evolving. It's systems engineering, where security is a necessary feature that's being grafted-on after-the-fact.

lifeisstillgood 6y ago

I think a better analogy is the human analog of what this is - spying. Not noticing a spy in your company, university, army headquarters for 15 months is hardly an indictment of one's counter-espionage - especially if as it seems they did very very little till Feb, and did it very carefully.

adolph 6y ago
seems is working hard here.

jodrellblank 6y ago

> "We would not let somebody use a new untested material in a bridge if it can not support the bridge just because "nobody knows how to make a bridge that works with the new material"" You might be interested to read about the Morandi Bridge collapse in Italy: https://www.engineering.com/story/italys-morandi-bridge-collapsewhat-do-we-know https://www.engineering.com/story/italys-morandi-bridge-coll... "When the Morandi Bridge was built, encasing a cable in concrete was innovative.", "the decision to pre-stress it was debatable", the concrete meant they couldn't check for cable rusting underneath, the concrete could have been supplied by the Mafia and under specification, "“We have used materials that are destined to deteriorate quickly, like those of the bridge in Genoa,” Bercich said in a post-collapse analysis". "Settimo Martinello, director of bridge inspection company 4EMME Service, told CNN that there are “about 15 to 20 bridges collapsing every year” in Italy." From another source: https://www.theguardian.com/cities/2019/feb/26/what-caused-the-genoa-morandi-bridge-collapse-and-the-end-of-an-italian-national-myth https://www.theguardian.com/cities/2019/feb/26/what-caused-t... "In the 1960s little was known about the interaction of materials, or the effects of pollution and climate on corrosion.", "Morandi himself was surprised to see the structure age faster than he had anticipated. In 1979 he issued a report detailing a number of interventions to protect the structure against pollution from nearby factories" It's not a collapse on its first day, but it undermines your suggestion that bridges are a solved problem, designed once with traditional methods, are well understood inside and out, and therefore never collapse. Or if that's too old, how about the celebratory Millennium Bridge in London, a footbridge opened in June 2000 and closed two days later because it couldn't withstand the footsteps? https://en.wikipedia.org/wiki/Millennium_Bridge,_London#Opening https://en.wikipedia.org/wiki/Millennium_Bridge,_London#Open...

tialaramex 6y ago
The Millennium Bridge was altered because the swaying was "alarming" but there's no particular reason so far as I know to think it would have caused structural failure, it's just not useful to have a pedestrian bridge that some fraction of pedestrians won't cross because they're scared, it isn't a theme park attraction.