5 ms·

"What Happened to Digital Resilience?" Was there ever such a time? If so then tell me when it was. "The latest chaos wasn’t caused by an adversary, but it pro

"What Happened to Digital Resilience?" Was there ever such a time? If so then tell me when it was. "The latest chaos wasn’t caused by an adversary, but it provided a road map of American vulnerabilities at a critical moment." I've no doubt that road maps of American vulnerabilities are currently being planned, roadmaped and stockpiled for future use by those who aren't on the best terms with the US. In one way I'm amazed at how laxadasical the US and others are towards these threats and that they have not done more to harden the vulnerabilities. On the other hand, it's obvious: cost is one factor but I reckon another bigger one is 'convenience'. Hardening systems against vulnerabilities means making them less convenient/easy to use and people instantly balk against that. Remember, this happened big-time when Microsoft introduced Windows especially Windows 95. To capture the market Microsoft made everything as easy as possible for nontechnical users—just click on something and it'd happen, things would happen with ease. And all this happened without due consideration to security. When viruses, vulnerabilities, breaches got out of hand restrictions were introduced which meant users had less freedom to do what they'd gotten used to doing. What Microsoft did was to get the world used to slack operating procedures and efforts reign this in has met with user resistance ever since. We're now stuck with a major problem that was easily foreseeable even before Microsoft launched Windows 95. Fixing it will be extremely difficult.

dralley 2y ago
This is an area where studying Ukraine's experience will be very useful (and probably has already been useful) There were years of cyberattacks against pretty much every peice of critical infrastructure they have. Things went down, there were disruptions, but they adapted. Sometimes by falling back to low-tech solutions, sometimes by developing new systems with robustness into new systems and purging the old (much easier to politically justify when the problem is tangible and immediate). I seem to recall that one of the first things we did when tensions started ramping up was sending teams of cyber security experts from the NSA to help them lock down and root out infiltrations.
- emseetech 2y ago
  | Sometimes by falling back to low-tech solutions My first thought in all this was wondering if there's a business opportunity for a consulting firm or startup that designs and manages offline paper backup systems that can quickly and seamlessly integrate back with digital systems once they come back online.
  - SoftTalker 2y ago
    The problem is that if you aren't regularly training employees on those manual fallback systems, when you have to suddenly activate them, nobody will know what to do. Even if they have been trained on what to do, the processes will not be second nature. In real use, they will hit situations that the paper forms or training didn't cover, and will have to make up something on the spot, which they will each do differently. Fully comprehensive, regularly trained manual operations are very expensive to develop and test. Only the most safety-critical organizations will be able to justify and have the resources to effectively implement them. Air-traffic control, hospitals, nuclear plants, etc. And, they already have done it.
  - hilbert42 2y ago
    "…offline paper backup systems that can quickly and seamlessly integrate back with digital systems once they come back online." It's not offline paper backups that are needed but rather the reverse—offline paper-based systems used as masters! The heart of any critical infrastructure—specifically the part of the heart that's the most vulnerable is comparatively small compared to the large masses of ancillary data and thus could be managed on a paper-based database (as they once were before computers). With computers and IT infrastructure as they're currently implemented—not as computer science says they ought to be—a secure filing cabinet/paper-based database is much more secure than an ephemeral one that has no physical or volumetric presence and which takes precious little to shove it from one end of the planet to the other. The caveat is of course, the database must be secured against physical access and located in a secure building, etc. Let me state why. Comparatively speaking, in recent times there are very few secure bank vaults and such that criminals have breached. The number is so small I can't remember when I'd last heard of a bank vault robbery. Another way of looking at it is to ask yourself when was gold last stolen from Fort Knox or the Bank of England, or $100 bills stolen from the US treasury/mint before their distribution. Why so few robberies you may well ask. We've had hundreds of years of experience locking up these valuables and although the current systems used to secure them aren't watertight and likely never will be, they're nevertheless sufficiently secure to the extent that the few breaches that do occur from time to time are manageable. With physical security, we've found a workable balance between security and workability. With the few robberies that occur it's not worth the effort of tightening security further, to do so would not only add considerably to the cost but also physical access would be more difficult thus less convenient to use because of the additional protocols that would have to be put in place to reach the higher security level. Also, think for a moment that if you could gain access to a secured paper-based database how quickly could you copy it, and how would you copy it? Right, both would be very difficult. On the other hand once an electronic database has been breached megabytes if not gigabytes can be sucked out within seconds. In practice, the electronic/digital world has nothing as 'bulletproof' as a physically secure system. Given the statistics—the rate of cyber breaches, personal data stolen, Bitcoin thefts, etc., etc. that occur not on a yearly but rather on a daily basis, one simply can't argue that collectively IT/electronic systems are more secure than physical, paper-based ones. Back to the physical database: a secure paper-based database would always be offline, if some data are needed from it then they have to be extracted manually, then vetted and encrypted before being put on line (that's if it's actually necessary to put highly secure stuff on line at all). As things stand, owners of information have a choice, store it in an electronic system and take advantage of the operational advantages that it offers or use secure paper-based storage and suffer the inconvenience. One can't have it both ways. The reason why we've so may data breaches is that the average punter far prefers electronic data systems for their convenience. On evidence, convenience is seen as more important, in practice its value far outweighs data security and integrity.
pjc50 2y ago
The "cyber agencies" focus on offence, because that's easy to score points with and appear to be doing something, whereas defence is a very boring job of securing a zillion outdated endpoints. Or trying to get profitable megacorps to do something less vulnerable and less profitable.
- TrueDuality 2y ago
  Offense is also easy in that there is a ton of software out there, and you just need to find one vulnerability. There is a "win" condition" Defense is impossible as there is a ton of software and you need to protect all of it every time, there is only a "lose" condition.
- tintor 2y ago
  The "cyber agencies" could focus on pen testing of domestic companies, and issuing fines to insecure ones.
TrueDuality 2y ago
> Was there ever such a time? If so then tell me when it was. It was a goal for a long time, and I'd say we use to be more resilient pre-cloud SaaS auto-update everything. When every software solution installation is on private networks, with fundamentally different architectures (both machine and topology), along with a wide selection of even very poor quality software, was a lot more resilient than what we have today. Today a single outage in a single service (say AWS) can grind a large number of companies to a halt. A bad update like this one immediately impacts everyone all at once and has a domino effect. That didn't use to happen. We've been concentrating our collective architecture into a few best practice tools but that all become single points of failure for not only digital attacks, but misconfigurations, mismanagement, company failures, exhausted underpaid engineers, optimizations, etc. > Hardening systems against vulnerabilities means making them less convenient/easy to use and people instantly balk against that. This isn't necessarily true, and I'd argue quite the opposite direction has been happening in the security industry over the past decade or so. People realized that hard security would only cause users to find simple predictable bypasses that would overall _weaken_ the security posture. You just have to look at the evolution of NIST recommendations around passwords to see this happening. Must change a password every 90 days that can't be the same as your last 10 passwords and complex password requirements? Well users are going to use the minimum size in predictable patterns and just increment a number at the end. Those old password hashes you have to keep around to check if the user is reusing the password? Those are a liability that, when broken, tell the attacker which pattern each user is using. Not the case anymore and there is a lot more usable security rolled that is entirely transparent to end users or almost entirely transparent. Think about how prevalent and bad captchas used to be on the website and how easy they were to circumvent. Cloudflare's and Google's captcha solution are pretty transparent and has much greater efficacy than the old ones. Did Microsoft's general and on-going laxness contribute to bad security practices? Absolutely, but that is one ecosystem that had weird other by the nature of how inherently unstable that environment was and is not and hasn't except for maybe a brief peak ever been a core foundation of the internet infrastructure, just enterprise infrastructure unfortunately. They definitely never got the memo about usable or transparent security. I hope they're at least trying behind the scenes now.
- hilbert42 2y ago
lenerdenator 2y ago
> In one way I'm amazed at how laxadasical the US and others are towards these threats and that they have not done more to harden the vulnerabilities. On the other hand, it's obvious: cost is one factor but I reckon another bigger one is 'convenience'. Hardening systems against vulnerabilities means making them less convenient/easy to use and people instantly balk against that. "Show me the incentives, and I'll show you the outcomes." - Charlie Munger. We do not incentivize companies to operate secure, redundant, reliable computer systems. We incentivize companies to make the number at the bottom of the spreadsheet beat the expectations some analyst in Lower Manhattan set 90 days prior. And since companies handle the majority of societal work in the United States, that's how most critical systems are designed. Now, there's a chance that this will play out in court, and that Crowdstrike will have to be bought out to make up for the damages their customers suffered starting on July 19th. However, that will take years, and the outcome could very well be that the plaintiffs will receive symbolic or even no damages. By then, the market will have hedged, captured regulatory authorities, cut its losses, and just altogether moved on. The assets will be purchased in a firesale by people who see this as "creative destruction" and won't care that peoples' lives were put at risk because of this. And the cycle will continue.
- hilbert42 2y ago
  ...And I can't agree more. The question is what can be done, if anything. But I've a solution in my wildest dreams as a dictator. :-)
  - lenerdenator 2y ago
    You introduce consequences against the people who have created the system that we're in now.
    - dasil003 2y ago
      Even assuming you could narrow this down to a small enough set of people that can credibly be held responsible for creating the system we have now, and assuming you could impose consequences on them without violating their civil rights, and assuming they learnt their lesson and would actively take precautions to avoid their actions leading to such a systemic failure in the future, at best this would only influence those particular actors to avoid the previous failures. The next systemic failure would look quite different on the ground and come from different individuals pursuing different goals who would not have learnt any of the previous lessons. The only people who would see the connection would be more experienced people and/or intellectuals looking from a higher zoom level, but likely would not be empowered to really do anything to stop it given all the direct financial incentives motivating a much larger group of people to direct action. If our culture had more respect for elders and/or thinkers that could be a start, but even then it would still be an uphill battle in a capitalist society.
binary132 2y ago
What makes you think only a foreign adversary might want illegitimate access to our computers?
kortilla 2y ago
>Was there ever such a time? If so then tell me when it was. The 90s and into the early 2000s at least. You would get laughed out the room and then fucking fired if you hooked anything critical up to the internet.
- hilbert42 2y ago
  "You would get laughed out the room and then fucking fired if you hooked anything critical up to the internet." Perhaps this happened where you were, and lucky you it seems you were in a good environment. But back then I was in IT management and I had precious little power to stop it especially given other senior managers were the culprits. The operation had another function and not IT as its primary role. Moreover, I saw very simular problems in other organizations that I was familiar with. Also, during that period I was with another outfit whose principal function was surveillance—not of people but of info and physical stuff and I can assure you that whilst the system worked well try as we might it wasn't watertight.
  - kortilla 2y ago
    This happened everywhere. I worked in a company offering management agents that had additional features if they hooked up to the internet (“cloud management” or “SaaS” before that term existed). Hospitals would never hook stuff to the internet. Industrial control systems, etc were all huge show stoppers. No offense, but I think you’re thinking of a later era. Most protocols back then had literally no auth or auth that was never deployed. The thought of critical systems safe enough to be exposed to the internet was just really unfathomable.
- autoexec 2y ago
  I agree. Constant internet access and the assumption that other people should be able to push new code to your machine and have it run without you even being aware of it has killed all hope of resiliency. I miss the days when any application that dared to phone home even just to check for updates was considered spyware. Today there is are huge numbers of people who have access to install and run whatever new code they want on our systems whenever they feel like it. If it's not the AV software, it's the browser, or the video card, or the mouse driver, or windows itself. It's totally unmanageable.
joe_the_user 2y ago
>> "What Happened to Digital Resilience?" > Was there ever such a time? If so then tell me when it was It seems very plausible that "digital resilience" that this has been buzz phrase repeated often enough in meetings of security-adjacent corporate bureaucrats that some number of people convinced themselves it was a real thing. And the same divorced-from-specifics approach allows these decision makers to paper over any and all choices that inherently weakened security 'cause the triage needed to partially protect the resulting structurally insecure system can be presented with similar glowing buzz phrases.

dralley 2y ago

This is an area where studying Ukraine's experience will be very useful (and probably has already been useful) There were years of cyberattacks against pretty much every peice of critical infrastructure they have. Things went down, there were disruptions, but they adapted. Sometimes by falling back to low-tech solutions, sometimes by developing new systems with robustness into new systems and purging the old (much easier to politically justify when the problem is tangible and immediate). I seem to recall that one of the first things we did when tensions started ramping up was sending teams of cyber security experts from the NSA to help them lock down and root out infiltrations.

emseetech 2y ago
| Sometimes by falling back to low-tech solutions My first thought in all this was wondering if there's a business opportunity for a consulting firm or startup that designs and manages offline paper backup systems that can quickly and seamlessly integrate back with digital systems once they come back online.
- SoftTalker 2y ago
  The problem is that if you aren't regularly training employees on those manual fallback systems, when you have to suddenly activate them, nobody will know what to do. Even if they have been trained on what to do, the processes will not be second nature. In real use, they will hit situations that the paper forms or training didn't cover, and will have to make up something on the spot, which they will each do differently. Fully comprehensive, regularly trained manual operations are very expensive to develop and test. Only the most safety-critical organizations will be able to justify and have the resources to effectively implement them. Air-traffic control, hospitals, nuclear plants, etc. And, they already have done it.
- hilbert42 2y ago
  "…offline paper backup systems that can quickly and seamlessly integrate back with digital systems once they come back online." It's not offline paper backups that are needed but rather the reverse—offline paper-based systems used as masters! The heart of any critical infrastructure—specifically the part of the heart that's the most vulnerable is comparatively small compared to the large masses of ancillary data and thus could be managed on a paper-based database (as they once were before computers). With computers and IT infrastructure as they're currently implemented—not as computer science says they ought to be—a secure filing cabinet/paper-based database is much more secure than an ephemeral one that has no physical or volumetric presence and which takes precious little to shove it from one end of the planet to the other. The caveat is of course, the database must be secured against physical access and located in a secure building, etc. Let me state why. Comparatively speaking, in recent times there are very few secure bank vaults and such that criminals have breached. The number is so small I can't remember when I'd last heard of a bank vault robbery. Another way of looking at it is to ask yourself when was gold last stolen from Fort Knox or the Bank of England, or $100 bills stolen from the US treasury/mint before their distribution. Why so few robberies you may well ask. We've had hundreds of years of experience locking up these valuables and although the current systems used to secure them aren't watertight and likely never will be, they're nevertheless sufficiently secure to the extent that the few breaches that do occur from time to time are manageable. With physical security, we've found a workable balance between security and workability. With the few robberies that occur it's not worth the effort of tightening security further, to do so would not only add considerably to the cost but also physical access would be more difficult thus less convenient to use because of the additional protocols that would have to be put in place to reach the higher security level. Also, think for a moment that if you could gain access to a secured paper-based database how quickly could you copy it, and how would you copy it? Right, both would be very difficult. On the other hand once an electronic database has been breached megabytes if not gigabytes can be sucked out within seconds. In practice, the electronic/digital world has nothing as 'bulletproof' as a physically secure system. Given the statistics—the rate of cyber breaches, personal data stolen, Bitcoin thefts, etc., etc. that occur not on a yearly but rather on a daily basis, one simply can't argue that collectively IT/electronic systems are more secure than physical, paper-based ones. Back to the physical database: a secure paper-based database would always be offline, if some data are needed from it then they have to be extracted manually, then vetted and encrypted before being put on line (that's if it's actually necessary to put highly secure stuff on line at all). As things stand, owners of information have a choice, store it in an electronic system and take advantage of the operational advantages that it offers or use secure paper-based storage and suffer the inconvenience. One can't have it both ways. The reason why we've so may data breaches is that the average punter far prefers electronic data systems for their convenience. On evidence, convenience is seen as more important, in practice its value far outweighs data security and integrity.

pjc50 2y ago

The "cyber agencies" focus on offence, because that's easy to score points with and appear to be doing something, whereas defence is a very boring job of securing a zillion outdated endpoints. Or trying to get profitable megacorps to do something less vulnerable and less profitable.

TrueDuality 2y ago
Offense is also easy in that there is a ton of software out there, and you just need to find one vulnerability. There is a "win" condition" Defense is impossible as there is a ton of software and you need to protect all of it every time, there is only a "lose" condition.
tintor 2y ago
The "cyber agencies" could focus on pen testing of domestic companies, and issuing fines to insecure ones.

TrueDuality 2y ago

> Was there ever such a time? If so then tell me when it was. It was a goal for a long time, and I'd say we use to be more resilient pre-cloud SaaS auto-update everything. When every software solution installation is on private networks, with fundamentally different architectures (both machine and topology), along with a wide selection of even very poor quality software, was a lot more resilient than what we have today. Today a single outage in a single service (say AWS) can grind a large number of companies to a halt. A bad update like this one immediately impacts everyone all at once and has a domino effect. That didn't use to happen. We've been concentrating our collective architecture into a few best practice tools but that all become single points of failure for not only digital attacks, but misconfigurations, mismanagement, company failures, exhausted underpaid engineers, optimizations, etc. > Hardening systems against vulnerabilities means making them less convenient/easy to use and people instantly balk against that. This isn't necessarily true, and I'd argue quite the opposite direction has been happening in the security industry over the past decade or so. People realized that hard security would only cause users to find simple predictable bypasses that would overall _weaken_ the security posture. You just have to look at the evolution of NIST recommendations around passwords to see this happening. Must change a password every 90 days that can't be the same as your last 10 passwords and complex password requirements? Well users are going to use the minimum size in predictable patterns and just increment a number at the end. Those old password hashes you have to keep around to check if the user is reusing the password? Those are a liability that, when broken, tell the attacker which pattern each user is using. Not the case anymore and there is a lot more usable security rolled that is entirely transparent to end users or almost entirely transparent. Think about how prevalent and bad captchas used to be on the website and how easy they were to circumvent. Cloudflare's and Google's captcha solution are pretty transparent and has much greater efficacy than the old ones. Did Microsoft's general and on-going laxness contribute to bad security practices? Absolutely, but that is one ecosystem that had weird other by the nature of how inherently unstable that environment was and is not and hasn't except for maybe a brief peak ever been a core foundation of the internet infrastructure, just enterprise infrastructure unfortunately. They definitely never got the memo about usable or transparent security. I hope they're at least trying behind the scenes now.

hilbert42 2y ago

lenerdenator 2y ago

> In one way I'm amazed at how laxadasical the US and others are towards these threats and that they have not done more to harden the vulnerabilities. On the other hand, it's obvious: cost is one factor but I reckon another bigger one is 'convenience'. Hardening systems against vulnerabilities means making them less convenient/easy to use and people instantly balk against that. "Show me the incentives, and I'll show you the outcomes." - Charlie Munger. We do not incentivize companies to operate secure, redundant, reliable computer systems. We incentivize companies to make the number at the bottom of the spreadsheet beat the expectations some analyst in Lower Manhattan set 90 days prior. And since companies handle the majority of societal work in the United States, that's how most critical systems are designed. Now, there's a chance that this will play out in court, and that Crowdstrike will have to be bought out to make up for the damages their customers suffered starting on July 19th. However, that will take years, and the outcome could very well be that the plaintiffs will receive symbolic or even no damages. By then, the market will have hedged, captured regulatory authorities, cut its losses, and just altogether moved on. The assets will be purchased in a firesale by people who see this as "creative destruction" and won't care that peoples' lives were put at risk because of this. And the cycle will continue.

hilbert42 2y ago
...And I can't agree more. The question is what can be done, if anything. But I've a solution in my wildest dreams as a dictator. :-)
- lenerdenator 2y ago
  You introduce consequences against the people who have created the system that we're in now.
  - dasil003 2y ago
    Even assuming you could narrow this down to a small enough set of people that can credibly be held responsible for creating the system we have now, and assuming you could impose consequences on them without violating their civil rights, and assuming they learnt their lesson and would actively take precautions to avoid their actions leading to such a systemic failure in the future, at best this would only influence those particular actors to avoid the previous failures. The next systemic failure would look quite different on the ground and come from different individuals pursuing different goals who would not have learnt any of the previous lessons. The only people who would see the connection would be more experienced people and/or intellectuals looking from a higher zoom level, but likely would not be empowered to really do anything to stop it given all the direct financial incentives motivating a much larger group of people to direct action. If our culture had more respect for elders and/or thinkers that could be a start, but even then it would still be an uphill battle in a capitalist society.

binary132 2y ago

What makes you think only a foreign adversary might want illegitimate access to our computers?

kortilla 2y ago

>Was there ever such a time? If so then tell me when it was. The 90s and into the early 2000s at least. You would get laughed out the room and then fucking fired if you hooked anything critical up to the internet.

hilbert42 2y ago
"You would get laughed out the room and then fucking fired if you hooked anything critical up to the internet." Perhaps this happened where you were, and lucky you it seems you were in a good environment. But back then I was in IT management and I had precious little power to stop it especially given other senior managers were the culprits. The operation had another function and not IT as its primary role. Moreover, I saw very simular problems in other organizations that I was familiar with. Also, during that period I was with another outfit whose principal function was surveillance—not of people but of info and physical stuff and I can assure you that whilst the system worked well try as we might it wasn't watertight.
- kortilla 2y ago
  This happened everywhere. I worked in a company offering management agents that had additional features if they hooked up to the internet (“cloud management” or “SaaS” before that term existed). Hospitals would never hook stuff to the internet. Industrial control systems, etc were all huge show stoppers. No offense, but I think you’re thinking of a later era. Most protocols back then had literally no auth or auth that was never deployed. The thought of critical systems safe enough to be exposed to the internet was just really unfathomable.
autoexec 2y ago
I agree. Constant internet access and the assumption that other people should be able to push new code to your machine and have it run without you even being aware of it has killed all hope of resiliency. I miss the days when any application that dared to phone home even just to check for updates was considered spyware. Today there is are huge numbers of people who have access to install and run whatever new code they want on our systems whenever they feel like it. If it's not the AV software, it's the browser, or the video card, or the mouse driver, or windows itself. It's totally unmanageable.

joe_the_user 2y ago

>> "What Happened to Digital Resilience?" > Was there ever such a time? If so then tell me when it was It seems very plausible that "digital resilience" that this has been buzz phrase repeated often enough in meetings of security-adjacent corporate bureaucrats that some number of people convinced themselves it was a real thing. And the same divorced-from-specifics approach allows these decision makers to paper over any and all choices that inherently weakened security 'cause the triage needed to partially protect the resulting structurally insecure system can be presented with similar glowing buzz phrases.