9 ms·

Patching is hard; so what?

120 points by runesoerensen 9y ago

empath75 9y ago
I know equifax is probably dealing with a hairy legacy system, but really you should design all your systems to be able to push out a fully tested blue/green deploy in hours not months. There's no excuse not to with Jenkins/aws/docker/etc. You should be testing and pushing updates regularly as a matter of course, not as a red alert only when security vulnerabilities are published.
- ams6110 9y ago
  That's not the mindset of enterprise IT though. The mindset of enterprise IT is "if it works, don't touch it." This mindset developed because of decades of dealing with arcane, complicated software that was at the same time critical to business operations. I'm aware of a major university that still has a three-week "change freeze" at the start of every semester because of this attiude and I'm sure they are not the only one.
  - topkeker 9y ago
    I've worked in enterprise security teams for the last decade. This is spot on. It's retarded and will never change because there is no accountability. Corporate IT don't give no fucks.
    - CiPHPerCoder 9y ago
      Then let's hope Equifax shakes the foundation of their egos and they start to give fucks about this sort of thing.
    - toast0 9y ago
      Corporate IT is doing what software developers have taught them through experience and pavlovian condition: Don't touch anything, because it will break if you touch it. IFF people who release software learn how to release security patches and bugfixes that don't include new features or break existing parts (except when breaking changes are actually needed), then, after several years of good results from pushing changes, Corporate IT will be conditioned to just push security fixes, it's not a big deal, plus it's important.
  - empath75 9y ago
    Right. Which is why ci/cd and devops, etc were invented. If you're not patching and updating frequently, you're committing business malpractice.
  - watwut 9y ago
    Three weeks freeze twice a year during the most critical time does not sound that horrible to me. Software developers tend to break system with changes. That is not some kind of mythical rare issue - that happens often (with some vendors very often).
    - user5994461 9y ago
      Agreed. A company that doesn't have change freezes at the most critical times has very poor practices.
- watwut 9y ago
  The "change takes time" issue does not happen because of technical problems. It happens because of organizational problems. Technical problems with patching are easiest to solve. Side note: fully tested within hours assumes small stable software. Large software with frequent changes into it needs to be manually tested. If you have microservices, you still have to test whether something important did not broke.
chasb 9y ago
For teams that use a DevOps model, fast, predictable deploys that can be safely rolled back are important for security, for this reason. If deploys are like playing Jenga on a sailboat, you're not going to be able to patch fast or safely. That said, even becoming aware a CVE exists in the first place is still a problem for many teams. There are plenty of good options, it's just underinvested in early on.
- inetknght 9y ago
  Even if they learn of a CVE there's no guarantee a patch exists for it. You might even be stuck with no alternative but to disable whatever the attack vector is... which could be key to your business.
- gerdesj 9y ago
  What on earth is a "DevOps model"? (yes I did see your ahem model). Please forgive me if I don't hire you to keep my kool ... whatever ... webby thing safe. I understand that you are commentating in a second language but you used the DevOps safe word and automatically lose.
- rdtsc 9y ago
  Exactly. I'd say the check of how fast can a security patch be deployed from the moment upstream OS or library releases is a good benchmark for a team/system and should be evaluated at the same level as log audits, what ports are open and what openssl versions are running. > it's just underinvested in early on. And for cases where there is no personal or financial data involved it is shameful and just a bad practice to not have the system which allow fast security patch deployed. For a system that holds sensitive personal information for hundreds of millions of people it should a very serious issue not to patch the system for months after vulnerability was known.
  - Johnny555 9y ago
    Exactly. I'd say the check of how fast can a security patch be deployed from the moment upstream OS or library releases is a good benchmark for a team/system and should be evaluated at the same level as log audits, what ports are open and what openssl versions are running. Deploying patches is easy, validating that they aren't going to break anything (sometimes in hard to detect ways) is hard.
lucb1e 9y ago
TL;DR: patching is hard, so use defense in depth to minimize/mitigate the impact of individual components becoming insecure from time to time.
guelo 9y ago
Patching isn't that hard. Once Equifax finally noticed the intrusion (intrusion detection system?), they took the system down and patched it within a day.
- hyperpape 9y ago
  I'm working with a fifteen year old app that's slowly modernizing. There's two types of updates, the normal kind that's slow, and "everything is on fire, get the fix into production ASAP, then pray". The latter kind is super quick, and it's what Equifax did here. But that doesn't mean patching is easy for them. It means patching without doing whatever laborious QA/manual testing they normally do is easy. (Btw: it might sound like I'm defending Equifax. I'm not. I fully agree with the article. They have a responsibility to figure out how to secure their systems. "We can't patch quickly" may be true for them, but they needed to figure out how to change that).
  - sqldba 9y ago
    Yes. You've hit the nail on the head. They'll have such a tight bureaucracy on patching that it's impossible (which is an IT problem in their eyes) but the second a senior manager says just do it - its quick and painless. I see this over and over again everywhere.
    - hyperpape 9y ago
      It's quick, but it's risky, because testing is manual, you don't have blue-green deploys, etc. So it's not that it becomes painless, it's just that their are circumstances where you accept the risk.
  - _asummers 9y ago
    > It means patching without doing whatever laborious QA/manual testing they normally do is easy. Everyone learns how to do Continuous Deployment when prod is on fire.
    - hyperpape 9y ago
shawkinaw 9y ago
Maybe patching is hard. But the real question is why one vuln in their web framework allowed unfettered access to basically all of their data. There are much deeper problems here than slow patching.
- Johnny555 9y ago
  the real question is why one vuln in their web framework allowed unfettered access to basically all of their data The hack involved "primarily names, Social Security numbers, birth dates, addresses and, in some instances, driver's license numbers", that's a very limited subset of the data they hold on people. I'm not defending them, but this was not "basically all of their data".
rdtsc 9y ago
> Specifically, these folks point out that patching is hard. The gist of these points is that you can’t expect a major corporation to rapidly deploy something as complex Then they should be taken to court and be forced to pay for it. It's like saying about a company which runs trains and didn't do maintenance which caused an accident to happen "Well, you see maintaining the brakes on the trains is too hard so don't blame them too much...". > The gist of these points is that you can’t expect a major corporation to rapidly deploy something as complex as a major framework patch across their production systems. So those corporations will be hacked, and then they should be dragged to court, made to pay, some might even go bankrupt and disappear altogether. Eventually only the companies that manage to build better infrastructure will survive. Oh well, one can dream...
- sillysaurus3 9y ago
  The fact that Equifax reported the breach at all represents a good faith understanding: they tell the world, and in return the world does not make them cease to exist. If you kill companies over security incidents, then magically you will no longer hear about security incidents.
  - zaptheimpaler 9y ago
    They aren't doing us a favor that we have to reciprocate by reporting a breach.. its more of a responsibility. If its not required by law already, it should be.
  - mirashii 9y ago
    It's not a good faith gesture, it's required by law almost everywhere they operate that they notify of a security breach of PII. http://www.ncsl.org/research/telecommunications-and-information-technology/security-breach-notification-laws.aspx http://www.ncsl.org/research/telecommunications-and-informat...
    - sillysaurus3 9y ago
      And yet, as if by magic, you will no longer hear about security incidents if you kill companies.
chronid 9y ago
> I don’t operate production systems, but I have helped to design a couple of them. So I understand something about the assumptions you make when building them. Start by operating production systems. You will rapidly discover that patching is not a technical issue (as I already said last time there was a patching discussion on HN). Patching is technically "easy". But if the business does not prioritize (and allocate time towards) patching (and testing that systems work with after) patching won't get done. There are features to develop and deadlines to meet and new systems to turn on and old systems to retire and oh-my-god don't touch those servers otherwise the vendor will not support the prod environment anymore (no longer certified, yay!) and it costs us 30000$ to get certified again. Too many times I see HN assuming that everyone is running on a public cloud with B/G deployments and immutable infrastructure you can rebuild and redeploy easily. Unfortunately that's not the case.
- sqldba 9y ago
  Those "don't touch" systems are death. It's usually in contracts with the big players like Honeywell and others. You literally can't touch them without breaking the contract. The second you call for support and they find a patch - you have to uninstall before they continue - even if it's unrelated. BTW hands up if that includes WannnaCrypt. Yep.
  - alkonaut 9y ago
    This is the same situation as those MRI machines with XP on them. The answer is don't buy those machines or don't connect machines you can't patch to the network. If hospitals can't buy them because those are the criteria, then industry will change. Don't sign contracts that leaves you in the situation that you can't patch machines.
    - ams6110 9y ago
      If a hospital doesn't have an MRI machine, it won't be able to compete. It's really not an option to say "I won't buy it unless X." You buy the one that meets most of your needs for the best price. Not buying is not really a choice.
sqldba 9y ago
Patching at an OS level (not what the author is talking about, probably) is extremely extremely hard - from an enterprise Microsoft platform production point of view 1) Because there's no budget for tools. It's mostly WSUS and SCCM if you're lucky. 2) Because staff don't even know how to use those tools properly. Can't blame them because there's very poor documentation from MS and very little accessible training on design patterns that integrate them into large corps. 3) It's outsourced. So it's even worse than above. 4) Because management fights it every step of the way. They hate patching because they're stuck in the 80s where they think every patch is deeply documented and you should only apply what's necessary. And you should only apply security - ignoring updates which give critical stability fixes. 5) Because the enterprise as a whole is so poorly documented it's unclear who owns what or if a rollback is possible and what it would affect. So everything is in CYA overdrive. 6) And then nobody working there wants to own it because it's not sexy to patch. (As a geek I think patching is sexy and very fulfilling work, but I'm in the extreme minority). None of this excuses anything. But most largish companies are still in the 80s, with IT managers in the 80s. I can't wait for them all to die.
rythie 9y ago
Is it so hard though? Do Facebook, Google etc. have this problem too? I doubt it. Systems need to be designed to be patched regularly from the outset. Yes, defense in depth can help, but it's not defense in depth if you one of the layers is pretty much always broken. If they are relying on legacy systems that can't be proactively and regularly be patched, they shouldn't be holding that data. I think part of the problem here, is that we can't see the risks companies are running with our data and goverment doesn't regulate it. P.S. I do manage systems, though not on this scale.
- hn_throwaway_99 9y ago
  > Is it so hard though? Do Facebook, Google etc. have this problem too? I doubt it. Companies with monopolistic pricing power like Facebook and Google are a poor comparison for almost any other company. Many problems become very easy when you have a near infinite supply of money. Other businesses need to work with much more difficult constraints.
  - genmud 9y ago
    Average IT spend runs between 2 and 5% of revenue for fortune 500s. For reference... GM makes 166 billion/year in revenue, and on the low end their IT spend is probably more than 2 billion/year. In 2004 they were making the news because they went from 4B/year in 1998 to 3B/year. In 2006 they were making 190B/year, so I doubt their IT budget has changed significantly and is on the low end compared to other companies in the fortune 500 list. Facebook and other technology companies might spend a lot more on IT, but that's part of them doing business. Don't confuse the fact that technology companies do technology better with the fact that any company in the fortune 500 list have HUGE IT budgets. At a large company that I used to work for we had an IT budget of more than 1 billion USD and they didn't patch very well either. Our budget was actually smaller than many of our peers. If you have a IT infrastructure/application budget of 1B or more and you can't patch, your doing it wrong and there is no excuse. For a full baked/managed solution, you can find folks to do an entire program for less than 12m/year for a 200k node network.
lsefwegw 9y ago
https://socialnews.rakuten.co.jp/link/1411239 https://socialnews.rakuten.co.jp/link/1411239
alkonaut 9y ago
It's pretty simple to patch if you designed the system to be patchable in the first place and assign enormous cost to not patching. If you assume running an outdated version of struts for 3months is a $1B expense then you'll patch even if it means disruption to business operations, or will breach some conract. And if the system is properly designed then anything can be rolled back if it turns out there were unforeseen consequences.
nickpsecurity 9y ago
"I don’t dispute this point. It’s absolutely valid." I do dispute it. Patching is not hard. It's incredibly easy to the point that some places and projects have the process automated from notification to application to testing for breakage to deployment. For reducing downtime, one can use clusters with rolling deployments of patches. Patching is only hard for things such as web applications when the company's application or processes are done in a way that makes patching hard. As in, they have to be incompetent or just not care about IT. Far as competence, your claim about fragile systems is a good example that might have happened. " then you’ve implicitly made the decision that you’re never ever going to allow those vulnerabilities to fester." You're thinking like an engineer that cares about quality. You instead should think like an Equifax CIO or something. To start with, this is a company that collects PII against people's will to sell to third parties for their main goal of huge profits. Politics plays more a role than engineering talent in people getting the senior, management positions in such companies. They also tend to chase whatever is popular among Fortune 500, esp with cheap labor or ecosystems available. Java was one of the fads that financial sector was all over. Combine all this to have a company whose fad-chasing CIO keeps costs down and profits up applying the thing he or she read in a computer magazine with the cheapest talent available on a tight budget. The result of their work is a pile of garbage they have trouble patching. If you doubt this, just look at the security of the web site they deployed for credit monitoring and apparently to help hackers get at people interested in credit monitoring. Or they just made mistakes so easily avoided that they're either inexperienced beginners or don't care at all. "So what would those systems look like?" Well, they would have built it some time ago. So, let's work our way from old, high-assurance security toward something commercial and affordable from at least 2000-2005 era. The original work in securing data involved security kernels: https://www.acsac.org/secshelf/book001/19.pdf https://www.acsac.org/secshelf/book001/19.pdf Several of those are still available but expensive. Both security kernels and databases such as Trusted Rubix built for them. Today, those look more like the next link with companies such as Sirrix and Green Hills selling them commercially: https://os.inf.tu-dresden.de/papers_ps/nizza.pdf https://os.inf.tu-dresden.de/papers_ps/nizza.pdf In any case, we'd need a robust combination of OS, database, and application code. Nothing hits the database without going through the app server first. So, we embed our security policy into app server. How to implement it? Ever since Dijkstra's THE OS (1960's), we knew to specify correct behavior with preconditions, invariants, and post conditions. Then use anything from formal analysis to testing to runtime checks to ensure that behavior is enforced. The security kernels did the former where Design-by-Contract in Eiffel used tests and runtime checks. Got to pick method with best bang for buck. Two links to illustrate safer languages of past with Ada 2005 since we're looking at earlier stuff. http://www.adacore.com/knowledge/technical-papers/safe-secure/ http://www.adacore.com/knowledge/technical-papers/safe-secur... https://www.eiffel.com/values/design-by-contract/introduction/ https://www.eiffel.com/values/design-by-contract/introductio... Given labor situation, we use DbC in a safe language. All prior work showed simplicity was necessary for security. So, it would be an app server done in Eiffel, Ada, or (if absolutely necessary) Java/C++ using Design-by-Contract methodology plus lots of testing. The web component would be a middleware done similarly that basically translates web actions to simple protocol the server uses for requests/responses. Any new types of problems with web apps are mitigated in the simple framework a la Airship CMS, caught with some kind of monitoring a la Spectre, or both. This would have to run on something robust. You'd have the database/storage, app server, web component, management, and monitoring which is behind one-way link (data diode) w/ logs copied to write-only media. A company from likely mainframe background now run by wise technologists might first make the software target IBM but portable. Balancing reliability, security, and cost, the best targets would be AS/400 or AIX. The former is a capability-oriented architecture with few 0-days whereas other is a rock-solid UNIX. Portability means they'd migrate part or all of this to new OS's as they showed up. Eventually they'd notice OpenBSD's security advantages or buy one of those certified-secure kernels w/ POSIX layer. Depends on their viewpoint and budget. Probably throw some clustering software with rolling releases in there since OpenVMS and NonStop had high-availability in the 1980's. It was well-known strategy. So, there was a straight-forward way to make a robust stack with a web front end which gets more robust over time. This could be done in high availability or just easy patches. They instead end up with fad-driven crud possibly running on other crud that's easy to hack but hard to patch. Just bad engineering that's typical in the market versus methods such as I described. Also note that there were (are) large institutions using some of what I recommended for those benefits. The tools and people just cost a bit more which doesn't align with incentives of companies as greedy and IT hating as Equifax. Sure turned out extra profitable, didn't it? ;)
BrandoElFollito 9y ago
The whole problem boils down to a choice :security or availability. If you prefer to have systems 100% up (no downtime for patches and fixing potential issues after the patch) then it is fine. The actual risk (described beyond "having security") has to be accepted by the owner of the data / service and the problem is over. I have yet to see a CFO who will accept in written that his systems are prone to hacking, theft, manipulation. Until the CISO does not force the company to sign off such risks, he or she will be the one who fucked up.

120 points by runesoerensen 9y ago

I know equifax is probably dealing with a hairy legacy system, but really you should design all your systems to be able to push out a fully tested blue/green deploy in hours not months. There's no excuse not to with Jenkins/aws/docker/etc. You should be testing and pushing updates regularly as a matter of course, not as a red alert only when security vulnerabilities are published.

ams6110 9y ago
That's not the mindset of enterprise IT though. The mindset of enterprise IT is "if it works, don't touch it." This mindset developed because of decades of dealing with arcane, complicated software that was at the same time critical to business operations. I'm aware of a major university that still has a three-week "change freeze" at the start of every semester because of this attiude and I'm sure they are not the only one.
- topkeker 9y ago
  I've worked in enterprise security teams for the last decade. This is spot on. It's retarded and will never change because there is no accountability. Corporate IT don't give no fucks.
  - CiPHPerCoder 9y ago
    Then let's hope Equifax shakes the foundation of their egos and they start to give fucks about this sort of thing.
  - toast0 9y ago
    Corporate IT is doing what software developers have taught them through experience and pavlovian condition: Don't touch anything, because it will break if you touch it. IFF people who release software learn how to release security patches and bugfixes that don't include new features or break existing parts (except when breaking changes are actually needed), then, after several years of good results from pushing changes, Corporate IT will be conditioned to just push security fixes, it's not a big deal, plus it's important.
- empath75 9y ago
  Right. Which is why ci/cd and devops, etc were invented. If you're not patching and updating frequently, you're committing business malpractice.
- watwut 9y ago
  Three weeks freeze twice a year during the most critical time does not sound that horrible to me. Software developers tend to break system with changes. That is not some kind of mythical rare issue - that happens often (with some vendors very often).
  - user5994461 9y ago
    Agreed. A company that doesn't have change freezes at the most critical times has very poor practices.
watwut 9y ago
The "change takes time" issue does not happen because of technical problems. It happens because of organizational problems. Technical problems with patching are easiest to solve. Side note: fully tested within hours assumes small stable software. Large software with frequent changes into it needs to be manually tested. If you have microservices, you still have to test whether something important did not broke.

chasb 9y ago

For teams that use a DevOps model, fast, predictable deploys that can be safely rolled back are important for security, for this reason. If deploys are like playing Jenga on a sailboat, you're not going to be able to patch fast or safely. That said, even becoming aware a CVE exists in the first place is still a problem for many teams. There are plenty of good options, it's just underinvested in early on.

inetknght 9y ago
Even if they learn of a CVE there's no guarantee a patch exists for it. You might even be stuck with no alternative but to disable whatever the attack vector is... which could be key to your business.
gerdesj 9y ago
What on earth is a "DevOps model"? (yes I did see your ahem model). Please forgive me if I don't hire you to keep my kool ... whatever ... webby thing safe. I understand that you are commentating in a second language but you used the DevOps safe word and automatically lose.
rdtsc 9y ago
Exactly. I'd say the check of how fast can a security patch be deployed from the moment upstream OS or library releases is a good benchmark for a team/system and should be evaluated at the same level as log audits, what ports are open and what openssl versions are running. > it's just underinvested in early on. And for cases where there is no personal or financial data involved it is shameful and just a bad practice to not have the system which allow fast security patch deployed. For a system that holds sensitive personal information for hundreds of millions of people it should a very serious issue not to patch the system for months after vulnerability was known.
- Johnny555 9y ago
  Exactly. I'd say the check of how fast can a security patch be deployed from the moment upstream OS or library releases is a good benchmark for a team/system and should be evaluated at the same level as log audits, what ports are open and what openssl versions are running. Deploying patches is easy, validating that they aren't going to break anything (sometimes in hard to detect ways) is hard.

lucb1e 9y ago

TL;DR: patching is hard, so use defense in depth to minimize/mitigate the impact of individual components becoming insecure from time to time.

guelo 9y ago

Patching isn't that hard. Once Equifax finally noticed the intrusion (intrusion detection system?), they took the system down and patched it within a day.

hyperpape 9y ago
I'm working with a fifteen year old app that's slowly modernizing. There's two types of updates, the normal kind that's slow, and "everything is on fire, get the fix into production ASAP, then pray". The latter kind is super quick, and it's what Equifax did here. But that doesn't mean patching is easy for them. It means patching without doing whatever laborious QA/manual testing they normally do is easy. (Btw: it might sound like I'm defending Equifax. I'm not. I fully agree with the article. They have a responsibility to figure out how to secure their systems. "We can't patch quickly" may be true for them, but they needed to figure out how to change that).
- sqldba 9y ago
  Yes. You've hit the nail on the head. They'll have such a tight bureaucracy on patching that it's impossible (which is an IT problem in their eyes) but the second a senior manager says just do it - its quick and painless. I see this over and over again everywhere.
  - hyperpape 9y ago
    It's quick, but it's risky, because testing is manual, you don't have blue-green deploys, etc. So it's not that it becomes painless, it's just that their are circumstances where you accept the risk.
- _asummers 9y ago
  > It means patching without doing whatever laborious QA/manual testing they normally do is easy. Everyone learns how to do Continuous Deployment when prod is on fire.
  - hyperpape 9y ago

shawkinaw 9y ago

Maybe patching is hard. But the real question is why one vuln in their web framework allowed unfettered access to basically all of their data. There are much deeper problems here than slow patching.

Johnny555 9y ago
the real question is why one vuln in their web framework allowed unfettered access to basically all of their data The hack involved "primarily names, Social Security numbers, birth dates, addresses and, in some instances, driver's license numbers", that's a very limited subset of the data they hold on people. I'm not defending them, but this was not "basically all of their data".

rdtsc 9y ago

> Specifically, these folks point out that patching is hard. The gist of these points is that you can’t expect a major corporation to rapidly deploy something as complex Then they should be taken to court and be forced to pay for it. It's like saying about a company which runs trains and didn't do maintenance which caused an accident to happen "Well, you see maintaining the brakes on the trains is too hard so don't blame them too much...". > The gist of these points is that you can’t expect a major corporation to rapidly deploy something as complex as a major framework patch across their production systems. So those corporations will be hacked, and then they should be dragged to court, made to pay, some might even go bankrupt and disappear altogether. Eventually only the companies that manage to build better infrastructure will survive. Oh well, one can dream...

sillysaurus3 9y ago
The fact that Equifax reported the breach at all represents a good faith understanding: they tell the world, and in return the world does not make them cease to exist. If you kill companies over security incidents, then magically you will no longer hear about security incidents.
- zaptheimpaler 9y ago
  They aren't doing us a favor that we have to reciprocate by reporting a breach.. its more of a responsibility. If its not required by law already, it should be.
- mirashii 9y ago
  It's not a good faith gesture, it's required by law almost everywhere they operate that they notify of a security breach of PII. http://www.ncsl.org/research/telecommunications-and-information-technology/security-breach-notification-laws.aspx http://www.ncsl.org/research/telecommunications-and-informat...
  - sillysaurus3 9y ago
    And yet, as if by magic, you will no longer hear about security incidents if you kill companies.

chronid 9y ago

> I don’t operate production systems, but I have helped to design a couple of them. So I understand something about the assumptions you make when building them. Start by operating production systems. You will rapidly discover that patching is not a technical issue (as I already said last time there was a patching discussion on HN). Patching is technically "easy". But if the business does not prioritize (and allocate time towards) patching (and testing that systems work with after) patching won't get done. There are features to develop and deadlines to meet and new systems to turn on and old systems to retire and oh-my-god don't touch those servers otherwise the vendor will not support the prod environment anymore (no longer certified, yay!) and it costs us 30000$ to get certified again. Too many times I see HN assuming that everyone is running on a public cloud with B/G deployments and immutable infrastructure you can rebuild and redeploy easily. Unfortunately that's not the case.

sqldba 9y ago
Those "don't touch" systems are death. It's usually in contracts with the big players like Honeywell and others. You literally can't touch them without breaking the contract. The second you call for support and they find a patch - you have to uninstall before they continue - even if it's unrelated. BTW hands up if that includes WannnaCrypt. Yep.
- alkonaut 9y ago
  This is the same situation as those MRI machines with XP on them. The answer is don't buy those machines or don't connect machines you can't patch to the network. If hospitals can't buy them because those are the criteria, then industry will change. Don't sign contracts that leaves you in the situation that you can't patch machines.
  - ams6110 9y ago
    If a hospital doesn't have an MRI machine, it won't be able to compete. It's really not an option to say "I won't buy it unless X." You buy the one that meets most of your needs for the best price. Not buying is not really a choice.

sqldba 9y ago

Patching at an OS level (not what the author is talking about, probably) is extremely extremely hard - from an enterprise Microsoft platform production point of view 1) Because there's no budget for tools. It's mostly WSUS and SCCM if you're lucky. 2) Because staff don't even know how to use those tools properly. Can't blame them because there's very poor documentation from MS and very little accessible training on design patterns that integrate them into large corps. 3) It's outsourced. So it's even worse than above. 4) Because management fights it every step of the way. They hate patching because they're stuck in the 80s where they think every patch is deeply documented and you should only apply what's necessary. And you should only apply security - ignoring updates which give critical stability fixes. 5) Because the enterprise as a whole is so poorly documented it's unclear who owns what or if a rollback is possible and what it would affect. So everything is in CYA overdrive. 6) And then nobody working there wants to own it because it's not sexy to patch. (As a geek I think patching is sexy and very fulfilling work, but I'm in the extreme minority). None of this excuses anything. But most largish companies are still in the 80s, with IT managers in the 80s. I can't wait for them all to die.

rythie 9y ago

Is it so hard though? Do Facebook, Google etc. have this problem too? I doubt it. Systems need to be designed to be patched regularly from the outset. Yes, defense in depth can help, but it's not defense in depth if you one of the layers is pretty much always broken. If they are relying on legacy systems that can't be proactively and regularly be patched, they shouldn't be holding that data. I think part of the problem here, is that we can't see the risks companies are running with our data and goverment doesn't regulate it. P.S. I do manage systems, though not on this scale.

hn_throwaway_99 9y ago
> Is it so hard though? Do Facebook, Google etc. have this problem too? I doubt it. Companies with monopolistic pricing power like Facebook and Google are a poor comparison for almost any other company. Many problems become very easy when you have a near infinite supply of money. Other businesses need to work with much more difficult constraints.
- genmud 9y ago
  Average IT spend runs between 2 and 5% of revenue for fortune 500s. For reference... GM makes 166 billion/year in revenue, and on the low end their IT spend is probably more than 2 billion/year. In 2004 they were making the news because they went from 4B/year in 1998 to 3B/year. In 2006 they were making 190B/year, so I doubt their IT budget has changed significantly and is on the low end compared to other companies in the fortune 500 list. Facebook and other technology companies might spend a lot more on IT, but that's part of them doing business. Don't confuse the fact that technology companies do technology better with the fact that any company in the fortune 500 list have HUGE IT budgets. At a large company that I used to work for we had an IT budget of more than 1 billion USD and they didn't patch very well either. Our budget was actually smaller than many of our peers. If you have a IT infrastructure/application budget of 1B or more and you can't patch, your doing it wrong and there is no excuse. For a full baked/managed solution, you can find folks to do an entire program for less than 12m/year for a 200k node network.

lsefwegw 9y ago

https://socialnews.rakuten.co.jp/link/1411239 https://socialnews.rakuten.co.jp/link/1411239

alkonaut 9y ago

It's pretty simple to patch if you designed the system to be patchable in the first place and assign enormous cost to not patching. If you assume running an outdated version of struts for 3months is a $1B expense then you'll patch even if it means disruption to business operations, or will breach some conract. And if the system is properly designed then anything can be rolled back if it turns out there were unforeseen consequences.

nickpsecurity 9y ago

"I don’t dispute this point. It’s absolutely valid." I do dispute it. Patching is not hard. It's incredibly easy to the point that some places and projects have the process automated from notification to application to testing for breakage to deployment. For reducing downtime, one can use clusters with rolling deployments of patches. Patching is only hard for things such as web applications when the company's application or processes are done in a way that makes patching hard. As in, they have to be incompetent or just not care about IT. Far as competence, your claim about fragile systems is a good example that might have happened. " then you’ve implicitly made the decision that you’re never ever going to allow those vulnerabilities to fester." You're thinking like an engineer that cares about quality. You instead should think like an Equifax CIO or something. To start with, this is a company that collects PII against people's will to sell to third parties for their main goal of huge profits. Politics plays more a role than engineering talent in people getting the senior, management positions in such companies. They also tend to chase whatever is popular among Fortune 500, esp with cheap labor or ecosystems available. Java was one of the fads that financial sector was all over. Combine all this to have a company whose fad-chasing CIO keeps costs down and profits up applying the thing he or she read in a computer magazine with the cheapest talent available on a tight budget. The result of their work is a pile of garbage they have trouble patching. If you doubt this, just look at the security of the web site they deployed for credit monitoring and apparently to help hackers get at people interested in credit monitoring. Or they just made mistakes so easily avoided that they're either inexperienced beginners or don't care at all. "So what would those systems look like?" Well, they would have built it some time ago. So, let's work our way from old, high-assurance security toward something commercial and affordable from at least 2000-2005 era. The original work in securing data involved security kernels: https://www.acsac.org/secshelf/book001/19.pdf https://www.acsac.org/secshelf/book001/19.pdf Several of those are still available but expensive. Both security kernels and databases such as Trusted Rubix built for them. Today, those look more like the next link with companies such as Sirrix and Green Hills selling them commercially: https://os.inf.tu-dresden.de/papers_ps/nizza.pdf https://os.inf.tu-dresden.de/papers_ps/nizza.pdf In any case, we'd need a robust combination of OS, database, and application code. Nothing hits the database without going through the app server first. So, we embed our security policy into app server. How to implement it? Ever since Dijkstra's THE OS (1960's), we knew to specify correct behavior with preconditions, invariants, and post conditions. Then use anything from formal analysis to testing to runtime checks to ensure that behavior is enforced. The security kernels did the former where Design-by-Contract in Eiffel used tests and runtime checks. Got to pick method with best bang for buck. Two links to illustrate safer languages of past with Ada 2005 since we're looking at earlier stuff. http://www.adacore.com/knowledge/technical-papers/safe-secure/ http://www.adacore.com/knowledge/technical-papers/safe-secur... https://www.eiffel.com/values/design-by-contract/introduction/ https://www.eiffel.com/values/design-by-contract/introductio... Given labor situation, we use DbC in a safe language. All prior work showed simplicity was necessary for security. So, it would be an app server done in Eiffel, Ada, or (if absolutely necessary) Java/C++ using Design-by-Contract methodology plus lots of testing. The web component would be a middleware done similarly that basically translates web actions to simple protocol the server uses for requests/responses. Any new types of problems with web apps are mitigated in the simple framework a la Airship CMS, caught with some kind of monitoring a la Spectre, or both. This would have to run on something robust. You'd have the database/storage, app server, web component, management, and monitoring which is behind one-way link (data diode) w/ logs copied to write-only media. A company from likely mainframe background now run by wise technologists might first make the software target IBM but portable. Balancing reliability, security, and cost, the best targets would be AS/400 or AIX. The former is a capability-oriented architecture with few 0-days whereas other is a rock-solid UNIX. Portability means they'd migrate part or all of this to new OS's as they showed up. Eventually they'd notice OpenBSD's security advantages or buy one of those certified-secure kernels w/ POSIX layer. Depends on their viewpoint and budget. Probably throw some clustering software with rolling releases in there since OpenVMS and NonStop had high-availability in the 1980's. It was well-known strategy. So, there was a straight-forward way to make a robust stack with a web front end which gets more robust over time. This could be done in high availability or just easy patches. They instead end up with fad-driven crud possibly running on other crud that's easy to hack but hard to patch. Just bad engineering that's typical in the market versus methods such as I described. Also note that there were (are) large institutions using some of what I recommended for those benefits. The tools and people just cost a bit more which doesn't align with incentives of companies as greedy and IT hating as Equifax. Sure turned out extra profitable, didn't it? ;)

BrandoElFollito 9y ago

The whole problem boils down to a choice :security or availability. If you prefer to have systems 100% up (no downtime for patches and fixing potential issues after the patch) then it is fine. The actual risk (described beyond "having security") has to be accepted by the owner of the data / service and the problem is over. I have yet to see a CFO who will accept in written that his systems are prone to hacking, theft, manipulation. Until the CISO does not force the company to sign off such risks, he or she will be the one who fucked up.