5 ms·

It's sort of curious, how exactly do we avoid the "if it ain't broke, don't fix it" mentality for companies that aren't super tech-focused? Like, sure, maybe a

It's sort of curious, how exactly do we avoid the "if it ain't broke, don't fix it" mentality for companies that aren't super tech-focused? Like, sure, maybe a company like Facebook or Google can fairly easily be up-to-date, they have a million engineers running around there, but a lot of companies decidedly aren't tech-focused.

ivalm 5y ago
I'm not sure it needs to be "fixed." Certain systems probably can be run for decades without meaningful updates as long as security vulnerabilities are properly addressed. For air-gapped systems you can potentially run with no updates/patches for many, many decades. The main issue is hardware support life-cycle.
- tombert 5y ago
  That's fair; upon thinking about it, basically every appliance I have (oven, microwave, etc) has some kind of computer in it, but they have no network access and I will probably have all of them for several decades.
- sofixa 5y ago
  The problem is that unless the software is actively worked on and supported by someone, fixing a decades old vulnerability is going to be complicated when it does show up ( not to mention that the main way it can show up if you don't do regular pentesting is by a malicious actor exploiting it, aka too late). The people who set it up are retired, dead, or don't remember; nobody is using that tech anymore so community resources would be of limited use. Unless there's a vendor still alive and actively maintaining your old version in the case of third party software, you're pretty close to screwed. For first party software your only hope is that the people setting it up did a good job with documentation. It's drastically easier to maintain and patching recent software. You don't need to have everything written in JS and running on Kubernetes for that.
  - shapefrog 5y ago
    1. It is far easier to fix the old bare bones system that doenst really break than the new shiny all singing dancing system. You dont ever update the code, so nothing new ever breaks. 2. There is a very old guy, who does absolutely fuck all, is grumpy and at the annual review department heads ask why we are paying them 3x anyone else in the department. He is the guy who keeps the legacy system alive. 3. The big project focuses on messaging between the legacy system and the new system, which breaks constantly and is unreliable anyway. More than that, I personally have seen multiple careers of senior managers, 7/8 figure salaries, destroyed beyond repair over new IT projects. $20mil 2 year project becomes a 5 year $180mil white elephant. You can seed a software company, develop a product and deploy it in these organisations faster and cheaper.
mnd999 5y ago
When I was there Goldmans was very tech focused. They’ve built a lot of interesting stuff internally, including SecDb which has been mentioned elsewhere on this thread. They rarely talk about it externally though, and it’s all built to facilitate their primary businesses.
squeaky-clean 5y ago
If it ain't broke, why would it need to be fixed?
kakwa_ 5y ago
The "if it ain't broke, don't fix it" mind set is only one part of the equation. Most organizations acknowledge that it's generally preferable to run up to date and modern software. But at the same time, updating or, even worse, replacing a piece of technology can be extremely complex, and also quite risky, so people naturally chose the path of least resistance and keep old pieces software (or hardware) running. And even Facebook or Google are likely not immune to that. For example, Google used to run RedHat 7.1 (release in 2001) for a very long time, only transitioning recently to a more modern Debian OS base. http://events17.linuxfoundation.org/sites/events/files/slides/ProdNG_0.pdf http://events17.linuxfoundation.org/sites/events/files/slide... This is of course with huge caveats as Google has the resources and the people to basically maintain their own distribution, and practically did so upgrading this RH 7.1 base with security and bug fix patches, and including newer components such as recent kernels. But overall, the base system Google used was quite old, almost to a scary extent to be honest. Also, the upgrade process detailed in the slides is really interesting and show how complex and involved it was for Google. For organizations less able to attract very good engineers this kind of upgrade would either be extremely painful (bugs, downtime, etc), or even nearly impossible.

ivalm 5y ago

I'm not sure it needs to be "fixed." Certain systems probably can be run for decades without meaningful updates as long as security vulnerabilities are properly addressed. For air-gapped systems you can potentially run with no updates/patches for many, many decades. The main issue is hardware support life-cycle.

tombert 5y ago
That's fair; upon thinking about it, basically every appliance I have (oven, microwave, etc) has some kind of computer in it, but they have no network access and I will probably have all of them for several decades.
sofixa 5y ago
The problem is that unless the software is actively worked on and supported by someone, fixing a decades old vulnerability is going to be complicated when it does show up ( not to mention that the main way it can show up if you don't do regular pentesting is by a malicious actor exploiting it, aka too late). The people who set it up are retired, dead, or don't remember; nobody is using that tech anymore so community resources would be of limited use. Unless there's a vendor still alive and actively maintaining your old version in the case of third party software, you're pretty close to screwed. For first party software your only hope is that the people setting it up did a good job with documentation. It's drastically easier to maintain and patching recent software. You don't need to have everything written in JS and running on Kubernetes for that.
- shapefrog 5y ago
  1. It is far easier to fix the old bare bones system that doenst really break than the new shiny all singing dancing system. You dont ever update the code, so nothing new ever breaks. 2. There is a very old guy, who does absolutely fuck all, is grumpy and at the annual review department heads ask why we are paying them 3x anyone else in the department. He is the guy who keeps the legacy system alive. 3. The big project focuses on messaging between the legacy system and the new system, which breaks constantly and is unreliable anyway. More than that, I personally have seen multiple careers of senior managers, 7/8 figure salaries, destroyed beyond repair over new IT projects. $20mil 2 year project becomes a 5 year $180mil white elephant. You can seed a software company, develop a product and deploy it in these organisations faster and cheaper.

mnd999 5y ago

When I was there Goldmans was very tech focused. They’ve built a lot of interesting stuff internally, including SecDb which has been mentioned elsewhere on this thread. They rarely talk about it externally though, and it’s all built to facilitate their primary businesses.

squeaky-clean 5y ago

If it ain't broke, why would it need to be fixed?

kakwa_ 5y ago

The "if it ain't broke, don't fix it" mind set is only one part of the equation. Most organizations acknowledge that it's generally preferable to run up to date and modern software. But at the same time, updating or, even worse, replacing a piece of technology can be extremely complex, and also quite risky, so people naturally chose the path of least resistance and keep old pieces software (or hardware) running. And even Facebook or Google are likely not immune to that. For example, Google used to run RedHat 7.1 (release in 2001) for a very long time, only transitioning recently to a more modern Debian OS base. http://events17.linuxfoundation.org/sites/events/files/slides/ProdNG_0.pdf http://events17.linuxfoundation.org/sites/events/files/slide... This is of course with huge caveats as Google has the resources and the people to basically maintain their own distribution, and practically did so upgrading this RH 7.1 base with security and bug fix patches, and including newer components such as recent kernels. But overall, the base system Google used was quite old, almost to a scary extent to be honest. Also, the upgrade process detailed in the slides is really interesting and show how complex and involved it was for Google. For organizations less able to attract very good engineers this kind of upgrade would either be extremely painful (bugs, downtime, etc), or even nearly impossible.