3 ms·

Here’s a fun one from a couple gigs ago: I was working as a DC tech at a small collocation provider, roughly 120 people company-wide. It was a standard shop th

Here’s a fun one from a couple gigs ago: I was working as a DC tech at a small collocation provider, roughly 120 people company-wide. It was a standard shop these days: O365 apps, Chrome on Windows 10, Slack, etc., totally managed and locked down. The IT staff were completely unable to control the Windows patching process. They made a lot of noise about gathering office hours for people so they could do OS & app updates when you weren’t working. Except it never, not one time worked. It was company folklore all the times the Windows patching process took down engineering devices that were actively in use performing upgrades or changes to core routing and switching devices. So because IT couldn’t control patching, core network services were repeatedly lost during maintenance windows. Because the host performing the work would suddenly reboot to patch itself. Now, I don’t know what lack of competence led to this situation. Maybe Windows Update is truly a horror story. But I’ll tell you one thing for certain: IT didn’t give one shit about the impact their mistakes had on production, on morale, or frankly about much of anything. Like the rest of the workforce, IT is overwhelmed, overworked, under appreciated, and fed up with the BS. But it sure seems like they go out of their way to screw people over.

BrandoElFollito 4y ago
I work in IT security, so I am probably one of those people who screw other people over by pushing patches to be installed no matter what. Behind the scene, it is a never ending war between me who says "if your want your data to be safe, we need to patch", and the business that says "no, those is quarter rend (or stuff like this)'. So I say" on, sign this paper what you start that you do not want your data to be secure" And it goes on forever. This is exhausting for everyone, but IT is zero the very bottom of thefood chain, we cannot claim sales or new products and provide z utility like water or trash collection. So this is not always that someone is trying to screw you over, out is just that technology works the way it works, with the money that is invested in it.
- hn_version_0023 4y ago
  Your comment exemplifies the attitude I'm talking about. You've presented a scenario that's false in it's premise: either your data is safe or it isn't. But really, whether a thing is protected isn't quite so zero or one. Whether your production environment is actually at risk for the exploit being patched is a reasonable thing to ask in all scenarios. "What's our exposure here? Can it wait a few days?" would be a reasonable thing to ask. But in my experience that isn't how it goes. It goes how you suggested: ITSecOps presents an easy-to-decide case based on a false premise, and then immediately asks for CYA if the company says no. Over time, this becomes a rubber stamp from management, who are also demoralized by the process. The solution is to not present every security patch as a sky-is-falling event, but rather to be reasonable about the threats one faces and the impact to the business of forcing updates.
  - pixl97 4y ago
    >Whether your production environment is actually at risk for the exploit being patched is a reasonable thing to ask in all scenarios. "What's our exposure here? Can it wait a few days?" would be a reasonable thing to ask. You would think it do be that way, but it don't.... Especially with things like Microsoft's megapatches they distribute these days. You are asking for the potential impact of anywhere from 10 to 100 patches custom tailored to the particular applications you're running on your server. Now you're asking for some large number of manhours of in depth analysis in a department that's already short on time. And as often as there is a patch that breaks the application, there is a patch that was released as a low impact that suddenly is high impact because it can be chained into an exploit, and now IT/SecOps is looking at the servers history to make sure it wasn't exploited at some time in the past. What is helpful here is not delaying patches, but instead having good testing environments that run the same version of the production application so the patch can be tested there first. Of course that costs a lot so you don't see it as often as you should.
  - BrandoElFollito 4y ago
    > You've presented a scenario that's false in it's premise: either your data is safe or it isn't. Do you work in cybersecurity? have you done security assessment - the real, technicals ones before? I did. I coded (a tiny, tiny) part of the Linux kernel. My teams break systems for a living. If you have actually done security assessments than you know, of course, that there is no way to assess whether a vulnerability is low or high before shit hits the fan. When MS says "critical", then I say "critical" as well. I am not going to plat smart ass and then realize that in the shitty cathedral of systems businesses build there is THE configuration that allows for an exploit. Anything lower than critical - I do not care/ This can be done later. Bit is it is critical, it will be applied immediately, breaking systems and important presentations of the C-whatever. If the business has something against me then feel free to escalate to my management and tell them that they do want their data to be insecure. Because this is what they do want. Some tried - their systems are patched as the others. Everyone today is a doctor, football coach an,d recently cybersecurty expert. Yes, maybe your IT team sucks, but others just want to do their fucking dmn, work.
    - 4y ago
- deleted 4y ago
  [deleted]
- digitallyfree 4y ago
  I think compromises could be made in this case - like giving users a reasonable amount of time to apply the patches before the update is performed forcibly. I have seen important meetings stalled because the presenter was only given two hours to reboot his computer after Windows updates were applied, for instance. Same for big Windows feature updates occuring in the middle of the day and stopping work for an hour. But with say twelve hours or so then the user could reboot in the evening or during lunch break without affecting productivity.
  - BrandoElFollito 4y ago
    It depends on the vulnerability. as I commented elsewhere, below Critical I do not care. Critical needs to be patched NOW, breaking things if needed. Including the important presentation.
- ianai 4y ago
  If you want Windows boxes to stay updated then have a way to replace the entire machine and replicate all needed application data (including sql server databases and anything proprietary) within minutes or hours at no notice. The Windows Update Service is a practical nightmare. It will get corrupted. It will require an entire re-image of the machine to fix. It will happen on production servers and client endpoints without regard to core business impact. It might even happen at a rate of a few per hundred of installs updated.

BrandoElFollito 4y ago

I work in IT security, so I am probably one of those people who screw other people over by pushing patches to be installed no matter what. Behind the scene, it is a never ending war between me who says "if your want your data to be safe, we need to patch", and the business that says "no, those is quarter rend (or stuff like this)'. So I say" on, sign this paper what you start that you do not want your data to be secure" And it goes on forever. This is exhausting for everyone, but IT is zero the very bottom of thefood chain, we cannot claim sales or new products and provide z utility like water or trash collection. So this is not always that someone is trying to screw you over, out is just that technology works the way it works, with the money that is invested in it.

hn_version_0023 4y ago
Your comment exemplifies the attitude I'm talking about. You've presented a scenario that's false in it's premise: either your data is safe or it isn't. But really, whether a thing is protected isn't quite so zero or one. Whether your production environment is actually at risk for the exploit being patched is a reasonable thing to ask in all scenarios. "What's our exposure here? Can it wait a few days?" would be a reasonable thing to ask. But in my experience that isn't how it goes. It goes how you suggested: ITSecOps presents an easy-to-decide case based on a false premise, and then immediately asks for CYA if the company says no. Over time, this becomes a rubber stamp from management, who are also demoralized by the process. The solution is to not present every security patch as a sky-is-falling event, but rather to be reasonable about the threats one faces and the impact to the business of forcing updates.
- pixl97 4y ago
  >Whether your production environment is actually at risk for the exploit being patched is a reasonable thing to ask in all scenarios. "What's our exposure here? Can it wait a few days?" would be a reasonable thing to ask. You would think it do be that way, but it don't.... Especially with things like Microsoft's megapatches they distribute these days. You are asking for the potential impact of anywhere from 10 to 100 patches custom tailored to the particular applications you're running on your server. Now you're asking for some large number of manhours of in depth analysis in a department that's already short on time. And as often as there is a patch that breaks the application, there is a patch that was released as a low impact that suddenly is high impact because it can be chained into an exploit, and now IT/SecOps is looking at the servers history to make sure it wasn't exploited at some time in the past. What is helpful here is not delaying patches, but instead having good testing environments that run the same version of the production application so the patch can be tested there first. Of course that costs a lot so you don't see it as often as you should.
- BrandoElFollito 4y ago
  > You've presented a scenario that's false in it's premise: either your data is safe or it isn't. Do you work in cybersecurity? have you done security assessment - the real, technicals ones before? I did. I coded (a tiny, tiny) part of the Linux kernel. My teams break systems for a living. If you have actually done security assessments than you know, of course, that there is no way to assess whether a vulnerability is low or high before shit hits the fan. When MS says "critical", then I say "critical" as well. I am not going to plat smart ass and then realize that in the shitty cathedral of systems businesses build there is THE configuration that allows for an exploit. Anything lower than critical - I do not care/ This can be done later. Bit is it is critical, it will be applied immediately, breaking systems and important presentations of the C-whatever. If the business has something against me then feel free to escalate to my management and tell them that they do want their data to be insecure. Because this is what they do want. Some tried - their systems are patched as the others. Everyone today is a doctor, football coach an,d recently cybersecurty expert. Yes, maybe your IT team sucks, but others just want to do their fucking dmn, work.
  - 4y ago
deleted 4y ago
[deleted]
digitallyfree 4y ago
I think compromises could be made in this case - like giving users a reasonable amount of time to apply the patches before the update is performed forcibly. I have seen important meetings stalled because the presenter was only given two hours to reboot his computer after Windows updates were applied, for instance. Same for big Windows feature updates occuring in the middle of the day and stopping work for an hour. But with say twelve hours or so then the user could reboot in the evening or during lunch break without affecting productivity.
- BrandoElFollito 4y ago
  It depends on the vulnerability. as I commented elsewhere, below Critical I do not care. Critical needs to be patched NOW, breaking things if needed. Including the important presentation.
ianai 4y ago
If you want Windows boxes to stay updated then have a way to replace the entire machine and replicate all needed application data (including sql server databases and anything proprietary) within minutes or hours at no notice. The Windows Update Service is a practical nightmare. It will get corrupted. It will require an entire re-image of the machine to fix. It will happen on production servers and client endpoints without regard to core business impact. It might even happen at a rate of a few per hundred of installs updated.