4 ms·
> "50k windows-based endpoints or so. All down." I'm a dev rather than infra guy, but I'm pretty sure everywhere I've worked which has a large server estate ha
by m-i-l 2y ago
> "50k windows-based endpoints or so. All down."
I'm a dev rather than infra guy, but I'm pretty sure everywhere I've worked which has a large server estate has always done rolling patch updates, i.e. over multiple days (if critical) or multiple weekends (if routine), not blast every single machine everywhere all at once.
- EdwardDiego 2y agoThis feels like an auto-update functionality. For something that's running in kernel space (presumably, if it can BSOD you?) Which is fucking terrifying.
- fredoralive 2y agoIf this comment tree: https://news.ycombinator.com/item?id=41003390 https://news.ycombinator.com/item?id=41003390 is correct, someone at Crowdstrike looked at their documented update staging process, slammed their beer down, and said: "Fuck it, let's test it in production", and just pushed it to everyone.
- averageRoyalty 2y agoEven if true, the orgs whose machines they are have the responsibility to validate patches.
- sofixa 2y agoThis is not a patch per se, it was Crowdstrike updating their virus definition or whatever it's called internal database. Such things are usually enabled by default to auto-update, because otherwise you lose a big part of the interest (if there's any) of running an antivirus.
- fredoralive 2y agoSurely their should be at least some staging on update files as well, to avoid the "oops, we accidentally blacklisted explorer.exe" type things (or, indeed, this)?
- saberdancer 2y agoCompanies have staging and test process but CS bypassed it and deployed to prod.
- minetest2048 2y agoIf I understand the thread correctly, CS bypassed the organization's staging system
- CoastalCoder 2y agoI'm guessing there's a lesson to be learned here.
- perlgeek 2y agoWhich of course begs the question: How were they able to do that? Was there no internal review? What about automated processes? For an organization it's always the easiest, most convenient answer to blame a single scapegoat, maybe fire them... but if a single bad decision or error from an employee has this kind of impact, there's always a lack of safety nets.
- heraldgeezer 2y agoCrowdstrike auto-updates. Please do not spread misinfo.