5 ms·

Well written story... also: > I have personally experienced network partions happening in two ways: all nodes in the cluster being updated at the same time thr

Well written story... also: > I have personally experienced network partions happening in two ways: all nodes in the cluster being updated at the same time through Windows update and firewall rules. I'd rather try to build a four story brick building with horse dung for masonry mortar, than run my backend on windows boxes. How many cumulative hours of horror has Windows Update unleashed on the human race?

logifail 4y ago
> I'd rather try to build a four story brick building with horse dung for masonry mortar, than run my backend on windows boxes. How many cumulative hours of horror has Windows Update unleashed on the human race? +1 War story: an ex-client of mine was on a different continent demoing a product (involving a physical device talking to a software backend hosted by a partner company) at a neutral site to a potential new customer. Months and months of talks and prep, dry run demos in the weeks before, everyone was quietly confident. Scope for selling lots and lots of these devices if the demo goes well, but the marketspace is fairly ruthless. Minutes before the team are due to present, a sudden and total comms loss between the device and the backend. Cue frantic calls which due to the time zones even involved getting the CEO of the partner company out of bed. Minutes ticking by.. Potential customer not impressed, said: you have ten minutes, after that we're leaving. As the potential customer was walking out of the building, the partner company finally tracked down the issue. Windows Update, on a backend database server.
- buran77 4y ago
  In all honesty you should rarely (if ever) patch or reboot all Prod servers in a single wave. No matter how much testing you think you've done in your development or pre-production environments something will still surprise you in prod and that costs more. Always and without exception, no matter what OS or software you use, make sure that after a patching cycle (or impactful change) you have a chunk of your environment untouched and still fully functioning. There will be situations where you can't, special cases where you're left with few options. If the system is that critical have 2 of everything (clusters, databases, etc.) in a way where the mirror can stay untouched and keep your business going. And if it needs saying, something like updates or reboots should never happen truly automatically (as in the software decided when or if to do it). A human should decide how to automate that and have full control over when and if it happens.
- justsomehnguy 4y ago
  I don't remember any server systems to reboot for updates if not configured to do so. And I've seen enough WinSvr in my life. Probably nobody consulted a sysadmin when they designed the system. *grin* Windows 10 behaviour is another story, but I would always stand on what it's a self-inflicted wound - people blamed MS for not updating soon enough... while disabling or postponing updates for literally years. MS acted. Poorly, yes, because by that time it ascended to General Motors levels of bureaucracy[0], but again - this is another story. [0] https://www.joelonsoftware.com/2006/06/16/my-first-billg-review/ https://www.joelonsoftware.com/2006/06/16/my-first-billg-rev...
  - jeroenhd 4y ago
    That's the thing I notice when I see people complaining about Windows choosing to update itself: I get the feeling people just don't want to reboot their machine for five minutes every week. For at least 80% of the comments I read, I'm pretty sure these people would never apply updates if they had the opportunity to. Sure, Windows update takes a while, but it's still so much better than whatever the hell Apple is doing. Despite the fact that I abhor removing the possibility to choose what your device does when, I think forced automatic updates are net win for computer users worldwide. In fact, I suspect Microsoft may need to be even more forceful, because even though consumer Windows rebooted itself at the time, wannacry and friends still managed to spread across the globe using an exploit that had been patched in an important security updates months before.
quantum_mcts 4y ago
Stopped reading when saw "Windows" to check the comments. Here it is.
hinkley 4y ago
I was part of a pretty large meetup for many years, and all through the dotcom boom and the following years the Windows Laptop behavior was utter trash. Any time a presenter showed up we had to fuck around for ten minutes to get their projector settings working, and for some reason I was the only one who could remember to ask them to turn off their screen saver. If I came late or someone else volunteered to help, we’d get halfway through their presentation, they would launch into a long anecdote or answer a question from the audience, the screen saver or sleep mode would kick in, and when they unlocked they would discover that Windows had completely forget the projector settings, and we’d have to stop for five minutes in the middle of a thought in order to fix it. There are so many other, better reasons to say it besides this or automatic updates, but Fuck Microsoft. Right in the ear.
radicalbyte 4y ago
It's configurable and particularly if you're running a production system you'll be wanting your own domain controller configured correctly to control Windows Update. If you're using the OS version which is designed to by used on Grandma's desktop as a server then you should expect a world of pain. Me? I try to avoid using Windows hosts because .Net runs great under Linux and you don't need to have 5 different Windows VMs running on your dev machine to have an accurate development environment.
jiggawatts 4y ago
What does this have to do with Windows? Mass-reboots or network outages happen for all sorts of reasons. Power outages happen. Network switches fail. Firewalls get misconfigured. Scripts get mis-scheduled. Windows Update is just one of many ways I've seen all-at-once reboots of servers. I don't see how this would be better on Linux. Do Linux admins never make mistakes in cron jobs? Is Linux magically immune to switch hardware failures? If RabbitMQ couldn't tolerate reboots, then that's on RabbitMQ, not Windows. PS: SQL Server AlwaysOn Availability Groups will happily restart after an all-servers simultaneous reboot. ZooKeeper won't. By "design", apparently. It converts a temporary outage into a permanent one... on purpose.
- chrisandchris 4y ago
  The major difference between the examples you mention and Windows Update is that your examples are _failures_ while Windows Update is considered a feature. I usually don't "apply router failures" while I do apply updates to Windows. When was the last time a switch just restarted to apply updates? i do not dislike Windows Update but I want to control it and apply things that are convenient to me.
  - jiggawatts 4y ago
    > When was the last time a switch just restarted to apply updates? When it's updated by a network team. It's a semi-regular occurrence in many workplaces. Windows Update is not forced upon Server editions. It can be scheduled at will by administrators. This complaint is the equivalent of saying that Linux is terrible because an admin scheduled "sudo apt-get update" to run on every node of a cluster at the same instant. Why would admins running updates concurrently be a failure of Windows, but not a failure of Linux? I seriously don't get this complaint, ESPECIALLY because the article makes it clear that: 1. The fault was in RabbitMQ, because I quote: "The default partition handling strategy is ignore which means to just enter the partitioned state and keep trucking along in this “split brain” mode thereby thrusting your cluster in to total chaos." [1] 2. Windows update wasn't inherently at fault: "The fix for Windows update was the ensure that nodes in the cluster are patched at different times." I bet the default partition configuration setting of RabbitMQ is also wrong when running on Linux, and the behaviour would be identically bad if an admin scheduled updates to run concurrently on all cluster nodes. [1] That's clown-shoes programming, which is why I'm never using RabbitMQ for anything ever. I've only heard bad things, like "the default settings result in corruption and data loss."
what-the-grump 4y ago
There is a thing called high availability, we do it in Linux, we do it in windows, we do it in microservices, PaaS deployments. Patching and rebooting all your nodes in a cluster at once is not highly available. But hey, let’s focus on windows hate instead.
- cies 4y ago
  > But hey, let’s focus on windows hate instead. I think GP acutely points out that one of the problems the author has with RabbitMQ is actually a problem with (a misconfiguration of) their underlying OS, MS Windows. If you've been bitten by this, I understand the hate completely.
- yonixw 4y ago
  You are correct, I woke up one day to 5 different servers in different env not responding. Across clouds (AWS+Azure). Total meltdown, are we hacked? Is the cloud down? Nope. Just ubuntu deploying docker update that had problem restarting the docker service again. Simple restart fixed all of them. But I guess people do have some bias, since on PC, windows updates are really predatory.
smileybarry 4y ago
If you use Windows Server and not just repurpose a client SKU, you get full control over which updates, schedule, distribution, restart times, etc. IIRC it even ships in "update manually" mode, you have to manually enable automatic updates. And if you set-up something like WSUS I think you can even set gradual update rollout with health checks.
- iasay 4y ago
  Sort of. Sometimes it goes wrong. I've seen a primary SQL Server go down because WUA took up enough CPU and interrupts to stop the cluster service health pings leading to a BSOD. This happened outside any schedule.
  - smileybarry 4y ago
    Guessing this was in the XP-2003 era if you mentioned WUA? Windows Update had a couple rewrites in Windows 7 & Windows 8.1 to solve the old "checking for updates takes too much CPU % if you have X+ updates installed". It's dramatically better now.
    - iasay 4y ago
      This was last year on windows server 2019. I say WUA. I don’t know what the current version of it is called. There are still problems.
      - smileybarry 4y ago
        Ah, I see. I just remember WUA from that one EXE people ran on Windows XP to manually check for updates.
KingOfCoders 4y ago
I need to add this story: Our Oracle database on Windows NT was fast then always slow (in the 90s, we could not afford an IBM/SUN for Oracle). When I went to the computer it was fast again. Back at my desk it got slow. Reason: Pipes software rendered OpenGL screen saver.
leephillips 4y ago
“Well written”—with a horrible grammatical error right in the title.

logifail 4y ago

> I'd rather try to build a four story brick building with horse dung for masonry mortar, than run my backend on windows boxes. How many cumulative hours of horror has Windows Update unleashed on the human race? +1 War story: an ex-client of mine was on a different continent demoing a product (involving a physical device talking to a software backend hosted by a partner company) at a neutral site to a potential new customer. Months and months of talks and prep, dry run demos in the weeks before, everyone was quietly confident. Scope for selling lots and lots of these devices if the demo goes well, but the marketspace is fairly ruthless. Minutes before the team are due to present, a sudden and total comms loss between the device and the backend. Cue frantic calls which due to the time zones even involved getting the CEO of the partner company out of bed. Minutes ticking by.. Potential customer not impressed, said: you have ten minutes, after that we're leaving. As the potential customer was walking out of the building, the partner company finally tracked down the issue. Windows Update, on a backend database server.

buran77 4y ago
In all honesty you should rarely (if ever) patch or reboot all Prod servers in a single wave. No matter how much testing you think you've done in your development or pre-production environments something will still surprise you in prod and that costs more. Always and without exception, no matter what OS or software you use, make sure that after a patching cycle (or impactful change) you have a chunk of your environment untouched and still fully functioning. There will be situations where you can't, special cases where you're left with few options. If the system is that critical have 2 of everything (clusters, databases, etc.) in a way where the mirror can stay untouched and keep your business going. And if it needs saying, something like updates or reboots should never happen truly automatically (as in the software decided when or if to do it). A human should decide how to automate that and have full control over when and if it happens.
justsomehnguy 4y ago
I don't remember any server systems to reboot for updates if not configured to do so. And I've seen enough WinSvr in my life. Probably nobody consulted a sysadmin when they designed the system. *grin* Windows 10 behaviour is another story, but I would always stand on what it's a self-inflicted wound - people blamed MS for not updating soon enough... while disabling or postponing updates for literally years. MS acted. Poorly, yes, because by that time it ascended to General Motors levels of bureaucracy[0], but again - this is another story. [0] https://www.joelonsoftware.com/2006/06/16/my-first-billg-review/ https://www.joelonsoftware.com/2006/06/16/my-first-billg-rev...
- jeroenhd 4y ago
  That's the thing I notice when I see people complaining about Windows choosing to update itself: I get the feeling people just don't want to reboot their machine for five minutes every week. For at least 80% of the comments I read, I'm pretty sure these people would never apply updates if they had the opportunity to. Sure, Windows update takes a while, but it's still so much better than whatever the hell Apple is doing. Despite the fact that I abhor removing the possibility to choose what your device does when, I think forced automatic updates are net win for computer users worldwide. In fact, I suspect Microsoft may need to be even more forceful, because even though consumer Windows rebooted itself at the time, wannacry and friends still managed to spread across the globe using an exploit that had been patched in an important security updates months before.

quantum_mcts 4y ago

Stopped reading when saw "Windows" to check the comments. Here it is.

hinkley 4y ago

I was part of a pretty large meetup for many years, and all through the dotcom boom and the following years the Windows Laptop behavior was utter trash. Any time a presenter showed up we had to fuck around for ten minutes to get their projector settings working, and for some reason I was the only one who could remember to ask them to turn off their screen saver. If I came late or someone else volunteered to help, we’d get halfway through their presentation, they would launch into a long anecdote or answer a question from the audience, the screen saver or sleep mode would kick in, and when they unlocked they would discover that Windows had completely forget the projector settings, and we’d have to stop for five minutes in the middle of a thought in order to fix it. There are so many other, better reasons to say it besides this or automatic updates, but Fuck Microsoft. Right in the ear.

radicalbyte 4y ago

It's configurable and particularly if you're running a production system you'll be wanting your own domain controller configured correctly to control Windows Update. If you're using the OS version which is designed to by used on Grandma's desktop as a server then you should expect a world of pain. Me? I try to avoid using Windows hosts because .Net runs great under Linux and you don't need to have 5 different Windows VMs running on your dev machine to have an accurate development environment.

jiggawatts 4y ago

What does this have to do with Windows? Mass-reboots or network outages happen for all sorts of reasons. Power outages happen. Network switches fail. Firewalls get misconfigured. Scripts get mis-scheduled. Windows Update is just one of many ways I've seen all-at-once reboots of servers. I don't see how this would be better on Linux. Do Linux admins never make mistakes in cron jobs? Is Linux magically immune to switch hardware failures? If RabbitMQ couldn't tolerate reboots, then that's on RabbitMQ, not Windows. PS: SQL Server AlwaysOn Availability Groups will happily restart after an all-servers simultaneous reboot. ZooKeeper won't. By "design", apparently. It converts a temporary outage into a permanent one... on purpose.

chrisandchris 4y ago
The major difference between the examples you mention and Windows Update is that your examples are _failures_ while Windows Update is considered a feature. I usually don't "apply router failures" while I do apply updates to Windows. When was the last time a switch just restarted to apply updates? i do not dislike Windows Update but I want to control it and apply things that are convenient to me.
- jiggawatts 4y ago
  > When was the last time a switch just restarted to apply updates? When it's updated by a network team. It's a semi-regular occurrence in many workplaces. Windows Update is not forced upon Server editions. It can be scheduled at will by administrators. This complaint is the equivalent of saying that Linux is terrible because an admin scheduled "sudo apt-get update" to run on every node of a cluster at the same instant. Why would admins running updates concurrently be a failure of Windows, but not a failure of Linux? I seriously don't get this complaint, ESPECIALLY because the article makes it clear that: 1. The fault was in RabbitMQ, because I quote: "The default partition handling strategy is ignore which means to just enter the partitioned state and keep trucking along in this “split brain” mode thereby thrusting your cluster in to total chaos." [1] 2. Windows update wasn't inherently at fault: "The fix for Windows update was the ensure that nodes in the cluster are patched at different times." I bet the default partition configuration setting of RabbitMQ is also wrong when running on Linux, and the behaviour would be identically bad if an admin scheduled updates to run concurrently on all cluster nodes. [1] That's clown-shoes programming, which is why I'm never using RabbitMQ for anything ever. I've only heard bad things, like "the default settings result in corruption and data loss."

what-the-grump 4y ago

There is a thing called high availability, we do it in Linux, we do it in windows, we do it in microservices, PaaS deployments. Patching and rebooting all your nodes in a cluster at once is not highly available. But hey, let’s focus on windows hate instead.

cies 4y ago
> But hey, let’s focus on windows hate instead. I think GP acutely points out that one of the problems the author has with RabbitMQ is actually a problem with (a misconfiguration of) their underlying OS, MS Windows. If you've been bitten by this, I understand the hate completely.
yonixw 4y ago
You are correct, I woke up one day to 5 different servers in different env not responding. Across clouds (AWS+Azure). Total meltdown, are we hacked? Is the cloud down? Nope. Just ubuntu deploying docker update that had problem restarting the docker service again. Simple restart fixed all of them. But I guess people do have some bias, since on PC, windows updates are really predatory.

smileybarry 4y ago

If you use Windows Server and not just repurpose a client SKU, you get full control over which updates, schedule, distribution, restart times, etc. IIRC it even ships in "update manually" mode, you have to manually enable automatic updates. And if you set-up something like WSUS I think you can even set gradual update rollout with health checks.

iasay 4y ago
Sort of. Sometimes it goes wrong. I've seen a primary SQL Server go down because WUA took up enough CPU and interrupts to stop the cluster service health pings leading to a BSOD. This happened outside any schedule.
- smileybarry 4y ago
  Guessing this was in the XP-2003 era if you mentioned WUA? Windows Update had a couple rewrites in Windows 7 & Windows 8.1 to solve the old "checking for updates takes too much CPU % if you have X+ updates installed". It's dramatically better now.
  - iasay 4y ago
    This was last year on windows server 2019. I say WUA. I don’t know what the current version of it is called. There are still problems.
    - smileybarry 4y ago
      Ah, I see. I just remember WUA from that one EXE people ran on Windows XP to manually check for updates.

KingOfCoders 4y ago

I need to add this story: Our Oracle database on Windows NT was fast then always slow (in the 90s, we could not afford an IBM/SUN for Oracle). When I went to the computer it was fast again. Back at my desk it got slow. Reason: Pipes software rendered OpenGL screen saver.

leephillips 4y ago

“Well written”—with a horrible grammatical error right in the title.