6 ms·

The zero-downtime culture is pretty out of control. Your flight can be delayed for hours, your roads can be closed for weeks, your internet can be out for hours

by vlunkr 2y ago

The zero-downtime culture is pretty out of control. Your flight can be delayed for hours, your roads can be closed for weeks, your internet can be out for hours because someone cut a line, that's all ok, but someone can't get into your website for 3 minutes to check the status of their order and we all must lose our minds and publicly apologize and explain in detail what happened and why it will never happen again.

blastro 2y ago
Every time the stock market shits itself, the large brokerages seem to "go down"
Alupis 2y ago
The obvious difference is no matter how long the county or city takes to reopen that road - you'll go right back to using it because you don't have an alternate choice. For a website - particularly ecommerce websites - you have many choices. Rarely is a product only sold on a particular website. Being down when someone is trying to place an order can and does result in losing the sale, and potentially the customer forever. Customer loyalty is often fickle - why would they wait around for you to fix your stuff when they can simply place their order on Amazon or another retail website?
- vasco 2y ago
  You are of course right, but in practice I've seen many more incidents created by doing changes to make things more robust than from simpler incremental product changes that usually are feature flagged and so on. At deeper levels usually there's less ability to do such containment (or is too expensive or takes too long or people are lazy) and so many times I wonder if it's better to do the trade-off or just keep things simple and eat only the "simple" sources of downtime to fix. For example the classic thing is to always have minimum of 3 or 5 nodes for every stateful system. But in some companies, 1 hour of planned downtime on Monday mornings at 7AM to 8AM for operations and upgrades (which you only use when you need) + eating the times when the machine actually dies, would be less downtime than all the times you'd go down because of problems related to the very thing that should make you more robust. An incident here because replication lag was too high, an incident there because the quorum keeping system ran out of space etc and you're probably already behind. And then we have kubernetes. At some point it does make sense, when you have enough people and complexity to deal with this properly, but usually we do it too early.
- vlunkr 2y ago
  That really depends on what type of service we're talking about. Yeah, if you're doing really high volume sales on products that can be purchased elsewhere for close to the same price and shipping options, then uptime is critical. That's not the situation most of us are in.
  - Alupis 2y ago
    > That's not the situation most of us are in. From an ecommerce website's perspective - it is exactly the situation most of us are in. Unless you are the manufacturer for your own goods, then you are competing on often-thin margins within a crowded industry with lots of competition where everyone more-or-less offers the same price. In an age where more and more people don't even consider shopping anywhere but Amazon, being down exactly when that customer decided to grace your website is akin to giving yourself a nice, stinging papercut. Do this often enough and long enough, those papercuts will start to leave scars. So ya, being down is unacceptable for most ecommerce websites. The website literally generates your revenue, and without it you receive no revenue.
    - lovethevoid 2y ago
      >Unless you are the manufacturer for your own goods, then you are competing on often-thin margins within a crowded industry with lots of competition where everyone more-or-less offers the same price. That's not true. You don't have to manufacture your own goods to not compete with others at a commodity level. It's quite literally why branding is so important. But I do find it funny you wound up doing exactly what the above commenter stated, and are acting like this is a world ending event for a small-medium sized ecommerce site to be down for a while.
      - Alupis 2y ago
        > It's quite literally why branding is so important. Most of the goods you purchase are from brands not owned by the retail website. Pick a hobby and consider the prominent brands, then consider the prominent websites people in that hobby often patronize - they are very often not the same. > and are acting like this is a world ending event for a small-medium sized ecommerce site to be down for a while Because it kind of is. The smaller you are, the more significant of an impact an outage has. Nobody is going to think "this website I've never heard of totally isn't working right now, so I'll just wait around until they fix it". No, instead, they will go on to their next google search result - or Amazon. This hurts even worse if they clicked one of your ads while your website is down... costing you actual money and not just the sale opportunity. A customer that might have had an expected lifetime of 1-3+ orders now becomes 0. Websites don't have "business hours", and routine maintenance is not acceptable for customers that shop when it's convenient for themselves. Remember, ecommerce exists as a better way to do over-the-phone and catalog/mail ordering. One of those benefits is you can order whenever you want - so it is in fact important for the website to be online at close to 100% as possible.
- throwaway22032 2y ago
  Sure. If you have a couple of short outages in a year and say 2% of your customers view the website during that time, half quit and never come back over it, that's it. You've lost 1% of your customers. You seem to be stating that any loss at all no matter how small is a complete disaster without really qualifying it. Yet one could easily make the point - well, what about the customers you don't serve at all because they want to buy a rucksack and you don't sell them? That number could be far higher than the ones affected by outages. I think this probably just depends on how big the numbers are to be honest. Yes, if 1% of people are millions of dollars, then sure, we care. Usually it's not anything close to being like that.
  - fsociety 2y ago
    With the right analysis I can show how your outage increased your business value.
photonthug 2y ago
Great point. I honestly can't remember the last time that I've had a smooth experience with any kind of business or commercial thing, even when I'm the one paying for services. Bank declines transactions randomly when balance is sufficient, on vendors that have been used before. Orders of equipment over $10k are days late, no explanation provided, no number to speak to a human, just ineffective support robots. Planning days or weeks around other people's problems is pretty normal, but really, that just means that I get to postpone going to the DMV and getting the run-around from them. Not sure if it's quiet-quitting or what, but the last few years (basically since covid) my consumer experiences in the US are feeling a lot more.. shall we say, European? And I guess that's fine, pretty much everything really can wait. But it's hard to maintain one's own sense of professional urgency to keep things smooth for others when literally nothing is smooth for you.
- janalsncm 2y ago
  > Orders of equipment over $10k are days late, no explanation provided, no number to speak to a human, just ineffective support robots. I would be furious. If I spend $10k I want to be speaking with a human if something goes wrong. I feel like in the US we have given up on decent customer experiences.
  - mensetmanusman 2y ago
    This is what labor shortage feels like.
- ryandrake 2y ago
  Every company seems to just be phoning it in lately. The number of times I've ordered something from an online retailer, and either 1. the wrong thing arrived, 2. the thing arrived broken/doa, or 3. the thing never arrived, has at least tripled since say 5 years ago. And to a lot of companies' credit, their indifferent but friendly customer support tends to just say "<yawn> just keep whatever we sent you and we'll refund you. You can try to order the thing again, and we'll maybe send you the right, working thing next time. Or maybe not. Who knows?"
mlyle 2y ago
You know, I think if there was a little more tendency to explain, and trust in those explanations, people would like airlines, local transportation departments, etc, a whole lot more. Without information, it is easy to assume incompetence, or worse, just not caring.
mcguire 2y ago
Are you assuming airline employees aren't scrambling when a flight is delayed? Or that road crews don't get called in the middle of the night? A minor power outage gets a crew called no matter what time it is; a major outage will bring in crews from different states, much less power companies.
- 91bananas 2y ago
  But none of them are working in panic-mode as them doing this work is just an expected thing. As OP has pointed out, downtime in other industries is expected and agreed upon as OK. For some reason because it's on a computer that does not apply to our industry, we are to go in to panic-mode to fix, and write about how it will never happen again, stock prices fall, people lose their jobs.
  - sillyfluke 2y ago
    In the case of flights, isn't it rare that it is a single employee or just a couple of employees that are tasked or held responsible for the flight taking off on time? Funny story though, veering a little OT, but recently I had to haul ass from one end of the airport to the other due to arriving on a delayed flight with a connecting flight with a different airline. As I was about to enter the airplane, after getting my boarding pass scanned, completely drenched in sweat, the flight attendant saw me and said the captain wanted to see me and ushered me into the cockpit. I found myself in the cockpit, where the captain proceeded to talk to me in the local foreign language while I was thinking, "Wtf, isn't it illegal for me to be in here after 9-11?" I then started repeating, in the local language, that I didn't understand them. But that didn't have the desired effect intially and they kept talking. I repeated myself both in their language and English until the captain finally said, in English, "Are you a passenger?" When I said yes, he told me I could sit down. When I sat down, the only rational explanation that I could come up with was that despite being in civilian clothes I was mistaken for a pilot. Not just any pilot, but one of the pilots that was supposed to be in the cockpit. I could only conjecture that he was a no-show and that they had to bring in a substitute to replace him which caused the plane to be delayed and allowed me to make the flight. Any commercials pilots are free to correct my -- which normally should be a very silly -- theory. If it could possibly be true though, being the no-show pilot in that situation might be a little stressful.
groby_b 2y ago
The world would be a better place when airlines and ISPs worked that way too. (Roads work is usually communicated well in advance) Downtime for these is critically impacting people, and you're damn right you should explain what went wrong and how you'll avoid this in the future. Because the only reasonable alternative is nationalizing them. "I dun fucked up, but meh" is not an acceptable attitude.
- vlunkr 2y ago
  > "I dun fucked up, but meh" is not an acceptable attitude. I'm cool with it. I don't care that some hard drive went bad and the monitoring system didn't pick up on it because some queue was full. Who cares? If this isn't happening constantly, I assume they're going to learn their lesson.
  - groby_b 2y ago
    > "I assume they're going to learn their lesson." That's pretty much the opposite of "but meh". Absolutely, mistakes happen, they aren't the end of the world, but they need to trigger a learning process. Looking at airlines and ISPs, that process is, in the most charitable viewing, processing glacially.

by vlunkr 2y ago

blastro 2y ago

Every time the stock market shits itself, the large brokerages seem to "go down"

Alupis 2y ago

The obvious difference is no matter how long the county or city takes to reopen that road - you'll go right back to using it because you don't have an alternate choice. For a website - particularly ecommerce websites - you have many choices. Rarely is a product only sold on a particular website. Being down when someone is trying to place an order can and does result in losing the sale, and potentially the customer forever. Customer loyalty is often fickle - why would they wait around for you to fix your stuff when they can simply place their order on Amazon or another retail website?

vasco 2y ago
You are of course right, but in practice I've seen many more incidents created by doing changes to make things more robust than from simpler incremental product changes that usually are feature flagged and so on. At deeper levels usually there's less ability to do such containment (or is too expensive or takes too long or people are lazy) and so many times I wonder if it's better to do the trade-off or just keep things simple and eat only the "simple" sources of downtime to fix. For example the classic thing is to always have minimum of 3 or 5 nodes for every stateful system. But in some companies, 1 hour of planned downtime on Monday mornings at 7AM to 8AM for operations and upgrades (which you only use when you need) + eating the times when the machine actually dies, would be less downtime than all the times you'd go down because of problems related to the very thing that should make you more robust. An incident here because replication lag was too high, an incident there because the quorum keeping system ran out of space etc and you're probably already behind. And then we have kubernetes. At some point it does make sense, when you have enough people and complexity to deal with this properly, but usually we do it too early.
vlunkr 2y ago
That really depends on what type of service we're talking about. Yeah, if you're doing really high volume sales on products that can be purchased elsewhere for close to the same price and shipping options, then uptime is critical. That's not the situation most of us are in.
- Alupis 2y ago
  > That's not the situation most of us are in. From an ecommerce website's perspective - it is exactly the situation most of us are in. Unless you are the manufacturer for your own goods, then you are competing on often-thin margins within a crowded industry with lots of competition where everyone more-or-less offers the same price. In an age where more and more people don't even consider shopping anywhere but Amazon, being down exactly when that customer decided to grace your website is akin to giving yourself a nice, stinging papercut. Do this often enough and long enough, those papercuts will start to leave scars. So ya, being down is unacceptable for most ecommerce websites. The website literally generates your revenue, and without it you receive no revenue.
  - lovethevoid 2y ago
    >Unless you are the manufacturer for your own goods, then you are competing on often-thin margins within a crowded industry with lots of competition where everyone more-or-less offers the same price. That's not true. You don't have to manufacture your own goods to not compete with others at a commodity level. It's quite literally why branding is so important. But I do find it funny you wound up doing exactly what the above commenter stated, and are acting like this is a world ending event for a small-medium sized ecommerce site to be down for a while.
    - Alupis 2y ago
      > It's quite literally why branding is so important. Most of the goods you purchase are from brands not owned by the retail website. Pick a hobby and consider the prominent brands, then consider the prominent websites people in that hobby often patronize - they are very often not the same. > and are acting like this is a world ending event for a small-medium sized ecommerce site to be down for a while Because it kind of is. The smaller you are, the more significant of an impact an outage has. Nobody is going to think "this website I've never heard of totally isn't working right now, so I'll just wait around until they fix it". No, instead, they will go on to their next google search result - or Amazon. This hurts even worse if they clicked one of your ads while your website is down... costing you actual money and not just the sale opportunity. A customer that might have had an expected lifetime of 1-3+ orders now becomes 0. Websites don't have "business hours", and routine maintenance is not acceptable for customers that shop when it's convenient for themselves. Remember, ecommerce exists as a better way to do over-the-phone and catalog/mail ordering. One of those benefits is you can order whenever you want - so it is in fact important for the website to be online at close to 100% as possible.
throwaway22032 2y ago
Sure. If you have a couple of short outages in a year and say 2% of your customers view the website during that time, half quit and never come back over it, that's it. You've lost 1% of your customers. You seem to be stating that any loss at all no matter how small is a complete disaster without really qualifying it. Yet one could easily make the point - well, what about the customers you don't serve at all because they want to buy a rucksack and you don't sell them? That number could be far higher than the ones affected by outages. I think this probably just depends on how big the numbers are to be honest. Yes, if 1% of people are millions of dollars, then sure, we care. Usually it's not anything close to being like that.
- fsociety 2y ago
  With the right analysis I can show how your outage increased your business value.

photonthug 2y ago

Great point. I honestly can't remember the last time that I've had a smooth experience with any kind of business or commercial thing, even when I'm the one paying for services. Bank declines transactions randomly when balance is sufficient, on vendors that have been used before. Orders of equipment over $10k are days late, no explanation provided, no number to speak to a human, just ineffective support robots. Planning days or weeks around other people's problems is pretty normal, but really, that just means that I get to postpone going to the DMV and getting the run-around from them. Not sure if it's quiet-quitting or what, but the last few years (basically since covid) my consumer experiences in the US are feeling a lot more.. shall we say, European? And I guess that's fine, pretty much everything really can wait. But it's hard to maintain one's own sense of professional urgency to keep things smooth for others when literally nothing is smooth for you.

janalsncm 2y ago
> Orders of equipment over $10k are days late, no explanation provided, no number to speak to a human, just ineffective support robots. I would be furious. If I spend $10k I want to be speaking with a human if something goes wrong. I feel like in the US we have given up on decent customer experiences.
- mensetmanusman 2y ago
  This is what labor shortage feels like.
ryandrake 2y ago
Every company seems to just be phoning it in lately. The number of times I've ordered something from an online retailer, and either 1. the wrong thing arrived, 2. the thing arrived broken/doa, or 3. the thing never arrived, has at least tripled since say 5 years ago. And to a lot of companies' credit, their indifferent but friendly customer support tends to just say "<yawn> just keep whatever we sent you and we'll refund you. You can try to order the thing again, and we'll maybe send you the right, working thing next time. Or maybe not. Who knows?"

mlyle 2y ago

You know, I think if there was a little more tendency to explain, and trust in those explanations, people would like airlines, local transportation departments, etc, a whole lot more. Without information, it is easy to assume incompetence, or worse, just not caring.

mcguire 2y ago

Are you assuming airline employees aren't scrambling when a flight is delayed? Or that road crews don't get called in the middle of the night? A minor power outage gets a crew called no matter what time it is; a major outage will bring in crews from different states, much less power companies.

91bananas 2y ago
But none of them are working in panic-mode as them doing this work is just an expected thing. As OP has pointed out, downtime in other industries is expected and agreed upon as OK. For some reason because it's on a computer that does not apply to our industry, we are to go in to panic-mode to fix, and write about how it will never happen again, stock prices fall, people lose their jobs.
- sillyfluke 2y ago
  In the case of flights, isn't it rare that it is a single employee or just a couple of employees that are tasked or held responsible for the flight taking off on time? Funny story though, veering a little OT, but recently I had to haul ass from one end of the airport to the other due to arriving on a delayed flight with a connecting flight with a different airline. As I was about to enter the airplane, after getting my boarding pass scanned, completely drenched in sweat, the flight attendant saw me and said the captain wanted to see me and ushered me into the cockpit. I found myself in the cockpit, where the captain proceeded to talk to me in the local foreign language while I was thinking, "Wtf, isn't it illegal for me to be in here after 9-11?" I then started repeating, in the local language, that I didn't understand them. But that didn't have the desired effect intially and they kept talking. I repeated myself both in their language and English until the captain finally said, in English, "Are you a passenger?" When I said yes, he told me I could sit down. When I sat down, the only rational explanation that I could come up with was that despite being in civilian clothes I was mistaken for a pilot. Not just any pilot, but one of the pilots that was supposed to be in the cockpit. I could only conjecture that he was a no-show and that they had to bring in a substitute to replace him which caused the plane to be delayed and allowed me to make the flight. Any commercials pilots are free to correct my -- which normally should be a very silly -- theory. If it could possibly be true though, being the no-show pilot in that situation might be a little stressful.

groby_b 2y ago

The world would be a better place when airlines and ISPs worked that way too. (Roads work is usually communicated well in advance) Downtime for these is critically impacting people, and you're damn right you should explain what went wrong and how you'll avoid this in the future. Because the only reasonable alternative is nationalizing them. "I dun fucked up, but meh" is not an acceptable attitude.

vlunkr 2y ago
> "I dun fucked up, but meh" is not an acceptable attitude. I'm cool with it. I don't care that some hard drive went bad and the monitoring system didn't pick up on it because some queue was full. Who cares? If this isn't happening constantly, I assume they're going to learn their lesson.
- groby_b 2y ago
  > "I assume they're going to learn their lesson." That's pretty much the opposite of "but meh". Absolutely, mistakes happen, they aren't the end of the world, but they need to trigger a learning process. Looking at airlines and ISPs, that process is, in the most charitable viewing, processing glacially.