4 ms·

The big print giveth, and the small print taketh away. When the reality fails to live up to the marketing and hype, folks are naturally going to be upset. How

by lwhalen 12y ago

The big print giveth, and the small print taketh away. When the reality fails to live up to the marketing and hype, folks are naturally going to be upset. However, those of us who actually work in IT for a living expect downtime and architect for it. In a perfect world, Verizon's customer reaction should have been one of mild annoyance - "Oh well, guess we'll fail over to our secondary for that weekend". Anyone who drank the 'zero downtime' kool-aid will hopefully treat this as a (deserved) wake-up call that marketing does not excuse poor engineering on their part. Don't like the fact that your 'immortal' cloud provider goes down in planned (and sometimes unplanned) ways? Buy your own iron and do it yourself. It's the only way you can be completely 100% certain it's done right.

themartorana 12y ago
Two hours is sufferable. Two days is insane. Not everyone has the engineering talent or time to have auto-deploy multi-cloud failover. Not to mention that if you buy in to certain services, say, any AWS architecture beyond EC2, failover to another provider becomes a lot closer to impractical if not impossible. Two hours is sufferable...
- lwhalen 12y ago
  This is where good sysadmins, and teams thereof, earn their high salaries. You shouldn't have to suffer through a two-day outage, but it happens. A good sysadmin will insist on multi-provider hosting, be able to advise you on how not to get locked into $vendor's precious snowflake cloud implementation, and myriad other things.
  - falcolas 12y ago
    Yup. Hurricane Sandy should have been enough to remind folks of this. Natural (and unnatural) disasters happen, and you have to be prepared for this contingency. There are many tools out there which can help with this, but a competent sysadmin will help a hundredfold ensure that your business continues when the unthinkable happens. Heck, only today one of Amazon's new datacenters on the east coast had a 3 alarm fire[1]. Didn't end up having any instances fail, but we did notice some of our services have problems that coincided with the fire, and we were ready to fail over the moment instances started dropping. [1] http://money.cnn.com/2015/01/09/technology/amazon-data-center-fire/index.html http://money.cnn.com/2015/01/09/technology/amazon-data-cente...
- ademarre 12y ago
  > if you buy in to certain services, say, any AWS architecture beyond EC2, failover to another provider becomes a lot closer to impractical if not impossible. Yes. I say this all the time and continue to be amazed that this is such a foreign idea to many cloud users. When you stick to the infrastructure-as-a-service offerings, like EC2, and steer clear of cloud vendors' proprietary platform services, then you evade costly vendor lock-in. And this comes with very little added engineering cost. Granted, each organization will value vendor independence differently, but I suspect many organizations don't give enough consideration to worst-case scenarios.
ctdonath 12y ago
Don't confuse 'zero downtime' with TWO DAYS. Nobody expects true 'zero downtime', and a few minutes/hours down occurrences throughout the year adding up to a couple days may be tolerated, but... two straight days? no. Hey, let's try a little experiment on the subject. I'll sit here and do nothing starting right now, simulating unscheduled downtime, for the next two minutes. You sit there and do nothing starting right now, simulating unscheduled downtime, for the next two days. ETA: two days downtime divided by 365 days a year is over 15 minutes per day, or almost 2 hours per week ... any provider down that much & often would lose customers fast.
- duskwuff 12y ago
  Or look at it this way: if Verizon's service functions perfectly for the next year, they'll still only have 99.4% uptime over that period. That's pretty bad -- even most commodity web hosting providers do better than that.
  - lmm 12y ago
    The website I host on my home computer has better uptime than that.
snowwrestler 12y ago
The truth is that most websites and services actually can be down for two days and still survive. The culture of technology rightfully abhors downtime, and that's a good thing because it's our job to keep things up. But in reality there are very few companies who cannot survive 2 days of downtime. Sony Pictures is still in business, for example, despite an effective downtime of weeks.
- leeoniya 12y ago
  downtime costs money; frequently, a LOT of money. it's not a question of survival. severe downtime losses should be compensated, this is why SLAs exist. i can't put my team on involuntary furlough, citing shitty infrastructure choices.
- _delirium 12y ago
  Multi-day planned downtime for maintenance is even fairly common in Big Old Companies, especially for customer- and employee-facing systems. They just turn the thing off for a weekend, warn users in advance, and that's that. My bank's online banking seems to use that strategy. A few times a year, it tells me: sorry, but due to planned maintenance you'll have to check back Sunday evening. It's kind of surprising to me every time, but seems to be common. Some big companies do it with employee infrastructure like corporate email or network drives too. Some kind of system migration (or even physical server move) will mean on a random Saturday there's no email or file access. Announced in advance, sure, but it's still down for the day. Verizon telling you that some random servers are going to be down for two days admittedly might be worse than taking one of your own systems down for a maintenance window is, especially if you didn't expect in advance that such a thing was going to be a possibility, and made the mistake of putting really-can't-be-down stuff on Verizon's cloud.
  - tootie 12y ago
    Doing a rollout for a Fortune 100 company pretty soon and they built in an outage of a few hours. And it's literally because they won't spring for 1 extra server running apache even with 6 months lead time.
ef4 12y ago
> those of us who actually work in IT for a living expect downtime and architect for it. Does Verizon not work in IT for a living? Why does their architecture not allow them to switch over to a backup despite having many days of warning?
- lwhalen 12y ago
  I'm not saying Verizon is blameless here, but it's a matter of 'would you rather be right, or happy?' I personally will take 'happy' more often than not. You're absolutely right, Verizon shouldn't find a 48h downtime window acceptable. However, their customers who defined their business requirements to include a separate-vendor DR platform sure are (comparatively) happy right now.

by lwhalen 12y ago

themartorana 12y ago

Two hours is sufferable. Two days is insane. Not everyone has the engineering talent or time to have auto-deploy multi-cloud failover. Not to mention that if you buy in to certain services, say, any AWS architecture beyond EC2, failover to another provider becomes a lot closer to impractical if not impossible. Two hours is sufferable...

lwhalen 12y ago
This is where good sysadmins, and teams thereof, earn their high salaries. You shouldn't have to suffer through a two-day outage, but it happens. A good sysadmin will insist on multi-provider hosting, be able to advise you on how not to get locked into $vendor's precious snowflake cloud implementation, and myriad other things.
- falcolas 12y ago
  Yup. Hurricane Sandy should have been enough to remind folks of this. Natural (and unnatural) disasters happen, and you have to be prepared for this contingency. There are many tools out there which can help with this, but a competent sysadmin will help a hundredfold ensure that your business continues when the unthinkable happens. Heck, only today one of Amazon's new datacenters on the east coast had a 3 alarm fire[1]. Didn't end up having any instances fail, but we did notice some of our services have problems that coincided with the fire, and we were ready to fail over the moment instances started dropping. [1] http://money.cnn.com/2015/01/09/technology/amazon-data-center-fire/index.html http://money.cnn.com/2015/01/09/technology/amazon-data-cente...
ademarre 12y ago
> if you buy in to certain services, say, any AWS architecture beyond EC2, failover to another provider becomes a lot closer to impractical if not impossible. Yes. I say this all the time and continue to be amazed that this is such a foreign idea to many cloud users. When you stick to the infrastructure-as-a-service offerings, like EC2, and steer clear of cloud vendors' proprietary platform services, then you evade costly vendor lock-in. And this comes with very little added engineering cost. Granted, each organization will value vendor independence differently, but I suspect many organizations don't give enough consideration to worst-case scenarios.

ctdonath 12y ago

Don't confuse 'zero downtime' with TWO DAYS. Nobody expects true 'zero downtime', and a few minutes/hours down occurrences throughout the year adding up to a couple days may be tolerated, but... two straight days? no. Hey, let's try a little experiment on the subject. I'll sit here and do nothing starting right now, simulating unscheduled downtime, for the next two minutes. You sit there and do nothing starting right now, simulating unscheduled downtime, for the next two days. ETA: two days downtime divided by 365 days a year is over 15 minutes per day, or almost 2 hours per week ... any provider down that much & often would lose customers fast.

duskwuff 12y ago
Or look at it this way: if Verizon's service functions perfectly for the next year, they'll still only have 99.4% uptime over that period. That's pretty bad -- even most commodity web hosting providers do better than that.
- lmm 12y ago
  The website I host on my home computer has better uptime than that.

snowwrestler 12y ago

The truth is that most websites and services actually can be down for two days and still survive. The culture of technology rightfully abhors downtime, and that's a good thing because it's our job to keep things up. But in reality there are very few companies who cannot survive 2 days of downtime. Sony Pictures is still in business, for example, despite an effective downtime of weeks.

leeoniya 12y ago
downtime costs money; frequently, a LOT of money. it's not a question of survival. severe downtime losses should be compensated, this is why SLAs exist. i can't put my team on involuntary furlough, citing shitty infrastructure choices.
_delirium 12y ago
Multi-day planned downtime for maintenance is even fairly common in Big Old Companies, especially for customer- and employee-facing systems. They just turn the thing off for a weekend, warn users in advance, and that's that. My bank's online banking seems to use that strategy. A few times a year, it tells me: sorry, but due to planned maintenance you'll have to check back Sunday evening. It's kind of surprising to me every time, but seems to be common. Some big companies do it with employee infrastructure like corporate email or network drives too. Some kind of system migration (or even physical server move) will mean on a random Saturday there's no email or file access. Announced in advance, sure, but it's still down for the day. Verizon telling you that some random servers are going to be down for two days admittedly might be worse than taking one of your own systems down for a maintenance window is, especially if you didn't expect in advance that such a thing was going to be a possibility, and made the mistake of putting really-can't-be-down stuff on Verizon's cloud.
- tootie 12y ago
  Doing a rollout for a Fortune 100 company pretty soon and they built in an outage of a few hours. And it's literally because they won't spring for 1 extra server running apache even with 6 months lead time.

ef4 12y ago

> those of us who actually work in IT for a living expect downtime and architect for it. Does Verizon not work in IT for a living? Why does their architecture not allow them to switch over to a backup despite having many days of warning?

lwhalen 12y ago
I'm not saying Verizon is blameless here, but it's a matter of 'would you rather be right, or happy?' I personally will take 'happy' more often than not. You're absolutely right, Verizon shouldn't find a 48h downtime window acceptable. However, their customers who defined their business requirements to include a separate-vendor DR platform sure are (comparatively) happy right now.