5 ms·

First, determine your tolerance. How much does downtime per min cost you? How long can you be down? A lot of the time it may be cheaper to apologize to your

by mey 8y ago

First, determine your tolerance. How much does downtime per min cost you? How long can you be down? A lot of the time it may be cheaper to apologize to your customers than build a truly reliable system. Then start looking at points of failure and sort them based on severity and probability. Is your own software deployment going to generate more downtime per year than a regional aws outage? There are formal academic ways to determine what your overall availability is, but don't have those on hand. Suffice to say, it takes significant research, planning, execution, and testing to ensure a target availability. (See Netflix https://medium.com/netflix-techblog/the-netflix-simian-army-16e57fbab116 https://medium.com/netflix-techblog/the-netflix-simian-army-... ) if someone says they have 99.9% or better up time, they had better have proof in my mind (or a fat SLA violation payout) People outsource to cloud providers not because they are cheap, but because managing infra in house is hard. Also move fast and break things. Read AWS docs about availability, there are availability zones in a region, spread across those to minimize impact. Then test when something goes down. Fix/repeat.

gaius 8y ago
If all your competitors are down too, the equation changes, no?
- arachnophobe 8y ago
  Yep - If all my competitors are down the equation changes such that I _definitely_ want to be up. Any trade comes to me if it's urgent, and I appear more professional as I've got a functioning system. I might be an chancer running my entire system off an shoestring but being up when everyone else has taken a dive looks good.
devonkim 8y ago
People outsource to cloud providers because building / hiring / maintaining a team of decent engineers that provide a baseline industry bar of SLAs, SLOs is much more expensive than the eye watering costs of most cloud providers at even a IaaS level. Opex is tough. Most companies I’ve been at don’t offer multi region support for their services because it’s too expensive for the service provided even in so-called “price insensitive” enterprises (you can’t just make up a price that’s huge, they do have budgets still) and most of their customers are unwilling / unable to pay more for the extra availability. If your software is designed from the start better, multi region failovers should be fairly inexpensive though. But all the bolted on “multi region” software I’ve seen has been hideously expensive and oftentimes less reliable due to the design being soundly not able to tolerate failures well.
- user5994461 8y ago
  The baseline is that it takes 12 dedicated people across the world to run a 24/7 support operation. Considering that even tech companies hardly manage to have a pair of DevOps or Sysadmin, running one own infrastructure is completely out of question.
  - scarface74 8y ago
    Most small companies on AWS with revenue outsource support to an MSP.
    - devonkim 8y ago
      Any stats / surveys on this? Most smaller (< 50 employees) b2c enterprise-style SaaS companies I'm anecdotally familiar with barely have enough funds to hire ops engineers let alone outsource to MSPs (even if the MSP practices labor rate arbitrage). A lot of the criteria I'd wager may be conditions around pre-revenue status and funding conditions moreso than headcount. I'm trying to understand just how biased my own experiences and indirect experiences are with the wider US market and appreciate data.
      - user5994461 8y ago
        Small companies put every developer on call and here comes the support coverage. Of course, that doesn't make them knowledgeable to run stable infrastructure and they will move on as soon as they realize they are being abused to work overnight and week end.
        scarface74 8y ago
        I am developer and I have the knowledge to run stable infrastructure at least on a small company SAAS scale. But, I wouldn’t go near a company that expected developers to be on call for netops work. A company that doesn’t want the overhead of an MSP which in my experience is less than the cost of a full time Dev is not a company I’m going to work for. It would tell me a lot about thier mentality.

by mey 8y ago

gaius 8y ago

If all your competitors are down too, the equation changes, no?

arachnophobe 8y ago
Yep - If all my competitors are down the equation changes such that I _definitely_ want to be up. Any trade comes to me if it's urgent, and I appear more professional as I've got a functioning system. I might be an chancer running my entire system off an shoestring but being up when everyone else has taken a dive looks good.

devonkim 8y ago

People outsource to cloud providers because building / hiring / maintaining a team of decent engineers that provide a baseline industry bar of SLAs, SLOs is much more expensive than the eye watering costs of most cloud providers at even a IaaS level. Opex is tough. Most companies I’ve been at don’t offer multi region support for their services because it’s too expensive for the service provided even in so-called “price insensitive” enterprises (you can’t just make up a price that’s huge, they do have budgets still) and most of their customers are unwilling / unable to pay more for the extra availability. If your software is designed from the start better, multi region failovers should be fairly inexpensive though. But all the bolted on “multi region” software I’ve seen has been hideously expensive and oftentimes less reliable due to the design being soundly not able to tolerate failures well.

user5994461 8y ago
The baseline is that it takes 12 dedicated people across the world to run a 24/7 support operation. Considering that even tech companies hardly manage to have a pair of DevOps or Sysadmin, running one own infrastructure is completely out of question.
- scarface74 8y ago
  Most small companies on AWS with revenue outsource support to an MSP.
  - devonkim 8y ago
    Any stats / surveys on this? Most smaller (< 50 employees) b2c enterprise-style SaaS companies I'm anecdotally familiar with barely have enough funds to hire ops engineers let alone outsource to MSPs (even if the MSP practices labor rate arbitrage). A lot of the criteria I'd wager may be conditions around pre-revenue status and funding conditions moreso than headcount. I'm trying to understand just how biased my own experiences and indirect experiences are with the wider US market and appreciate data.
    - user5994461 8y ago
      Small companies put every developer on call and here comes the support coverage. Of course, that doesn't make them knowledgeable to run stable infrastructure and they will move on as soon as they realize they are being abused to work overnight and week end.
      - scarface74 8y ago
        I am developer and I have the knowledge to run stable infrastructure at least on a small company SAAS scale. But, I wouldn’t go near a company that expected developers to be on call for netops work. A company that doesn’t want the overhead of an MSP which in my experience is less than the cost of a full time Dev is not a company I’m going to work for. It would tell me a lot about thier mentality.