4 ms·

People outsource to cloud providers because building / hiring / maintaining a team of decent engineers that provide a baseline industry bar of SLAs, SLOs is muc

People outsource to cloud providers because building / hiring / maintaining a team of decent engineers that provide a baseline industry bar of SLAs, SLOs is much more expensive than the eye watering costs of most cloud providers at even a IaaS level. Opex is tough. Most companies I’ve been at don’t offer multi region support for their services because it’s too expensive for the service provided even in so-called “price insensitive” enterprises (you can’t just make up a price that’s huge, they do have budgets still) and most of their customers are unwilling / unable to pay more for the extra availability. If your software is designed from the start better, multi region failovers should be fairly inexpensive though. But all the bolted on “multi region” software I’ve seen has been hideously expensive and oftentimes less reliable due to the design being soundly not able to tolerate failures well.

user5994461 8y ago
The baseline is that it takes 12 dedicated people across the world to run a 24/7 support operation. Considering that even tech companies hardly manage to have a pair of DevOps or Sysadmin, running one own infrastructure is completely out of question.
- scarface74 8y ago
  Most small companies on AWS with revenue outsource support to an MSP.
  - devonkim 8y ago
    Any stats / surveys on this? Most smaller (< 50 employees) b2c enterprise-style SaaS companies I'm anecdotally familiar with barely have enough funds to hire ops engineers let alone outsource to MSPs (even if the MSP practices labor rate arbitrage). A lot of the criteria I'd wager may be conditions around pre-revenue status and funding conditions moreso than headcount. I'm trying to understand just how biased my own experiences and indirect experiences are with the wider US market and appreciate data.
    - user5994461 8y ago
      Small companies put every developer on call and here comes the support coverage. Of course, that doesn't make them knowledgeable to run stable infrastructure and they will move on as soon as they realize they are being abused to work overnight and week end.
      - scarface74 8y ago
        I am developer and I have the knowledge to run stable infrastructure at least on a small company SAAS scale. But, I wouldn’t go near a company that expected developers to be on call for netops work. A company that doesn’t want the overhead of an MSP which in my experience is less than the cost of a full time Dev is not a company I’m going to work for. It would tell me a lot about thier mentality.
    - scarface74 8y ago
      I don’t know about pre revenue companies, but, if you have to be secure and compliant with regulations from day one, you’re not going to take a chance with having a bunch of devs setting up your infrastructure. My bias probably comes from working mostly with companies in highly regulated fields. Besides, I’m assuming that the cost savings a small company can get from being billed under a much larger organization account would make up for it. That and having cheap shared netops support.
- Marsymars 8y ago
  What's the math/logic to get to 12 people?
  - Mtinie 8y ago
    Two people per 6-hour timezone chunk, with one overlapping person splitting each chunk? 2 at UTC 0-8 1 at UTC 4-12 2 at UTC 8-16 1 at UTC 12-20 2 at UTC 16-24 1 at UTC 20-04 (Repeat)
  - wstrange 8y ago
    Timezone coverage, vacations, training, sick days, etc.
scarface74 8y ago
That does bring up an interesting point. In hindsight, we already have duplicate infrastructure - a dev account and a production account. Why in the world was it decided to put both accounts in the same region? The separate account was setup partially on my insistence but it was set up in the same region. If needed, we could have done VPC peerings across regions. (https://aws.amazon.com/about-aws/whats-new/2017/11/announcing-support-for-inter-region-vpc-peering/ https://aws.amazon.com/about-aws/whats-new/2017/11/announcin...)
- dodobirdlord 8y ago
  There are some significant differences region to region in AWS. Different numbers of availability zones, different latencies based on datacenter location, different types and availabilities of EC2 instances, etc. I think it makes sense to develop in the same region that your production service runs in just so you don't shoot yourself in the foot by deploying something that runs fine in your development region but doesn't run as well in your production region.
- devonkim 8y ago
  In most of the cases I've seen so far with multiple regions / POP (4 different companies of different sizes and verticals) the software never being deployed in another reason is the fundamental reason. Reasons ranged from "we hard-coded us-east-1 everywhere and don't know how to test region independence it turns out" to "we have 3 weeks to try to make our software distributed between different regions but have nowhere near the resources to test it well let alone develop anything new for it." Automation is completely remedial and non-repeatable in most of these companies that rarely provision clean environments from scratch. By keeping regions the same between different environments, you reduce the number of possible differences. Some services in AWS are also not available in others (I'm quite familiar with AWS Data Pipeline not being available outside the "core" regions like us-east-1, eu-west-1) and having services in one region make usage of resources in another region is a huge change when most developers outside ones with technology literate customers are under the gun to push features out fast over sound design. The matrix of services and configurations necessary to mix and match regions and availability zones is non-trivial if you make extensive usage of AWS services above the IAAS layer. Also, cross-region VPC peering has a TON of limitations that rather annoying depending upon how well your network has been architected (by default in most companies outside enterprises with a deep bench of network engineers, this would be rated at "complete crap barely better than a typical home wifi network"). Heck, even though I'm non-dumb at networks I have to keep reminding myself of various cross-region VPC limitations when working with refactoring cross-region VPCs like where you can reference security groups, how to propagate Route 53 records, etc.

user5994461 8y ago

The baseline is that it takes 12 dedicated people across the world to run a 24/7 support operation. Considering that even tech companies hardly manage to have a pair of DevOps or Sysadmin, running one own infrastructure is completely out of question.

scarface74 8y ago
Most small companies on AWS with revenue outsource support to an MSP.
- devonkim 8y ago
  Any stats / surveys on this? Most smaller (< 50 employees) b2c enterprise-style SaaS companies I'm anecdotally familiar with barely have enough funds to hire ops engineers let alone outsource to MSPs (even if the MSP practices labor rate arbitrage). A lot of the criteria I'd wager may be conditions around pre-revenue status and funding conditions moreso than headcount. I'm trying to understand just how biased my own experiences and indirect experiences are with the wider US market and appreciate data.
  - user5994461 8y ago
    Small companies put every developer on call and here comes the support coverage. Of course, that doesn't make them knowledgeable to run stable infrastructure and they will move on as soon as they realize they are being abused to work overnight and week end.
    - scarface74 8y ago
      I am developer and I have the knowledge to run stable infrastructure at least on a small company SAAS scale. But, I wouldn’t go near a company that expected developers to be on call for netops work. A company that doesn’t want the overhead of an MSP which in my experience is less than the cost of a full time Dev is not a company I’m going to work for. It would tell me a lot about thier mentality.
  - scarface74 8y ago
    I don’t know about pre revenue companies, but, if you have to be secure and compliant with regulations from day one, you’re not going to take a chance with having a bunch of devs setting up your infrastructure. My bias probably comes from working mostly with companies in highly regulated fields. Besides, I’m assuming that the cost savings a small company can get from being billed under a much larger organization account would make up for it. That and having cheap shared netops support.
Marsymars 8y ago
What's the math/logic to get to 12 people?
- Mtinie 8y ago
  Two people per 6-hour timezone chunk, with one overlapping person splitting each chunk? 2 at UTC 0-8 1 at UTC 4-12 2 at UTC 8-16 1 at UTC 12-20 2 at UTC 16-24 1 at UTC 20-04 (Repeat)
- wstrange 8y ago
  Timezone coverage, vacations, training, sick days, etc.

scarface74 8y ago

That does bring up an interesting point. In hindsight, we already have duplicate infrastructure - a dev account and a production account. Why in the world was it decided to put both accounts in the same region? The separate account was setup partially on my insistence but it was set up in the same region. If needed, we could have done VPC peerings across regions. (https://aws.amazon.com/about-aws/whats-new/2017/11/announcing-support-for-inter-region-vpc-peering/ https://aws.amazon.com/about-aws/whats-new/2017/11/announcin...)

dodobirdlord 8y ago
There are some significant differences region to region in AWS. Different numbers of availability zones, different latencies based on datacenter location, different types and availabilities of EC2 instances, etc. I think it makes sense to develop in the same region that your production service runs in just so you don't shoot yourself in the foot by deploying something that runs fine in your development region but doesn't run as well in your production region.
devonkim 8y ago
In most of the cases I've seen so far with multiple regions / POP (4 different companies of different sizes and verticals) the software never being deployed in another reason is the fundamental reason. Reasons ranged from "we hard-coded us-east-1 everywhere and don't know how to test region independence it turns out" to "we have 3 weeks to try to make our software distributed between different regions but have nowhere near the resources to test it well let alone develop anything new for it." Automation is completely remedial and non-repeatable in most of these companies that rarely provision clean environments from scratch. By keeping regions the same between different environments, you reduce the number of possible differences. Some services in AWS are also not available in others (I'm quite familiar with AWS Data Pipeline not being available outside the "core" regions like us-east-1, eu-west-1) and having services in one region make usage of resources in another region is a huge change when most developers outside ones with technology literate customers are under the gun to push features out fast over sound design. The matrix of services and configurations necessary to mix and match regions and availability zones is non-trivial if you make extensive usage of AWS services above the IAAS layer. Also, cross-region VPC peering has a TON of limitations that rather annoying depending upon how well your network has been architected (by default in most companies outside enterprises with a deep bench of network engineers, this would be rated at "complete crap barely better than a typical home wifi network"). Heck, even though I'm non-dumb at networks I have to keep reminding myself of various cross-region VPC limitations when working with refactoring cross-region VPCs like where you can reference security groups, how to propagate Route 53 records, etc.