4 ms·

> We never thought our startup would be threatened by the unreliability of a company like Microsoft You will be threatened by your own unreliability of buildin

> We never thought our startup would be threatened by the unreliability of a company like Microsoft You will be threatened by your own unreliability of building something that's dependant on one region or one cloud.

websap 4y ago
This is an insidious argument to make. When building a startup you should choose 1 reliable cloud provider and use their best practices to support high availability.
- gtirloni 4y ago
  Cross-region architecture will be the first thing you hear about.
- pclmulqdq 4y ago
  No matter the provider, their best practices all say to be multi-region.
  - websap 4y ago
    Def not true with AWS, unless you reach a particular scale. Not for product market fit. My technology choices would be fully managed services so I could focus on my actual business.
    - Brian_K_White 4y ago
      def true with everything. what a ridiculous statement.
    - bradknowles 4y ago
      Read the "Well Architected" paper. Go multi-region.
      - websap 4y ago
        Can you link me to the well architected paper that talks about going multi-region for an early stage startup?
        pamelafox 4y ago
        This may be the reference: https://learn.microsoft.com/en-us/azure/architecture/framework/ https://learn.microsoft.com/en-us/azure/architecture/framewo... I just watched the second video on the page, and it does discuss multi-region a bit.
      - nixgeek 4y ago
        Unless eCF has made some major advancements in the last couple years, Amazon’s own retail business isn’t multi-region. So it’d be like the cobbler saying “Buy my shoes!” while wearing none, if AWS were to push everyone hard to have a multi-region strategy. At least on AWS you typically can find capacity (outside of accelerators) by being flexible on instance types (C, M, R), instance sizes, and availability zones. Sounds like this region OP is in for Azure is constrained such that even this advice doesn’t work.
  - Spivak 4y ago
    Multi-AZ and then grow into multi-region if the need arises. Multi-region is a huge lift the moment all your data must live in two regions simultaneously. Very few shops are experienced enough to run clusters across datacenters in a way that can handle the unhappy paths.
    - deleted 4y ago
      [deleted]
  - manv1 4y ago
    Multi-region in AWS means building it yourself. I suspect that the skills for real HA are atrophying because for 99% of the people multi-AZ is enough and most of the AWS stuff supports multi-az automagically. The problem with multi-region is that it means configuration, and there are probably lots of services that you can't actually configure to be multi-region. Cognito is one off the top of my head. It looks like the various aurora flavors do multi-region, but what about Neptune? SQS? API Gateway? AWS Lambda? MediaLive? Maybe you can hide all that behind DNS failover, maybe you can't. Real multi-region is basically means going back to old-school HA, and that was hard to do when it was your data centers. On AWS it'll be even harder. That isn't to say it's not possible, it's just a tremendous amount of work. I mean really, if us-east-1 is down 80% of the internet is screwed...so from an expectations point of view does HA of your particular service matter if that happens? Even for a financial outages happen. Once you have enough people it might be worth it. For a non mission critical startup? No fucking way.
janober 4y ago
Totally agree, we could for sure have build from the get go multi-region and multi-cloud but we had good reasons not to do it. Depending on the product, technology, ... would actually also strongly recommend almost every startup to do the same.
- whimsicalism 4y ago
  Seems bold to recommend everyone do the same as you when you are running in to problems you can't solve because of this exact choice you made.
  - janober 4y ago
    I am still 100% sure it was exactly the right decision. Was however in hindsight probably the right one to choose Azure and/or that data center.
    - ComputerGuru 4y ago
      If you go under because of this, will you still be 100% sure? Everything is for sure until it’s not.
- PaulHoule 4y ago
  No matter what stage of a service you are at you should have a documented procedure (ideally running a script) that can stand up a working instance of the system. This has vast benefits for agility and fast development when developers are not always fighting the build system and have a "no fear" attitude about deployment. If you have that, you can build a system in another region and be able to migrate wholesale to another region with more capacity and not be particularly concerned about the general problem of coordinating the service across multiple regions at the same time.
manv1 4y ago
A startup is about managing risks and spending your time/money appropriately. Your cloud provider running out of capacity isn't an obvious risk especially if it's just capacity for general compute. For some clouds that seem to be run on a manual process (IBM, Oracle) that would be expected, since they're sort of clunky. For other places (Rackspace, etc) it would uncommon. For a major provider like Azure, well, it's bizarre. I mean, the whole point of cloud is that it's all-you-can-eat. You would think that this would be something they would advertise/talk about up-front. But who would sign up if that was disclosed?

websap 4y ago

This is an insidious argument to make. When building a startup you should choose 1 reliable cloud provider and use their best practices to support high availability.

gtirloni 4y ago
Cross-region architecture will be the first thing you hear about.
pclmulqdq 4y ago
No matter the provider, their best practices all say to be multi-region.
- websap 4y ago
  Def not true with AWS, unless you reach a particular scale. Not for product market fit. My technology choices would be fully managed services so I could focus on my actual business.
  - Brian_K_White 4y ago
    def true with everything. what a ridiculous statement.
  - bradknowles 4y ago
    Read the "Well Architected" paper. Go multi-region.
    - websap 4y ago
      Can you link me to the well architected paper that talks about going multi-region for an early stage startup?
      - pamelafox 4y ago
        This may be the reference: https://learn.microsoft.com/en-us/azure/architecture/framework/ https://learn.microsoft.com/en-us/azure/architecture/framewo... I just watched the second video on the page, and it does discuss multi-region a bit.
    - nixgeek 4y ago
      Unless eCF has made some major advancements in the last couple years, Amazon’s own retail business isn’t multi-region. So it’d be like the cobbler saying “Buy my shoes!” while wearing none, if AWS were to push everyone hard to have a multi-region strategy. At least on AWS you typically can find capacity (outside of accelerators) by being flexible on instance types (C, M, R), instance sizes, and availability zones. Sounds like this region OP is in for Azure is constrained such that even this advice doesn’t work.
- Spivak 4y ago
  Multi-AZ and then grow into multi-region if the need arises. Multi-region is a huge lift the moment all your data must live in two regions simultaneously. Very few shops are experienced enough to run clusters across datacenters in a way that can handle the unhappy paths.
  - deleted 4y ago
    [deleted]
- manv1 4y ago
  Multi-region in AWS means building it yourself. I suspect that the skills for real HA are atrophying because for 99% of the people multi-AZ is enough and most of the AWS stuff supports multi-az automagically. The problem with multi-region is that it means configuration, and there are probably lots of services that you can't actually configure to be multi-region. Cognito is one off the top of my head. It looks like the various aurora flavors do multi-region, but what about Neptune? SQS? API Gateway? AWS Lambda? MediaLive? Maybe you can hide all that behind DNS failover, maybe you can't. Real multi-region is basically means going back to old-school HA, and that was hard to do when it was your data centers. On AWS it'll be even harder. That isn't to say it's not possible, it's just a tremendous amount of work. I mean really, if us-east-1 is down 80% of the internet is screwed...so from an expectations point of view does HA of your particular service matter if that happens? Even for a financial outages happen. Once you have enough people it might be worth it. For a non mission critical startup? No fucking way.

janober 4y ago

Totally agree, we could for sure have build from the get go multi-region and multi-cloud but we had good reasons not to do it. Depending on the product, technology, ... would actually also strongly recommend almost every startup to do the same.

whimsicalism 4y ago
Seems bold to recommend everyone do the same as you when you are running in to problems you can't solve because of this exact choice you made.
- janober 4y ago
  I am still 100% sure it was exactly the right decision. Was however in hindsight probably the right one to choose Azure and/or that data center.
  - ComputerGuru 4y ago
    If you go under because of this, will you still be 100% sure? Everything is for sure until it’s not.
PaulHoule 4y ago
No matter what stage of a service you are at you should have a documented procedure (ideally running a script) that can stand up a working instance of the system. This has vast benefits for agility and fast development when developers are not always fighting the build system and have a "no fear" attitude about deployment. If you have that, you can build a system in another region and be able to migrate wholesale to another region with more capacity and not be particularly concerned about the general problem of coordinating the service across multiple regions at the same time.

manv1 4y ago

A startup is about managing risks and spending your time/money appropriately. Your cloud provider running out of capacity isn't an obvious risk especially if it's just capacity for general compute. For some clouds that seem to be run on a manual process (IBM, Oracle) that would be expected, since they're sort of clunky. For other places (Rackspace, etc) it would uncommon. For a major provider like Azure, well, it's bizarre. I mean, the whole point of cloud is that it's all-you-can-eat. You would think that this would be something they would advertise/talk about up-front. But who would sign up if that was disclosed?