6 ms·

Oof, that sucks and I feel for you. That said... > setting up in a new region would be complicated for us. Sounds to me like you've got a few weeks to get thi

Oof, that sucks and I feel for you. That said... > setting up in a new region would be complicated for us. Sounds to me like you've got a few weeks to get this working. Deprioritize all other work, get everyone working on this little DevOps/Infra project. You should've been multi-region from the outset, if not multi-cloud. When using the public cloud, we do tend to take it all for granted and don't even think about the fact that physical hardware is required for our clusters and that, yes, they can run out. Anyways, however hard getting another region set up may be, it seems you've no choice but to prioritize that work now. May also want to look into other cloud providers as well, depending on how practical or how overkill going multi-cloud may or may not be for your needs. I wish you luck.

theteapot 4y ago
> everyone working on this little DevOps/Infra project. Everyone? That's not going to help.
deleted 4y ago
[deleted]
janober 4y ago
Thanks a lot! You are totally right, it is for sure something we will find a solution for. But honestly, do I not want to. As a startup, you have very few resources and deliberately place some exact bets. Deprioritizing everything to work on something for a long time that was not prioritized, just to then end up again where you were before (a working cloud solution) is the last thing any startup should be forced to do. Anyway, it seems like we do not have much choice here.
- xwowsersx 4y ago
  I hear you. It's not a fun position to be in. And sometimes you're correct to take calculated risks and maybe the expected value was positive here, despite what ended up happening. Without knowing the details about your services and infrastructure, it's hard for me to know what's involved in going multi-region now. Are you sure it's such a a gargantuan effort? I would've thought one person working full-time on this for a week or two would be enough, but again I don't know the details of your setup. One option would be to pay a consultant who is an expert in Azure/cloud stuff to come in and help. May not be cheap, but could be a lot better and quicker for you and better for the business, especially if none of you are really big experts in Azure. I've been here before (I think)...had to wear many hats and scramble to make sales, build the tech, act as de facto DevOps person even without a lot of experience doing it, etc. That is the way, but stuff happens. Happy to chat about specifics if you want to bounce ideas off of me or go through your particular situation. Can't promise I'll have concrete advice, but happy to talk it through.
  - janober 4y ago
    Thanks a lot, is really super nice of you and appreciated! Luckily we have somebody very knowledgeable on our team. Will tell him to reach out if he wants to have a peer to brain-storm some ideas.
    - xwowsersx 4y ago
      Glad to hear you have the right people. Good luck, my friend.
- martinald 4y ago
  I disagree with the other poster you should have been multiregion from the start. It adds a load of complexity and failure cases for early stage Startups. Very poor position to be in, apparently this happened in azure UK recently too.
  - dh2022 4y ago
    I do not think is a bad idea to be multi-region from start. For the most part Azure has at least two regions in each country (Germany North/West Central, UK South/West, Sweden South/Central, Norway East / West, UAE North / Central, France Central / South etc....)So if stuff happens being able to bring up your service in a different region in the same country could be helpful. I do not know specifics but it seems to me that having an abstraction layer on top of the region is not that hard to do (most of Azure services are supported in all regions). OF course, is a lot easier if done at the outset. Being forced to do it quickly and with little notice is no fun at all....
- hgsgm 4y ago
- jnsaff2 4y ago
  I feel for you. Also it sucks to be in this position. Let the scar you get from this is be a learning experience, hopefully you will not fall into the same trap again to trust this company. In my career I'm in a place where anyone suggesting I do work on Azure gets an instant doubling of my asking day-rate and I really hope the will be put off and find another victim for this gig. That said, another learning experience would be to use terraform or something (tbh for azure the only sane thing is terraform, ARM templates are just garbage). Having terraformed your one region switching to the other would be much easier, tho not trivial.
xwowsersx 4y ago
I'll reply to my own comment in response to a since-deleted reply that went something to the effect of "this is terrible advice for a young startup trying to get to product market for": I'm totally on board with the idea of being scrappy and taking shortcuts in order to get to PMF as soon as possible. However, it seems the proof is in the pudding here. If you can't service customers due to lack of compute resources, you can't get to PMF. Also, yes there are certain infrastructure and network topologies that would absolutely be overkill for a young startup. I don't think multi-region is one of those things. I don't have experience with Azure directly, but on every other cloud providers, going multi-region is not something that requires huge amounts of time or resources. You just need to be mindful of it from the outset. And if you decide not to be, then at least be intentional and conscious about the risk and have a plan in place for what happens when you get bit by deciding not to go multi-region.
- cheese123 4y ago
  GDPR might be a problem here. But this brings us to an important point: this is not your infrastructure, but someone elses.
  - xwowsersx 4y ago
    Good point, I didn't think about that.
  - soco 4y ago
    There are quite a few European regions, but we don't know how the others stand with their computing limits...
  - masom 4y ago
    GDPR isn't really related to the infrastructure, and isn't a problem if you built your product knowing you'll need to conform. Shopify is GDPR compliant, for all merchants, and runs on Google Cloud in multiple regions.
    - revicon 4y ago
websap 4y ago
Is it that much cheaper for you to build a new region on Azure versus getting setup on AWS? If you rely on Kubernetes for orchestration and have minimal cloud API dependency, it may be worth that evaluating this option. Also, do you have a TAM associated with your account? Are you just going through regular support channels? Can they deliver different instance types (not sure what the Azure parallel is), can they deliver short term capacity, etc? I would try to push Microsoft more here. It's not like they've stopped on-boarding new customers into that region right? What happens if you create a new account in that region?
- janober 4y ago
  We already tried to push Microsoft, sadly have they been not very helpful. Still trying to get in contact with somebody that can actually make a difference. After all, are we also not asking for a hundred machines. Can really not imagine that they can not somehow make the resources we require available.
  - dekhn 4y ago
    I'll ask again what the person above asked: do you have a TAM. If you don't you're at a big disadvantage.
    - janober 4y ago
      Sorry, missed that. No, we do not.
      - websap 4y ago
        Get one asap. Your TAM is the insider and should push for you.
    - cyptus 4y ago
      we tried escalating this through CASM and it did not work. The region is blocked for every quota, even a single instance.
  - deleted 4y ago
psychphysic 4y ago
> Deprioritize all other work, get everyone working on this little DevOps/Infra project. This is doubly worthwhile as if this stumble kills the startup (it can happen) this will be excellent experience to take to the next employer :)
- lanstin 4y ago
  Also a lesson here is don’t create any prod asserts manually ever. Terraform or some other software to define your cloud assets. Then this issue is just a matter of adding a top level loop or maybe adding a region parameter to a layer of software. Cloud is only efficient if you take the software defined every thing seriously. Otherwise it is premium hosting where you are likely a small fish.
snotrockets 4y ago
“Being multicloud from the outset” is a very silly idea for most use cases. The way to get more from most cloud is by becoming a partner, not just a customer. And the way to do that is increase dependency and usage.
- salawat 4y ago
  t. Sales department of Cloud provider
  - snotrockets 4y ago
    Good sales (which I'm not), is all about aligning with the customer: solve their problems for them, and they'd be happy to pay for it. Getting them to buy stuff they won't need is a sure way to loose future business.
astrostl 4y ago
"Multi-cloud from the outset" is probably the single-worst generic cloud advice that I think anyone could be given. In professional cloud consulting the rule of thumb is to do one cloud with excellence until you even think about another one. And even that is really just kicking the conversational can, as both becoming excellent and actually needing multi-cloud combined is one-in-a-billion.
- gumby 4y ago
  That pretty much binds your hands since in our experience the one provider who can do “one cloud with excellence” is AWS. (As an aside I also agree that multi cloud from the get go is a YAGNI violation. Just keep in the back of your mind “could we have an alternative to this?” when using your provider’s proprietary features.)
  - marcosdumay 4y ago
    That generalizes to every kind of lock-in: have a viable escape plan, but only execute it if you need or it becomes cheap enough that it won't harm you. Just having the plan is already expensive enough.
  - jiggawatts 4y ago
    My experience is the opposite: AWS has more features on paper but most of them exist only to tick a checkbox. Azure has more integrations between their offerings, as well as Azure Active Directory, and Microsoft 365.
    - 29athrowaway 4y ago
      And Active Directory integrates horribly with everything outside Microsoft.
      - SgtBastard 4y ago
        Azure Active Directory has both SAML and OpenID Connect endpoints… what’s missing?
mhitza 4y ago
I take offense with you comment. It's not the first time I'm hearing about multi-region/multi-cloud in online tech forums, however reality doesn't match. I don't want to be snarky, but when large service providers like AWS have their own crossregion downtime because one snowflake of a service in us-east-1 is down, I kind of dismiss the virtue signaling of high resilient multi-(az/region/cloud) ever existing in practice. If you can somehow have a separate database per region/cloud, sure, I can understand that, but if you have to shard your database across many clouds, I'd dread having to tame such a beast, especially within a startup.
- panarky 4y ago
  > dismiss the virtue signaling of high resilient ... So you're saying it's impossible to improve reliability from 97% to 99% because you can never make it to 100%.
  - quesera 4y ago
    If your single-AZ, single-region cloud is not giving you 3 or 4 9's of reliability out of the box, you are using the wrong cloud. Multi-AZ and multi-region add complexity and cost much more quickly than they add reliability. Sometimes it is worth it. Sometimes it is not.
    - champtar 4y ago
      Depends on your needs, but having your data & database multi-az to ensure durability can avoid you having to restart from backups. I'm thinking about an old AWS incident were they actually lost EBS data: https://www.bleepingcomputer.com/news/technology/amazon-aws-outage-shows-data-in-the-cloud-is-not-always-safe/ https://www.bleepingcomputer.com/news/technology/amazon-aws-... Also make sure your backups are in a different AZ (thinking about OVH ...) or region or even at a different provider.
rch 4y ago
Thanks largely to k8s, running on multiple cloud providers and your own hardware is a lot more convenient than it was a few years ago. Component interfaces and protocols are a lot more consistent across platforms as well.

theteapot 4y ago

> everyone working on this little DevOps/Infra project. Everyone? That's not going to help.

deleted 4y ago

[deleted]

janober 4y ago

Thanks a lot! You are totally right, it is for sure something we will find a solution for. But honestly, do I not want to. As a startup, you have very few resources and deliberately place some exact bets. Deprioritizing everything to work on something for a long time that was not prioritized, just to then end up again where you were before (a working cloud solution) is the last thing any startup should be forced to do. Anyway, it seems like we do not have much choice here.

xwowsersx 4y ago
I hear you. It's not a fun position to be in. And sometimes you're correct to take calculated risks and maybe the expected value was positive here, despite what ended up happening. Without knowing the details about your services and infrastructure, it's hard for me to know what's involved in going multi-region now. Are you sure it's such a a gargantuan effort? I would've thought one person working full-time on this for a week or two would be enough, but again I don't know the details of your setup. One option would be to pay a consultant who is an expert in Azure/cloud stuff to come in and help. May not be cheap, but could be a lot better and quicker for you and better for the business, especially if none of you are really big experts in Azure. I've been here before (I think)...had to wear many hats and scramble to make sales, build the tech, act as de facto DevOps person even without a lot of experience doing it, etc. That is the way, but stuff happens. Happy to chat about specifics if you want to bounce ideas off of me or go through your particular situation. Can't promise I'll have concrete advice, but happy to talk it through.
- janober 4y ago
  Thanks a lot, is really super nice of you and appreciated! Luckily we have somebody very knowledgeable on our team. Will tell him to reach out if he wants to have a peer to brain-storm some ideas.
  - xwowsersx 4y ago
    Glad to hear you have the right people. Good luck, my friend.
martinald 4y ago
I disagree with the other poster you should have been multiregion from the start. It adds a load of complexity and failure cases for early stage Startups. Very poor position to be in, apparently this happened in azure UK recently too.
- dh2022 4y ago
  I do not think is a bad idea to be multi-region from start. For the most part Azure has at least two regions in each country (Germany North/West Central, UK South/West, Sweden South/Central, Norway East / West, UAE North / Central, France Central / South etc....)So if stuff happens being able to bring up your service in a different region in the same country could be helpful. I do not know specifics but it seems to me that having an abstraction layer on top of the region is not that hard to do (most of Azure services are supported in all regions). OF course, is a lot easier if done at the outset. Being forced to do it quickly and with little notice is no fun at all....
hgsgm 4y ago
jnsaff2 4y ago
I feel for you. Also it sucks to be in this position. Let the scar you get from this is be a learning experience, hopefully you will not fall into the same trap again to trust this company. In my career I'm in a place where anyone suggesting I do work on Azure gets an instant doubling of my asking day-rate and I really hope the will be put off and find another victim for this gig. That said, another learning experience would be to use terraform or something (tbh for azure the only sane thing is terraform, ARM templates are just garbage). Having terraformed your one region switching to the other would be much easier, tho not trivial.

xwowsersx 4y ago

I'll reply to my own comment in response to a since-deleted reply that went something to the effect of "this is terrible advice for a young startup trying to get to product market for": I'm totally on board with the idea of being scrappy and taking shortcuts in order to get to PMF as soon as possible. However, it seems the proof is in the pudding here. If you can't service customers due to lack of compute resources, you can't get to PMF. Also, yes there are certain infrastructure and network topologies that would absolutely be overkill for a young startup. I don't think multi-region is one of those things. I don't have experience with Azure directly, but on every other cloud providers, going multi-region is not something that requires huge amounts of time or resources. You just need to be mindful of it from the outset. And if you decide not to be, then at least be intentional and conscious about the risk and have a plan in place for what happens when you get bit by deciding not to go multi-region.

cheese123 4y ago
GDPR might be a problem here. But this brings us to an important point: this is not your infrastructure, but someone elses.
- xwowsersx 4y ago
  Good point, I didn't think about that.
- soco 4y ago
  There are quite a few European regions, but we don't know how the others stand with their computing limits...
- masom 4y ago
  GDPR isn't really related to the infrastructure, and isn't a problem if you built your product knowing you'll need to conform. Shopify is GDPR compliant, for all merchants, and runs on Google Cloud in multiple regions.
  - revicon 4y ago

websap 4y ago

Is it that much cheaper for you to build a new region on Azure versus getting setup on AWS? If you rely on Kubernetes for orchestration and have minimal cloud API dependency, it may be worth that evaluating this option. Also, do you have a TAM associated with your account? Are you just going through regular support channels? Can they deliver different instance types (not sure what the Azure parallel is), can they deliver short term capacity, etc? I would try to push Microsoft more here. It's not like they've stopped on-boarding new customers into that region right? What happens if you create a new account in that region?

janober 4y ago
We already tried to push Microsoft, sadly have they been not very helpful. Still trying to get in contact with somebody that can actually make a difference. After all, are we also not asking for a hundred machines. Can really not imagine that they can not somehow make the resources we require available.
- dekhn 4y ago
  I'll ask again what the person above asked: do you have a TAM. If you don't you're at a big disadvantage.
  - janober 4y ago
    Sorry, missed that. No, we do not.
    - websap 4y ago
      Get one asap. Your TAM is the insider and should push for you.
  - cyptus 4y ago
    we tried escalating this through CASM and it did not work. The region is blocked for every quota, even a single instance.
- deleted 4y ago

psychphysic 4y ago

> Deprioritize all other work, get everyone working on this little DevOps/Infra project. This is doubly worthwhile as if this stumble kills the startup (it can happen) this will be excellent experience to take to the next employer :)

lanstin 4y ago
Also a lesson here is don’t create any prod asserts manually ever. Terraform or some other software to define your cloud assets. Then this issue is just a matter of adding a top level loop or maybe adding a region parameter to a layer of software. Cloud is only efficient if you take the software defined every thing seriously. Otherwise it is premium hosting where you are likely a small fish.

snotrockets 4y ago

“Being multicloud from the outset” is a very silly idea for most use cases. The way to get more from most cloud is by becoming a partner, not just a customer. And the way to do that is increase dependency and usage.

salawat 4y ago
t. Sales department of Cloud provider
- snotrockets 4y ago
  Good sales (which I'm not), is all about aligning with the customer: solve their problems for them, and they'd be happy to pay for it. Getting them to buy stuff they won't need is a sure way to loose future business.

astrostl 4y ago

"Multi-cloud from the outset" is probably the single-worst generic cloud advice that I think anyone could be given. In professional cloud consulting the rule of thumb is to do one cloud with excellence until you even think about another one. And even that is really just kicking the conversational can, as both becoming excellent and actually needing multi-cloud combined is one-in-a-billion.

gumby 4y ago
That pretty much binds your hands since in our experience the one provider who can do “one cloud with excellence” is AWS. (As an aside I also agree that multi cloud from the get go is a YAGNI violation. Just keep in the back of your mind “could we have an alternative to this?” when using your provider’s proprietary features.)
- marcosdumay 4y ago
  That generalizes to every kind of lock-in: have a viable escape plan, but only execute it if you need or it becomes cheap enough that it won't harm you. Just having the plan is already expensive enough.
- jiggawatts 4y ago
  My experience is the opposite: AWS has more features on paper but most of them exist only to tick a checkbox. Azure has more integrations between their offerings, as well as Azure Active Directory, and Microsoft 365.
  - 29athrowaway 4y ago
    And Active Directory integrates horribly with everything outside Microsoft.
    - SgtBastard 4y ago
      Azure Active Directory has both SAML and OpenID Connect endpoints… what’s missing?

mhitza 4y ago

I take offense with you comment. It's not the first time I'm hearing about multi-region/multi-cloud in online tech forums, however reality doesn't match. I don't want to be snarky, but when large service providers like AWS have their own crossregion downtime because one snowflake of a service in us-east-1 is down, I kind of dismiss the virtue signaling of high resilient multi-(az/region/cloud) ever existing in practice. If you can somehow have a separate database per region/cloud, sure, I can understand that, but if you have to shard your database across many clouds, I'd dread having to tame such a beast, especially within a startup.

panarky 4y ago
> dismiss the virtue signaling of high resilient ... So you're saying it's impossible to improve reliability from 97% to 99% because you can never make it to 100%.
- quesera 4y ago
  If your single-AZ, single-region cloud is not giving you 3 or 4 9's of reliability out of the box, you are using the wrong cloud. Multi-AZ and multi-region add complexity and cost much more quickly than they add reliability. Sometimes it is worth it. Sometimes it is not.
  - champtar 4y ago
    Depends on your needs, but having your data & database multi-az to ensure durability can avoid you having to restart from backups. I'm thinking about an old AWS incident were they actually lost EBS data: https://www.bleepingcomputer.com/news/technology/amazon-aws-outage-shows-data-in-the-cloud-is-not-always-safe/ https://www.bleepingcomputer.com/news/technology/amazon-aws-... Also make sure your backups are in a different AZ (thinking about OVH ...) or region or even at a different provider.

rch 4y ago

Thanks largely to k8s, running on multiple cloud providers and your own hardware is a lot more convenient than it was a few years ago. Component interfaces and protocols are a lot more consistent across platforms as well.