6 ms·

Their siloing strategy, which I'll roughly refer as resolving a request from a single AZ, is a good way to keep operations and monitoring simple. A past team o

Their siloing strategy, which I'll roughly refer as resolving a request from a single AZ, is a good way to keep operations and monitoring simple. A past team of mine managed services in a similar fashion. We had a couple (usually 2-4) single AZ clusters with a thin (Envoy) layer to balance traffic between clusters. We could detect incidents in a single cluster by comparing metrics across clusters. Mitigation was easy, we could drain a cluster in under a minute, redirecting traffic to the other ones. Most traffic was intra AZ, so it was fast and there was no cross-AZ networking fees. The downside is that most services were running in several clusters, so there was redundancy in compute, caches, etc. When we talked to people outside the company, e.g. solution architects from our cloud provider, they would be surprised at our architecture and immediately suggest multi-region clusters. I would joke that our single AZ clusters were a feature, not a bug. Nice to see other folks having success with a similar architecture!

ComputerGuru 3y ago
It sounds like you didn’t have persistent data, and were only offering compute? If there’s no need for a coherent master view accessible/writeable from all the clusters, there would be no reason to use multi-region cluster whatsoever.
- athoscouto 3y ago
  We did. But the persisted data didn't live inside those ephemeral compute clusters though.
  - lordofnorn 3y ago
    Yeah, keep stateful stuff and stateless stuff separate; separate clusters, network spaces, cloud accounts, likely a mix of all that. Clearly define boundaries and acceptable behavior within boundaries. Setup up telemetry and observability to monitor for threshold violations. Simple. Right?
    - catchnear4321 3y ago
      i mean you could also just spin up a reeeeeally big compute node and just do it all there. fewer things to monitor. fewer things that can fail. just log in from time to time to update packages. see, cloud doesn’t have to be complex.
      - lordofnorn 3y ago
        Hey you know what; you do you. I’m an EE by education. It’s electron state, silly leaky abstraction Stan’ing to my head. Different babble for allocation of memory and algorithmic manipulation of the values stored within. Correctness is important when it comes to results being mapped to human consumption and even then the subset of parameters to be be rigorous with can be made subj. personally I lean on a subset that includes biological health and well being and deprioritize religiosity
      - theamk 3y ago
        until you get "hardware failure" note from cloud provider. Or the person updating packages makes a typo and messes up a system. sure, use "pet" computers for experiments and dev.. but having produluction be a "cattle" makes your life so much less stressful.
        pclmulqdq 3y ago
        Having a few computers doesn't imply a "pets" approach to managing them.
        catchnear4321 3y ago
        more like a zoo, right?
        singron 3y ago
        I think everyone internalizes pets vs cattle a little differently, but I think people often don't realize that if you have cattle, then you now have a cattle ranch. I.e. you often create a new concept to manage your cattle, and you often run that like it's a pet. E.g. you want to think about your machines/containers like cattle, so you put them into a kubernetes cluster, which has become your new pet. If all your infra fits on one machine, it's way easier to have that as your pet the same way it's easier to have a dog than run a livestock operation.
  - slashdev 3y ago
    So your data store was still multi AZ? I’m a little confused how you’d serve the same user’s data consistently from multiple silos. Do you pin users to one AZ?
bushbaba 3y ago
The downside of single AZ clusters is capacity. If you have a need to drastically scale up the compute might not be available in a single AZ.
- athoscouto 3y ago
  Even though each cluster was single AZ the whole system wasn't, so we weren't bound by the capacity of a single AZ. Most of the situations where we needed to drastically scale up were known ahead of time as well (e.g. campaign from customer), and we would preallocate instances or even more clusters. I may be forcing my memory, but if I'm not mistaken, our auto scaling was setup in a way that the system could handle sudden load increases of ~50% without noticeable disruption. Spikes bigger than this could lead to increased latency and/or error rate.
  - kccqzy 3y ago
    That's another way of saying your typical utilization ratio is 66%. Which is on the low side honestly. That said, it's a trade off between efficiency and load spike tolerance. I trust that the trade off is made with informed decision.
    - rewmie 3y ago
      > That said, it's a trade off between efficiency and load spike tolerance. I trust that the trade off is made with informed decision. I don't think that relatively low utilization rates is the scenario that requires "informed decision". The only tradeoff in low utilization rate scenarios is cost, which might be outright cheaper and irrelevant once you do the math on the tradeoffs of using reserved instances vs the cost of scaling up with on-demand instances. You need to make a damn good case to chronically underprovision your system and expect it to autoscale your way into nickle-and-dime savings.
    - ec109685 3y ago
      66% isn’t low utilization. You’re always going to have micro spikes, and you never want to clip, so keeping some headroom around feels smart. Unless you co-mingle online and offline (batch) traffic on same hosts, flat response times and high utilization aren’t compatible.
jasonwatkinspdx 3y ago
Yeah, I talked with a business that used a similar architecture for the same reasons. It can be really effective in multi-tenant apps where each customers data is fully independent and private. They also used multiple Amazon organizational accounts as a security partition. It made a few things more difficult but they felt the peace of mind was worth it.
- mox1 3y ago
  My company has a pretty unique strategy where we have separate AWS accounts for each unit within the company. Each unit gets a prod and non-prod account. We have ~150 accounts, so roughly 75 different department, with some having not much and others have a lot of resources. Its complex, but provides a lot of nice security primitives. We have an overarching administrative account, but that doesnt get used (and lots of alarm bells go off when it is).
endymi0n 3y ago
It took me some time to realize that Cloud Solution Architects are also just slightly more technical sales people in disguise whose only mission is upselling you onto more dependency. Same thing about their PR, every CxO these days says they need "multi-cloud", whatever that means and the costs are usually enormous, while complexity rises — with questionable benefit. I did the math for our own stack and after a setback month in client revenue, and decided to put all our servers into a single AZ in a single region. The only multi-AZ, multi-region services are our backups. Surviving bad machines happens often enough that it's priced in via using Kubernetes, but losing a whole AZ is a freak accident that's just SO rare that calculating real business risk, it seemed apt to pretend it just doesn't happen (sorry, Google Cloud Paris customers). Call me reckless, but I haven't looked back ever since and it saves us thousands of dollars in intra-AZ fees per month alone.
- deleted 3y ago
  [deleted]
- anon84873628 3y ago
  Yeah, for many businesses it probably isn't necessary to have crazy short RTO and RPO. Just restore the most recent backup in a new region and point at the cloud provider outage report...
- dtx1 3y ago
  I think there's this general problem with cloud deployments I'm seeing happen more and more: People building this huge Multi AZ, Hyper Redundant, Multi-National, infinitly scaling Cloud Solution for something that requires a single VM and a Database. Most Companies just don't need that level of scale and would be better off building something smaller and when you actually do scale you rewrite it with the profits made from the smaller solutions. Of course there are many companies that do require something large but you should seriously consider if something smaller will do first. I think solutions like a 100% cloudflare workers based backend can sidestep this a little but usually, that's not possibly or even the right thing in every situation.
AugustoCAS 3y ago
I assume you were using AWS? I know some of the AZ of other cloud providers (Azure? Oracle? Google?) are not fully siloed. They might have independent power and networking, but be in the same physical location. I'm mentioning this for other people to be aware as one can easily make the assumption that an AZ is the same concept on all clouds, which is not true and painful to realise.
- ak007 3y ago
  Thanks for highlighting this. Indeed all CSPs are not the same
- jeremyjh 3y ago
  Actually I assumed AWS did it the same way as the others - I thought maybe they are in another building on a campus but I didn’t think that should be a factor in planning and that I should use regions for geographic redundancy anyway.
  - nijave 3y ago
    Afaik AWS AZs are physically separate. I think some maps exist. Around here, there's 3 AZs and they're multi building campuses about 10-20 miles apart situated in suburbs outside the city freeway belt.
    - jamesfinlayson 3y ago
      Yeah I definitely thought they were physically separate - I thought that an AZ might span multiple data centres too (could be wrong) and I thought they had to be at least 10km apart.
- pl4nty 3y ago
  Azure's zones are "physically separate", but it's unclear whether zones could be in the same building. Especially since they don't guarantee distance between zones - they just aim for 300mi (483km)

ComputerGuru 3y ago

It sounds like you didn’t have persistent data, and were only offering compute? If there’s no need for a coherent master view accessible/writeable from all the clusters, there would be no reason to use multi-region cluster whatsoever.

athoscouto 3y ago
We did. But the persisted data didn't live inside those ephemeral compute clusters though.
- lordofnorn 3y ago
  Yeah, keep stateful stuff and stateless stuff separate; separate clusters, network spaces, cloud accounts, likely a mix of all that. Clearly define boundaries and acceptable behavior within boundaries. Setup up telemetry and observability to monitor for threshold violations. Simple. Right?
  - catchnear4321 3y ago
    i mean you could also just spin up a reeeeeally big compute node and just do it all there. fewer things to monitor. fewer things that can fail. just log in from time to time to update packages. see, cloud doesn’t have to be complex.
    - lordofnorn 3y ago
      Hey you know what; you do you. I’m an EE by education. It’s electron state, silly leaky abstraction Stan’ing to my head. Different babble for allocation of memory and algorithmic manipulation of the values stored within. Correctness is important when it comes to results being mapped to human consumption and even then the subset of parameters to be be rigorous with can be made subj. personally I lean on a subset that includes biological health and well being and deprioritize religiosity
    - theamk 3y ago
      until you get "hardware failure" note from cloud provider. Or the person updating packages makes a typo and messes up a system. sure, use "pet" computers for experiments and dev.. but having produluction be a "cattle" makes your life so much less stressful.
      - pclmulqdq 3y ago
        Having a few computers doesn't imply a "pets" approach to managing them.
        catchnear4321 3y ago
        more like a zoo, right?
      - singron 3y ago
        I think everyone internalizes pets vs cattle a little differently, but I think people often don't realize that if you have cattle, then you now have a cattle ranch. I.e. you often create a new concept to manage your cattle, and you often run that like it's a pet. E.g. you want to think about your machines/containers like cattle, so you put them into a kubernetes cluster, which has become your new pet. If all your infra fits on one machine, it's way easier to have that as your pet the same way it's easier to have a dog than run a livestock operation.
- slashdev 3y ago
  So your data store was still multi AZ? I’m a little confused how you’d serve the same user’s data consistently from multiple silos. Do you pin users to one AZ?

bushbaba 3y ago

The downside of single AZ clusters is capacity. If you have a need to drastically scale up the compute might not be available in a single AZ.

athoscouto 3y ago
Even though each cluster was single AZ the whole system wasn't, so we weren't bound by the capacity of a single AZ. Most of the situations where we needed to drastically scale up were known ahead of time as well (e.g. campaign from customer), and we would preallocate instances or even more clusters. I may be forcing my memory, but if I'm not mistaken, our auto scaling was setup in a way that the system could handle sudden load increases of ~50% without noticeable disruption. Spikes bigger than this could lead to increased latency and/or error rate.
- kccqzy 3y ago
  That's another way of saying your typical utilization ratio is 66%. Which is on the low side honestly. That said, it's a trade off between efficiency and load spike tolerance. I trust that the trade off is made with informed decision.
  - rewmie 3y ago
    > That said, it's a trade off between efficiency and load spike tolerance. I trust that the trade off is made with informed decision. I don't think that relatively low utilization rates is the scenario that requires "informed decision". The only tradeoff in low utilization rate scenarios is cost, which might be outright cheaper and irrelevant once you do the math on the tradeoffs of using reserved instances vs the cost of scaling up with on-demand instances. You need to make a damn good case to chronically underprovision your system and expect it to autoscale your way into nickle-and-dime savings.
  - ec109685 3y ago
    66% isn’t low utilization. You’re always going to have micro spikes, and you never want to clip, so keeping some headroom around feels smart. Unless you co-mingle online and offline (batch) traffic on same hosts, flat response times and high utilization aren’t compatible.

jasonwatkinspdx 3y ago

Yeah, I talked with a business that used a similar architecture for the same reasons. It can be really effective in multi-tenant apps where each customers data is fully independent and private. They also used multiple Amazon organizational accounts as a security partition. It made a few things more difficult but they felt the peace of mind was worth it.

mox1 3y ago
My company has a pretty unique strategy where we have separate AWS accounts for each unit within the company. Each unit gets a prod and non-prod account. We have ~150 accounts, so roughly 75 different department, with some having not much and others have a lot of resources. Its complex, but provides a lot of nice security primitives. We have an overarching administrative account, but that doesnt get used (and lots of alarm bells go off when it is).

endymi0n 3y ago

It took me some time to realize that Cloud Solution Architects are also just slightly more technical sales people in disguise whose only mission is upselling you onto more dependency. Same thing about their PR, every CxO these days says they need "multi-cloud", whatever that means and the costs are usually enormous, while complexity rises — with questionable benefit. I did the math for our own stack and after a setback month in client revenue, and decided to put all our servers into a single AZ in a single region. The only multi-AZ, multi-region services are our backups. Surviving bad machines happens often enough that it's priced in via using Kubernetes, but losing a whole AZ is a freak accident that's just SO rare that calculating real business risk, it seemed apt to pretend it just doesn't happen (sorry, Google Cloud Paris customers). Call me reckless, but I haven't looked back ever since and it saves us thousands of dollars in intra-AZ fees per month alone.

deleted 3y ago
[deleted]
anon84873628 3y ago
Yeah, for many businesses it probably isn't necessary to have crazy short RTO and RPO. Just restore the most recent backup in a new region and point at the cloud provider outage report...
dtx1 3y ago
I think there's this general problem with cloud deployments I'm seeing happen more and more: People building this huge Multi AZ, Hyper Redundant, Multi-National, infinitly scaling Cloud Solution for something that requires a single VM and a Database. Most Companies just don't need that level of scale and would be better off building something smaller and when you actually do scale you rewrite it with the profits made from the smaller solutions. Of course there are many companies that do require something large but you should seriously consider if something smaller will do first. I think solutions like a 100% cloudflare workers based backend can sidestep this a little but usually, that's not possibly or even the right thing in every situation.

AugustoCAS 3y ago

I assume you were using AWS? I know some of the AZ of other cloud providers (Azure? Oracle? Google?) are not fully siloed. They might have independent power and networking, but be in the same physical location. I'm mentioning this for other people to be aware as one can easily make the assumption that an AZ is the same concept on all clouds, which is not true and painful to realise.

ak007 3y ago
Thanks for highlighting this. Indeed all CSPs are not the same
jeremyjh 3y ago
Actually I assumed AWS did it the same way as the others - I thought maybe they are in another building on a campus but I didn’t think that should be a factor in planning and that I should use regions for geographic redundancy anyway.
- nijave 3y ago
  Afaik AWS AZs are physically separate. I think some maps exist. Around here, there's 3 AZs and they're multi building campuses about 10-20 miles apart situated in suburbs outside the city freeway belt.
  - jamesfinlayson 3y ago
    Yeah I definitely thought they were physically separate - I thought that an AZ might span multiple data centres too (could be wrong) and I thought they had to be at least 10km apart.
pl4nty 3y ago
Azure's zones are "physically separate", but it's unclear whether zones could be in the same building. Especially since they don't guarantee distance between zones - they just aim for 300mi (483km)