4 ms·

Very few companies or systems need near-perfect uptime. Multi-region cloud engineering, especially once data is involved, is incredibly expensive. If you do nee

Very few companies or systems need near-perfect uptime. Multi-region cloud engineering, especially once data is involved, is incredibly expensive. If you do need the kind of resiliency you usually engineer it for just a very specific component rather than the entire system. An outage like this happens how often? Edit: Looks like this is affecting a single AZ... so bit different situation, but I would agree if you're not capable of surviving a single AZ outage in 2019 then your engineering team should be replaced.

geggam 7y ago
Or the PM team that wouldnt let the engineers do it right needs replaced. Always CYA guys... you will pay for this if you dont.
fenichelar 7y ago
Most of the connectivity issues we were seeing were with instances in one of the us-east-1 AZs but we were seeing issues in other AZs in us-east-1 as well. Not sure why AWS is acting like this issue in only affecting one AZ.
Dunedan 7y ago
> Very few companies or systems need near-perfect uptime. Multi-region cloud engineering, especially once data is involved, is incredibly expensive. Without data it doesn't even cost significantly more than running in a single region nowadays, if you are willing to go serverless. As serverless stuff (FaaS, ...) is pay for what you use and the provider handles the scaling automatically behind the curtain you can easily deploy to multiple regions without much additional cost. With data you have of course the cost of storing the data multiple times in the different regions (or to come up with some kind of sharding) and solving the consistency challenges that come with that, but at least services like DynamoDB and S3 offer cross-region replication out of the box nowadays and you don't have to provision any capacity like you used to (thanks to DynamoDB AutoScaling and so on). Once you have your application running in multiple regions you can direct users to the closest one, so they enjoy lower latencies. I believe for a lot of applications running cross-region just makes a lot of sense as it offers various benefits.
sgustard 7y ago
> your engineering team should be replaced My engineers are all React and CSS web developers. They don't know anything about multi tenant data resiliency. But they can make a real pretty "system down" page.

geggam 7y ago

Or the PM team that wouldnt let the engineers do it right needs replaced. Always CYA guys... you will pay for this if you dont.

fenichelar 7y ago

Most of the connectivity issues we were seeing were with instances in one of the us-east-1 AZs but we were seeing issues in other AZs in us-east-1 as well. Not sure why AWS is acting like this issue in only affecting one AZ.

Dunedan 7y ago

> Very few companies or systems need near-perfect uptime. Multi-region cloud engineering, especially once data is involved, is incredibly expensive. Without data it doesn't even cost significantly more than running in a single region nowadays, if you are willing to go serverless. As serverless stuff (FaaS, ...) is pay for what you use and the provider handles the scaling automatically behind the curtain you can easily deploy to multiple regions without much additional cost. With data you have of course the cost of storing the data multiple times in the different regions (or to come up with some kind of sharding) and solving the consistency challenges that come with that, but at least services like DynamoDB and S3 offer cross-region replication out of the box nowadays and you don't have to provision any capacity like you used to (thanks to DynamoDB AutoScaling and so on). Once you have your application running in multiple regions you can direct users to the closest one, so they enjoy lower latencies. I believe for a lot of applications running cross-region just makes a lot of sense as it offers various benefits.

sgustard 7y ago

> your engineering team should be replaced My engineers are all React and CSS web developers. They don't know anything about multi tenant data resiliency. But they can make a real pretty "system down" page.