4 ms·

How much of this is really meant for AWS sized problems?

by Dowwie 4y ago

twblalock 4y ago
If you deploy to multiple regions or datacenters it doesn't matter how big you are -- you should orchestrate deployments so you don't deploy to all of them at the same time. If something breaks, you'd rather break a subset of regions than all of them. You should also deploy new code to some kind of non-production environment before going to prod. Even companies with the best possible canarying in prod will still do that. Some of the technology involved is definitely AWS-scale, but the principles of software rollout are broadly applicable.
faizshah 4y ago
Even small companies deploy to staging environments before production. Your deployment pipeline most likely looks like: - Build your language artifacts - Build a container - Deploy that container somewhere - Run approvals like integ tests, unit tests etc. and monitor metrics to decide whether to deploy to your next environment or rollback. This is just a structured way of building these CD pipelines you could also use Jenkins or Google started building something called Skaffold for Google Cloud Deploy: https://skaffold.dev/docs/ https://skaffold.dev/docs/ As your company gets larger you'll want to limit the blast radius of new changes so that you can meet your SLAs and you'll want to use multiple AZs and cloud regions for HA. That's where the idea of waves and cells comes from.
birdymcbird 4y ago
thats true. even if not amazon scale.. lets say you building multi tenant SaaS product. cell-architecture provide strong tenant isolation avoid noisy neighbor problem. other benefits like deploy changes to one cell so tenant may request delaying upgrade if they need to do pre-req work before hand. for ordinary service use case or web site.. most probability do not need this. over engineering infrastructure when should focus on customer and problem domain..thing that increase top or bottom line. source: principal SDE who launched several services at amazon using cell architecture.
ygjb 4y ago
Not much. Having a function build pipeline takes a bit of work to get set up, but once it's up and running, and projects are built to depend on it, it takes away a fair bit of ops work. That can help smaller teams become more agile, and being able to walk through a consistent process across multiple products for each step from commit against a dev branch through deployment in production makes it easier to account for team members being away, training new folks, etc. This is especially important for "side projects" or other small services that end up becoming critical infrastructure but are really maintained by one or two team members and only have a couple of internal customers.
vishnugupta 4y ago
The very first line of the article[1] calls it out, "....for enterprise-grade deployment pipelines." So I guess it's relevant for Fortune 500ish companies. That said, a base version of this pipeline is very useful even for a small startups. I concede that it takes a bit of time (took me about a week) to setup end to end. By end to end I mean beginning with empty AWS account to setting up VPC, ECS, CloudFront, Fargate, Codepipeline and so on. But once set you don't have to touch it for months on end. Scaling up/down, if needed, is as simple as minor config change. [1] https://aws.amazon.com/blogs/aws/new_deployment_pipelines_reference_architecture_and_-reference_implementations/ https://aws.amazon.com/blogs/aws/new_deployment_pipelines_re...

How much of this is really meant for AWS sized problems?

by Dowwie 4y ago

How much of this is really meant for AWS sized problems?

If you deploy to multiple regions or datacenters it doesn't matter how big you are -- you should orchestrate deployments so you don't deploy to all of them at the same time. If something breaks, you'd rather break a subset of regions than all of them. You should also deploy new code to some kind of non-production environment before going to prod. Even companies with the best possible canarying in prod will still do that. Some of the technology involved is definitely AWS-scale, but the principles of software rollout are broadly applicable.

faizshah 4y ago

Even small companies deploy to staging environments before production. Your deployment pipeline most likely looks like: - Build your language artifacts - Build a container - Deploy that container somewhere - Run approvals like integ tests, unit tests etc. and monitor metrics to decide whether to deploy to your next environment or rollback. This is just a structured way of building these CD pipelines you could also use Jenkins or Google started building something called Skaffold for Google Cloud Deploy: https://skaffold.dev/docs/ https://skaffold.dev/docs/ As your company gets larger you'll want to limit the blast radius of new changes so that you can meet your SLAs and you'll want to use multiple AZs and cloud regions for HA. That's where the idea of waves and cells comes from.

birdymcbird 4y ago

thats true. even if not amazon scale.. lets say you building multi tenant SaaS product. cell-architecture provide strong tenant isolation avoid noisy neighbor problem. other benefits like deploy changes to one cell so tenant may request delaying upgrade if they need to do pre-req work before hand. for ordinary service use case or web site.. most probability do not need this. over engineering infrastructure when should focus on customer and problem domain..thing that increase top or bottom line. source: principal SDE who launched several services at amazon using cell architecture.

ygjb 4y ago

Not much. Having a function build pipeline takes a bit of work to get set up, but once it's up and running, and projects are built to depend on it, it takes away a fair bit of ops work. That can help smaller teams become more agile, and being able to walk through a consistent process across multiple products for each step from commit against a dev branch through deployment in production makes it easier to account for team members being away, training new folks, etc. This is especially important for "side projects" or other small services that end up becoming critical infrastructure but are really maintained by one or two team members and only have a couple of internal customers.

vishnugupta 4y ago

The very first line of the article[1] calls it out, "....for enterprise-grade deployment pipelines." So I guess it's relevant for Fortune 500ish companies. That said, a base version of this pipeline is very useful even for a small startups. I concede that it takes a bit of time (took me about a week) to setup end to end. By end to end I mean beginning with empty AWS account to setting up VPC, ECS, CloudFront, Fargate, Codepipeline and so on. But once set you don't have to touch it for months on end. Scaling up/down, if needed, is as simple as minor config change. [1] https://aws.amazon.com/blogs/aws/new_deployment_pipelines_reference_architecture_and_-reference_implementations/ https://aws.amazon.com/blogs/aws/new_deployment_pipelines_re...