12 ms·

I am in the process of migrating my stack from Elastic Beanstalk multi-container to Fargate so this looked like an interesting thing I could 'pick up'. This do

I am in the process of migrating my stack from Elastic Beanstalk multi-container to Fargate so this looked like an interesting thing I could 'pick up'. This does potentially unify the container definitions between compose and ContainerDefinitions in the task definitions, but for my self, that's not a super helpful tool. Much of the complexity of running Fargate is outside of Fargate, wiring everything up so Route 53 => CloudFront + WAFv2 => ALB => TargetGroup => Fargate w/ security groups, subnets underpinning it all. I can't recommend looking at this seriously as something to run in production. CloudFormation/Terraform is still the best place to sink your time.

mcintyre1994 6y ago
We’re thinking about doing a similar migration. One of my concerns is that the current Beanstalk infra isn’t described in code, and when I’ve experimented a little with Beanstalk in CloudFormation it’s been horrible to write - for something a lot simpler. I obviously want us to move to using CloudFormation or something like it if we migrate, how good or bad would you say using it (or Terraform if that’s what you settled on) for Fargate is?
- ramraj07 6y ago
  Have you considered experimenting with AWS CDK? The code looked far more elegant than cloudformation
  - mcintyre1994 6y ago
    I have done a tiny bit, probably worth trying to move some smaller CFN templates over to it in earnest to try it out a bit more. I can definitely see it being nicer for things like Beanstalk/Fargate.
- irjustin 6y ago
  If you're looking for confidence in infrastructure changes then I 100% recommend fighting through the 'horrible to write' CloudFormation/Terraform. CloudF is infra as yaml and Terra is as code. CloudF also has the CDK which can use TypeScript and be infra as code that 'transpiles' down to yaml. All options feel terrible. It's like learning a whole new programming language that takes 10 minutes to see if your code runs or which part got stuck. But once you've got your stack working, changes are a breeze and can be done with way way more confidence. Spinning up clone environments is barely any work. Is it terrible? Absolutely. Is it worth it? Again - absolutely. If I had to do it again, I would pick the CDK (more powerful), but the fight would've taken longer.
  - mcintyre1994 6y ago
    Yep I definitely agree with the importance, it’s probably the most impactful technical debt we have. Clone environments is a big driving force for it too. I guess what I was hoping is that Beanstalk wasn’t designed with infra as code in mind, but things like Fargate were, and somehow it had become less horrible. Maybe that’s just never going to happen though and it’s the nature of the problem. Appreciate the pointer toward CDK too, I have used it a tiny bit and it definitely seemed worth exploring further.
    - irjustin 6y ago
      The code differences between ElasticBeanstalk and Fargate are not very far, but JUST far enough to be annoying. Because you've worked with Beanstalk in the past, I would personally stick with it unless I'm looking to fight 2 issues at the same time - Learning Fargate & Learning CDK/CloudF/Terraform. Fargate is not a panacea - Elastic Beanstalk does have very cool options/features.
      - mcintyre1994 6y ago
        I really appreciate the insight here - thanks a lot :)
ramraj07 6y ago
Could you comment on why you are moving away from elastic beanstalk? We manage a fairly simple EB deployment but it seems perfect for our infra needs, and I'm not able to imagine why we might need to scale out of it - if any other services are required, I'm tempted to only launch multiple EB deployments instead of going the kubernetes/fargate route.
- irjustin 6y ago
  Largely I don't think it's worth moving off EB except in specific scenarios. I'm a Rails monolith and DB migrations are not well suited in ElasticBeanstalk when running docker containers. There's no way to run a single container that 'completes' and then returns whether it was successful or not. Currently I have a separate environment that holds a single EC2 (lots of idle time) to run migrations. It fires and the deployment process moves on. I have to hope the migration finishes before the next set of containers gets rotated out, 2-3 minutes. That's not a good policy to run a prod env by so I'm switching to Fargate where I can run a One Off task and poll for the result. Other benefits include not paying for that dedicated EC2 idle time. Not worrying about EC2 management. Direct access to parameter store and secrets (EB can't do secrets from Dockerrun.aws.json yet =( Drawbacks - I lose 'Rolling Update/Deployments based on Health' and HealthD. Another big one, I lose access to the container unless you run sshd itself (don't do it). I am running both environments side by side to evaluate but realistically the migration requirement is going to push me to a full Fargate. I could only run the migration step in Fargate while the rest in EB - best of both worlds? But somehow that feels dirty... we'll see what I end up doing.
  - ramraj07 6y ago
    I might not know what I'm talking about (our deployments don't do automatic migrations, for eg), but if I understand correctly, your problem is to synchronize migrations with deployments, is that accurate? Couldn't you use something like GitHub Actions to run migrations and then deploy to EB?
    - irjustin 6y ago
imsodrunklol 6y ago
I just want to mention that you can output the CloudFormation template that it parses out by saying `docker ecs compose convert`, which is always handy. You also could use other flags to help with joining clusters, services ,etc. Not saying it's a complete production ready too, but it is nice tool to help speed up the process. https://docs.docker.com/engine/context/ecs-integration/ https://docs.docker.com/engine/context/ecs-integration/
nostrebored 6y ago
I highly recommend the CDK here. A setup very similar to yours is just a few lines of code which you can synth into cloudformation templates
- 5h 6y ago
  I picked up CDK & built out the same essential setup + DB, an EC2-ECS instance for ad-hoc stuff and a bunch of things to appease Config & Security Hub in ~420 lines of quite sparse python, that builds out to ~3000 lines of cloudformation yaml. I feel CDK has the mix between optional granularity & high-level constructs just right, while this plugin looks nice for a quick MVP standup i'd be surprised to see it in use for production workloads
dkarp 6y ago
As an alternative to Terraform, we've had great experience using Pulumi for exactly this stack. Writing Pulumi config in Python like the rest of our code has been great.
kristianpaul 6y ago
Give a Try to CDK constructs for fargate (ecs-patterns) and Assets for registering Dockerfiles as ECR repos
indemnity 6y ago
I just spent this weekend setting this up to learn a bit of AWS for a toy project. I thought I would "just" quickly drop a Rust API server image in an ECS cluster. By the end of the weekend, I had the architecture you describe. - Route53 alias A record -> ALB DNS name - LetsEncrypt cert in IAM - ALB listener doing SSL termination using the cert -> forwarding to target group - ALB listener doing 80->443 redirect - Security group on ALB listener allowing only approved IP ranges in (not ready for this thing to be public yet) - Security group on ECS service only allowing ALB to connect - ECS cluster using Fargate - RDS instance only allowing ECS service to connect - CloudWatch log group for the container logs - Subnets - Secrets Manager for pulling Docker images from private GitHub Packages repo Did it all in Terraform, and then added GitHub Actions to the Terraform repository to do terraform validate on PR and terraform plan && terraform apply -auto-approve on merge. Then, yesterday, hooked up GitHub Actions on the Rust API server repository build a version tagged image and publish it to GitHub packages, create a PR in the Terraform repository to update the ECS task definition for the new image, and if it passes the PR checks, automatically merge it (which triggers the Terraform plan/apply run). It did seem complex the first time I did it, but looking back over both the AWS and GitHub Actions configuration, I wouldn't change too much. I feel fairly confident this is secure, and I understand most of the configuration options and why they are there. Something that "simplified" it for me would just become a straightjacket as I get more proficient with AWS. IaC 101 I guess, but I was chuffed when the Rube Goldberg machine whirred away after making a code commit to the Rust repo, and two minutes later my new code was running on ECS :) Considering writing up a blog post about it just to firm up my own understanding as well...
- mcintyre1994 6y ago
  Would definitely be interested in that blog if you write one :)
- yjftsjthsd-h 6y ago
  > Considering writing up a blog post about it just to firm up my own understanding as well... Please do; your setup looks quite nice:) One nit/question: > LetsEncrypt cert in IAM Why LetsEncrypt? AWS Certificate Manager is already free and integrated, IIRC
oh_hello 6y ago
I also often look at these tools and don't see clearly how they are meant to scale beyond small, isolated projects. The simple examples where you let the tool take your small project and start serving requests from AWS doesn't show me how I would use it in the real world.

mcintyre1994 6y ago

We’re thinking about doing a similar migration. One of my concerns is that the current Beanstalk infra isn’t described in code, and when I’ve experimented a little with Beanstalk in CloudFormation it’s been horrible to write - for something a lot simpler. I obviously want us to move to using CloudFormation or something like it if we migrate, how good or bad would you say using it (or Terraform if that’s what you settled on) for Fargate is?

ramraj07 6y ago
Have you considered experimenting with AWS CDK? The code looked far more elegant than cloudformation
- mcintyre1994 6y ago
  I have done a tiny bit, probably worth trying to move some smaller CFN templates over to it in earnest to try it out a bit more. I can definitely see it being nicer for things like Beanstalk/Fargate.
irjustin 6y ago
If you're looking for confidence in infrastructure changes then I 100% recommend fighting through the 'horrible to write' CloudFormation/Terraform. CloudF is infra as yaml and Terra is as code. CloudF also has the CDK which can use TypeScript and be infra as code that 'transpiles' down to yaml. All options feel terrible. It's like learning a whole new programming language that takes 10 minutes to see if your code runs or which part got stuck. But once you've got your stack working, changes are a breeze and can be done with way way more confidence. Spinning up clone environments is barely any work. Is it terrible? Absolutely. Is it worth it? Again - absolutely. If I had to do it again, I would pick the CDK (more powerful), but the fight would've taken longer.
- mcintyre1994 6y ago
  Yep I definitely agree with the importance, it’s probably the most impactful technical debt we have. Clone environments is a big driving force for it too. I guess what I was hoping is that Beanstalk wasn’t designed with infra as code in mind, but things like Fargate were, and somehow it had become less horrible. Maybe that’s just never going to happen though and it’s the nature of the problem. Appreciate the pointer toward CDK too, I have used it a tiny bit and it definitely seemed worth exploring further.
  - irjustin 6y ago
    The code differences between ElasticBeanstalk and Fargate are not very far, but JUST far enough to be annoying. Because you've worked with Beanstalk in the past, I would personally stick with it unless I'm looking to fight 2 issues at the same time - Learning Fargate & Learning CDK/CloudF/Terraform. Fargate is not a panacea - Elastic Beanstalk does have very cool options/features.
    - mcintyre1994 6y ago
      I really appreciate the insight here - thanks a lot :)

ramraj07 6y ago

Could you comment on why you are moving away from elastic beanstalk? We manage a fairly simple EB deployment but it seems perfect for our infra needs, and I'm not able to imagine why we might need to scale out of it - if any other services are required, I'm tempted to only launch multiple EB deployments instead of going the kubernetes/fargate route.

irjustin 6y ago
Largely I don't think it's worth moving off EB except in specific scenarios. I'm a Rails monolith and DB migrations are not well suited in ElasticBeanstalk when running docker containers. There's no way to run a single container that 'completes' and then returns whether it was successful or not. Currently I have a separate environment that holds a single EC2 (lots of idle time) to run migrations. It fires and the deployment process moves on. I have to hope the migration finishes before the next set of containers gets rotated out, 2-3 minutes. That's not a good policy to run a prod env by so I'm switching to Fargate where I can run a One Off task and poll for the result. Other benefits include not paying for that dedicated EC2 idle time. Not worrying about EC2 management. Direct access to parameter store and secrets (EB can't do secrets from Dockerrun.aws.json yet =( Drawbacks - I lose 'Rolling Update/Deployments based on Health' and HealthD. Another big one, I lose access to the container unless you run sshd itself (don't do it). I am running both environments side by side to evaluate but realistically the migration requirement is going to push me to a full Fargate. I could only run the migration step in Fargate while the rest in EB - best of both worlds? But somehow that feels dirty... we'll see what I end up doing.
- ramraj07 6y ago
  I might not know what I'm talking about (our deployments don't do automatic migrations, for eg), but if I understand correctly, your problem is to synchronize migrations with deployments, is that accurate? Couldn't you use something like GitHub Actions to run migrations and then deploy to EB?
  - irjustin 6y ago

imsodrunklol 6y ago

I just want to mention that you can output the CloudFormation template that it parses out by saying `docker ecs compose convert`, which is always handy. You also could use other flags to help with joining clusters, services ,etc. Not saying it's a complete production ready too, but it is nice tool to help speed up the process. https://docs.docker.com/engine/context/ecs-integration/ https://docs.docker.com/engine/context/ecs-integration/

nostrebored 6y ago

I highly recommend the CDK here. A setup very similar to yours is just a few lines of code which you can synth into cloudformation templates

5h 6y ago
I picked up CDK & built out the same essential setup + DB, an EC2-ECS instance for ad-hoc stuff and a bunch of things to appease Config & Security Hub in ~420 lines of quite sparse python, that builds out to ~3000 lines of cloudformation yaml. I feel CDK has the mix between optional granularity & high-level constructs just right, while this plugin looks nice for a quick MVP standup i'd be surprised to see it in use for production workloads

dkarp 6y ago

As an alternative to Terraform, we've had great experience using Pulumi for exactly this stack. Writing Pulumi config in Python like the rest of our code has been great.

kristianpaul 6y ago

Give a Try to CDK constructs for fargate (ecs-patterns) and Assets for registering Dockerfiles as ECR repos

indemnity 6y ago

I just spent this weekend setting this up to learn a bit of AWS for a toy project. I thought I would "just" quickly drop a Rust API server image in an ECS cluster. By the end of the weekend, I had the architecture you describe. - Route53 alias A record -> ALB DNS name - LetsEncrypt cert in IAM - ALB listener doing SSL termination using the cert -> forwarding to target group - ALB listener doing 80->443 redirect - Security group on ALB listener allowing only approved IP ranges in (not ready for this thing to be public yet) - Security group on ECS service only allowing ALB to connect - ECS cluster using Fargate - RDS instance only allowing ECS service to connect - CloudWatch log group for the container logs - Subnets - Secrets Manager for pulling Docker images from private GitHub Packages repo Did it all in Terraform, and then added GitHub Actions to the Terraform repository to do terraform validate on PR and terraform plan && terraform apply -auto-approve on merge. Then, yesterday, hooked up GitHub Actions on the Rust API server repository build a version tagged image and publish it to GitHub packages, create a PR in the Terraform repository to update the ECS task definition for the new image, and if it passes the PR checks, automatically merge it (which triggers the Terraform plan/apply run). It did seem complex the first time I did it, but looking back over both the AWS and GitHub Actions configuration, I wouldn't change too much. I feel fairly confident this is secure, and I understand most of the configuration options and why they are there. Something that "simplified" it for me would just become a straightjacket as I get more proficient with AWS. IaC 101 I guess, but I was chuffed when the Rube Goldberg machine whirred away after making a code commit to the Rust repo, and two minutes later my new code was running on ECS :) Considering writing up a blog post about it just to firm up my own understanding as well...

mcintyre1994 6y ago
Would definitely be interested in that blog if you write one :)
yjftsjthsd-h 6y ago
> Considering writing up a blog post about it just to firm up my own understanding as well... Please do; your setup looks quite nice:) One nit/question: > LetsEncrypt cert in IAM Why LetsEncrypt? AWS Certificate Manager is already free and integrated, IIRC

oh_hello 6y ago

I also often look at these tools and don't see clearly how they are meant to scale beyond small, isolated projects. The simple examples where you let the tool take your small project and start serving requests from AWS doesn't show me how I would use it in the real world.