4 ms·

Largely I don't think it's worth moving off EB except in specific scenarios. I'm a Rails monolith and DB migrations are not well suited in ElasticBeanstalk whe

Largely I don't think it's worth moving off EB except in specific scenarios. I'm a Rails monolith and DB migrations are not well suited in ElasticBeanstalk when running docker containers. There's no way to run a single container that 'completes' and then returns whether it was successful or not. Currently I have a separate environment that holds a single EC2 (lots of idle time) to run migrations. It fires and the deployment process moves on. I have to hope the migration finishes before the next set of containers gets rotated out, 2-3 minutes. That's not a good policy to run a prod env by so I'm switching to Fargate where I can run a One Off task and poll for the result. Other benefits include not paying for that dedicated EC2 idle time. Not worrying about EC2 management. Direct access to parameter store and secrets (EB can't do secrets from Dockerrun.aws.json yet =( Drawbacks - I lose 'Rolling Update/Deployments based on Health' and HealthD. Another big one, I lose access to the container unless you run sshd itself (don't do it). I am running both environments side by side to evaluate but realistically the migration requirement is going to push me to a full Fargate. I could only run the migration step in Fargate while the rest in EB - best of both worlds? But somehow that feels dirty... we'll see what I end up doing.

ramraj07 6y ago
I might not know what I'm talking about (our deployments don't do automatic migrations, for eg), but if I understand correctly, your problem is to synchronize migrations with deployments, is that accurate? Couldn't you use something like GitHub Actions to run migrations and then deploy to EB?
- irjustin 6y ago
  Yeah the goal is to get the steps to be: Build, Deploy & Run Migrations, Deploy & Rotate containers. Github Actions is CI/CD and it can be used to kick off any deploy step and manage it, but it's not one to actually perform the migration. That would be like Github Actions actually running the webserver directly. In this example Github Actions needs to kick off a `eb deploy`, which it can totally do (I personally use CircleCI) but `eb deploy` (elastic beanstalk in general) is not designed to run a short lived script on a single container AND wait for it to return. You can run a short lived script no problem, but you have no idea if it was successful or not because once the container becomes 'healthy' the command completes. What I'm really waiting for is the container.lastStatus:'STOPPED' and exitCode: 0. Can't do that on EB.
jpp 6y ago
We have a similar setup — containers managed by Fargate — and solved the issue of migrations by: 1) having our app containers include the DB migration logic 2) on container startup, “check and run migrations” before app startup, 3) the trick: acquire a lock in postgres as part of step 2, so that only one node at a time can run migrations. Migrations are run inside of a begin/commit, so with the lock we have reliable guarantees that 1) exactly one container at a time can try to run the migration; 2) the migration either completes or fails. This setup has the benefit too that, if the migration fails for whatever reason, no new app containers will start in production. That is: we can basically trigger a production deploy and if it fails, the deploy halts and the previous app version remains up and serving traffic. There are better ways of handling this, but for us at our scale, this has been both very simple and very reliable, which makes me happy. :)
- nthj 6y ago
  As I understand it, the concurrency-safe lock is built into Rails for a while. Step 3 shouldn’t be necessary. https://github.com/rails/rails/pull/22122 https://github.com/rails/rails/pull/22122
  - jpp 6y ago
    Ah, very nice! Our setup is on python, and because of some other dev tooling we have, it’s very easy for us to have a simple decorator function that grabs the advisory lock. We can apply that decorator to any python function, including the one that triggers our migrations. If on rails, seems like using the above link would be better.

ramraj07 6y ago

I might not know what I'm talking about (our deployments don't do automatic migrations, for eg), but if I understand correctly, your problem is to synchronize migrations with deployments, is that accurate? Couldn't you use something like GitHub Actions to run migrations and then deploy to EB?

irjustin 6y ago
Yeah the goal is to get the steps to be: Build, Deploy & Run Migrations, Deploy & Rotate containers. Github Actions is CI/CD and it can be used to kick off any deploy step and manage it, but it's not one to actually perform the migration. That would be like Github Actions actually running the webserver directly. In this example Github Actions needs to kick off a `eb deploy`, which it can totally do (I personally use CircleCI) but `eb deploy` (elastic beanstalk in general) is not designed to run a short lived script on a single container AND wait for it to return. You can run a short lived script no problem, but you have no idea if it was successful or not because once the container becomes 'healthy' the command completes. What I'm really waiting for is the container.lastStatus:'STOPPED' and exitCode: 0. Can't do that on EB.

jpp 6y ago

We have a similar setup — containers managed by Fargate — and solved the issue of migrations by: 1) having our app containers include the DB migration logic 2) on container startup, “check and run migrations” before app startup, 3) the trick: acquire a lock in postgres as part of step 2, so that only one node at a time can run migrations. Migrations are run inside of a begin/commit, so with the lock we have reliable guarantees that 1) exactly one container at a time can try to run the migration; 2) the migration either completes or fails. This setup has the benefit too that, if the migration fails for whatever reason, no new app containers will start in production. That is: we can basically trigger a production deploy and if it fails, the deploy halts and the previous app version remains up and serving traffic. There are better ways of handling this, but for us at our scale, this has been both very simple and very reliable, which makes me happy. :)

nthj 6y ago
As I understand it, the concurrency-safe lock is built into Rails for a while. Step 3 shouldn’t be necessary. https://github.com/rails/rails/pull/22122 https://github.com/rails/rails/pull/22122
- jpp 6y ago
  Ah, very nice! Our setup is on python, and because of some other dev tooling we have, it’s very easy for us to have a simple decorator function that grabs the advisory lock. We can apply that decorator to any python function, including the one that triggers our migrations. If on rails, seems like using the above link would be better.