5 ms·

Great writeup. How do you handle database migrations when using an otherwise automated CI/CD flow with gradual deployment?

rkwz 5y ago
You can use any normal DB migration tool. For k8s, I put the app's readiness probe to false, run the migrations and then toggle the probe back to true. Here are some migration libraries: Go - https://github.com/golang-migrate/migrate https://github.com/golang-migrate/migrate Node - https://github.com/salsita/node-pg-migrate https://github.com/salsita/node-pg-migrate
- slt2021 5y ago
  I wonder what happens during blue green or canary deployment? if your migration changes database schema in a way that affect previous version negatively? is it even possible to do blue/green deploy if your schema changes radically?
  - rkwz 5y ago
    It's definitely possible but we need to spend sometime to make sure the changes are backwards compatible and gradually rollout the destructive change. Here's a really good guide on how to do this - https://docs.gitlab.com/ee/development/avoiding_downtime_in_migrations.html https://docs.gitlab.com/ee/development/avoiding_downtime_in_...
    - slt2021 5y ago
      thank you very much! this is a treasure find for me
  - Mavvie 5y ago
    It's fine if you make the changes backwards compatible a version. And in general, any change can be done in a backwards compatible way (although it can be a PITA)
kparaju 5y ago
You could probably do something like (for django): python manage.py migrate on pod startup. If there are no changes, it'll do nothing. If there are changes, it'll do the migration.
- ogjunkyard 5y ago
  Yeah, it's basically this. I'm running this as an initContainer for my K8s-based deployments. Took me a bit to get everything going, but my stack is pretty much similar to OP's article, although not quite as advanced in the automated-deployment of containers and monitoring. I'm not at a position where usage needs heavy monitoring because I'm still in the pre-launch phase of things and I'm using this side project to learn stuff I've yet to get experience with at multiple companies.
wakatime 5y ago
He's using Django, so most likely Django Migrations which is built into that framework. If you're using Flask, you're probably using Alembic with SQLAlchemy. Those are the two main ways to handle schema migrations in Python.

rkwz 5y ago

You can use any normal DB migration tool. For k8s, I put the app's readiness probe to false, run the migrations and then toggle the probe back to true. Here are some migration libraries: Go - https://github.com/golang-migrate/migrate https://github.com/golang-migrate/migrate Node - https://github.com/salsita/node-pg-migrate https://github.com/salsita/node-pg-migrate

slt2021 5y ago
I wonder what happens during blue green or canary deployment? if your migration changes database schema in a way that affect previous version negatively? is it even possible to do blue/green deploy if your schema changes radically?
- rkwz 5y ago
  It's definitely possible but we need to spend sometime to make sure the changes are backwards compatible and gradually rollout the destructive change. Here's a really good guide on how to do this - https://docs.gitlab.com/ee/development/avoiding_downtime_in_migrations.html https://docs.gitlab.com/ee/development/avoiding_downtime_in_...
  - slt2021 5y ago
    thank you very much! this is a treasure find for me
- Mavvie 5y ago
  It's fine if you make the changes backwards compatible a version. And in general, any change can be done in a backwards compatible way (although it can be a PITA)

kparaju 5y ago

You could probably do something like (for django): python manage.py migrate on pod startup. If there are no changes, it'll do nothing. If there are changes, it'll do the migration.

ogjunkyard 5y ago
Yeah, it's basically this. I'm running this as an initContainer for my K8s-based deployments. Took me a bit to get everything going, but my stack is pretty much similar to OP's article, although not quite as advanced in the automated-deployment of containers and monitoring. I'm not at a position where usage needs heavy monitoring because I'm still in the pre-launch phase of things and I'm using this side project to learn stuff I've yet to get experience with at multiple companies.

wakatime 5y ago

He's using Django, so most likely Django Migrations which is built into that framework. If you're using Flask, you're probably using Alembic with SQLAlchemy. Those are the two main ways to handle schema migrations in Python.