5 ms·

Nice approach! "reload" was a nice touch to learn. What I miss usually are two parts of deploys that are often ignored, and are critical in production environm

Nice approach! "reload" was a nice touch to learn. What I miss usually are two parts of deploys that are often ignored, and are critical in production environments: 1. Revert/rollback to older versions. We're humans and despite having all the processes sometimes murphy's law applies on moderate to complex server setups. 2. There is git/svn for code tracking, but even more important is the database consistency. Versioned backups and restores should be also part of the whole setup. I'm currently looking into having the full setup with no downtime with open ears.

irremediable 11y ago
Any advice about versioned database backups?
- smt88 11y ago
  In my opinion, this is one of the major unsolved (or under-solved) issues in web development. You have to write and test migrations, and then you have to make sure the migrations are tied to your deployment/rollback process. If there's a service/tool that automates a lot of this or makes it safer, I'd be really happy to learn about it!
  - UK-AL 11y ago
    When I used to use django ages ago, there used to be Django south. I think its built in now.
    - metachris 11y ago
      Yes, its built in now: https://docs.djangoproject.com/en/1.8/topics/migrations/ https://docs.djangoproject.com/en/1.8/topics/migrations/ Of course a database rollback may loose information from any newly created fields.
      - jeffasinger 11y ago
        So if you're aiming for zero downtime rollbacks with Django, here's how adding a new field to a model might work: 1) Add a Migration that adds the new field, but allows null. Deploy. 2) Add the field to the model. Also make sure it sets it's default on write. Deploy. 3) Execute a background task to set the field to whatever it's default value should be. 4) Add migrations to enforce integrity and add indexes. Deploy. 5) Actually deploy code that needs the new field. Yes, this is super annoying, no, most people don't do this. Separating out #1 and #2 means you can always roll back all the way to right after #1 without losing any data. An extra, nullable field on a model with no indices on it shouldn't hurt anything.
        micah_chatt 11y ago
        Totally agree. I've been advocating this for months but the ease/simplicity of simply restarting the server with new code has won out for now.
- danialtz 11y ago
  I'm planning to do something like this workflow most probably reinventing the wheel along software history: - Every release is already tagged, and merged into master (git-flow). - There is a post-commit hook to backup the database with the tag and time. So I know which database tag is newer. - Store the encrypted zip on S3, since each developer can deploy. - Deploy, like cuu508 did. The problem is with restore. If there is no change in database it is more or less straight forward to download and import the db with the n-1 code tag. But if there are any inserts in database then a migration back should be applied to the latest stand of the database. I can imagine it's going to be a hefty workflow to do.
  - mateuszf 11y ago
    How about .. have one db with log of events (cqrs style), the other one is transactional and transactions are being performed based on ordered events. When deploying new version - backup the transactional db and keep a pointer to the last event in events db. Keep the events db running. If restoring - restore transactional db backend based on backup and "play back" all events from the events db. The distinction can also be made using one db and backing up / restoring subset of the tables.
    - josegonzalez 11y ago
      So basically kafka + another layer for database migrations?
- joeyspn 11y ago
  In docker-land you've got Flocker. I guess you could version the container with tags... https://clusterhq.com/ https://clusterhq.com/
deleted 11y ago
[deleted]

irremediable 11y ago

Any advice about versioned database backups?

smt88 11y ago
In my opinion, this is one of the major unsolved (or under-solved) issues in web development. You have to write and test migrations, and then you have to make sure the migrations are tied to your deployment/rollback process. If there's a service/tool that automates a lot of this or makes it safer, I'd be really happy to learn about it!
- UK-AL 11y ago
  When I used to use django ages ago, there used to be Django south. I think its built in now.
  - metachris 11y ago
    Yes, its built in now: https://docs.djangoproject.com/en/1.8/topics/migrations/ https://docs.djangoproject.com/en/1.8/topics/migrations/ Of course a database rollback may loose information from any newly created fields.
    - jeffasinger 11y ago
      So if you're aiming for zero downtime rollbacks with Django, here's how adding a new field to a model might work: 1) Add a Migration that adds the new field, but allows null. Deploy. 2) Add the field to the model. Also make sure it sets it's default on write. Deploy. 3) Execute a background task to set the field to whatever it's default value should be. 4) Add migrations to enforce integrity and add indexes. Deploy. 5) Actually deploy code that needs the new field. Yes, this is super annoying, no, most people don't do this. Separating out #1 and #2 means you can always roll back all the way to right after #1 without losing any data. An extra, nullable field on a model with no indices on it shouldn't hurt anything.
      - micah_chatt 11y ago
        Totally agree. I've been advocating this for months but the ease/simplicity of simply restarting the server with new code has won out for now.
danialtz 11y ago
I'm planning to do something like this workflow most probably reinventing the wheel along software history: - Every release is already tagged, and merged into master (git-flow). - There is a post-commit hook to backup the database with the tag and time. So I know which database tag is newer. - Store the encrypted zip on S3, since each developer can deploy. - Deploy, like cuu508 did. The problem is with restore. If there is no change in database it is more or less straight forward to download and import the db with the n-1 code tag. But if there are any inserts in database then a migration back should be applied to the latest stand of the database. I can imagine it's going to be a hefty workflow to do.
- mateuszf 11y ago
  How about .. have one db with log of events (cqrs style), the other one is transactional and transactions are being performed based on ordered events. When deploying new version - backup the transactional db and keep a pointer to the last event in events db. Keep the events db running. If restoring - restore transactional db backend based on backup and "play back" all events from the events db. The distinction can also be made using one db and backing up / restoring subset of the tables.
  - josegonzalez 11y ago
    So basically kafka + another layer for database migrations?
joeyspn 11y ago
In docker-land you've got Flocker. I guess you could version the container with tags... https://clusterhq.com/ https://clusterhq.com/

deleted 11y ago

[deleted]