5 ms·

The problem is when your deployment depends on external state like a database. Code rollbacks are trivial, rolling back state (if you even can) is not.

by ljm 3y ago

mcny 3y ago
I always say this - there is no such thing as a rollback. Anyone who pretends you can simply "roll back" a stateful application is out of their mind. If you can roll back and forward events, good for you but for most of us rollbacks are actually a new build, deployment, and a new set of tests. Every single time.
- perrygeo 3y ago
  The system can only move forwards. Just like a `git revert`, it's a new commit to an immutable history. This is why I've never used or understood the value of the "downgrade" feature in some database migration tools. If you need to revert, make a _new_ migration that fixes the problem. Your tooling/code/logs should reflect the true history of the system without cooking the books.
  - OJFord 3y ago
    In what sense is 'downgrade' not just that prepared ahead of time? Or do you just not like the name? Of course usually it would be followed by something that fixes it 'properly' - applying the original 'upgrade' again but with some correction, but typically that'll take longer than reverting to the previous version, and given it's gone wrong you'll probably want to take longer with it, test more, restore confidence.
    - perrygeo 3y ago
      downgrade implies that such a thing is possible. In a stateful system though, you cannot take back that which you have committed. It's quite literally impossible by definition.
      - OJFord 3y ago
        No it isn't? Forward migration: add date column, set = date(day, month, year) for existing records, drop is valid date constraint, drop day/month/year columns Down migration: add day/month/year columns, add is valid date constraint, set day/month/year = appropriate datepart of date, drop date column for trivial example.
  - tianzhou 3y ago
    >> This is why I've never used or understood the value of the "downgrade" feature in some database migration tools It's technically a roll forward, while to most users it's more nature to say rollback/revert
    - grecy 3y ago
      It’s moving forward to get back to where you started
  - nicoburns 3y ago
    I’ve found “downgrade” mostly useful for quick iteration during local development (where you may well not care about lost data). Realise your initial cut of the migration wasn’t quite right? Run “down” then fix it, then “up” again.
chaxor 3y ago
If a git commit hash was directly tied to a data hash at that state (IPFS), that would be trivial.
- tbrownaw 3y ago
  Or have the SAN make a snapshot of your db before the deploy, no fancy solution-looking-for-a-problem tech needed. Completely reverting all the state to some point in the past is easy and have been a solved problem for quite a while. The actual problem is undoing states changes related to the code change, while not losing state changes related to normal system activity. If I add some new fields to how my web forum records posts, and then find out that it's eating every tenth post and need to revert, it'd be good to not lose and posts made in the meantime.
  - chaxor 3y ago
    Good point, this system is much more useful for scientists and ML researchers, etc. wherein the code may generate datasets over several weeks or months of compute time, but produce some output that should be directly tied to the code. You're right though. Systems in the social media space dealing with users appending to a database typically don't have this requirement to be easily reproducible, so the local snapshots are the right choice.
- baq 3y ago
  And useless. Unless customers don’t care about their data.
  - chaxor 3y ago
    I don't understand your statement. If a git clone of a specific commit also pulls the data that is generated from that commit hash via IPFS simultaneously, then you can always have the data at whatever state you choose. How might this have a negative effect on conserving the data? Do you mean that there are additional structures required to manage keys if the data is encrypted?
- rjmunro 3y ago
dmoy 3y ago
Sure, that is still a problem. But that problem exists whether or not your flags/configs are pushed atomically with your code/binary changes. Having flags push separately adds another problem on top of data corruption.
mrweasel 3y ago
One of the very best development teams I worked with had an interesting take, they always did database migrations first. Any new state that was to be added to the system could only be done so by first adding the new database fields or tables. This ensure that version 1 of the code would work with version 1 and 2 of the database. They would then roll out version 2 of the code, but have the new features hidden behind a feature flags (in the database), ensuring that version 2 could run, without using the new database schema. Once they where confident that everything was still running on version 2 of the code and database they'd enable the new feature. Later the feature flag could be migrated from the database to a properties file, saving the database lookup. I wouldn't necessarily call this approach simple, but it was incredibly safe and rollbacks was always a none event.
- steffres 3y ago
  That sounds reasonable. But what about the case where the DB migration of version 2 would be incompatbile with code version 1, e.g. a column was dropped?
  - jaimebuelta 3y ago
    You NEVER do that in one go, you need to split it in several deployments. Dropping a column is relatively straightforward, in two steps. First deploy a version of the code that doesn’t use the column, then release the migration dropping the column. The typical example is the renaming of a column, which needs to be done in several steps: 1. Create the new column, copying the data from the old column (DB migration) both columns exist, but only the old one is used 2. Deploy new code that works with both columns, reading from old and writing new and old 3. Deploy data migration (DB migration) that ensures old and new columns has the same values (to ensure data consistency). At this point, there are no “old column only” writes by the code deployed in previous step 4. Deploy new code using only new column. Old column is deprecated 5. Delete old column At any given point, code versions N (current) and N-1 (previous) are compatible with the DB. Any change on the DB is done in advance in a backwards compatible way.
    - 3y ago

The problem is when your deployment depends on external state like a database. Code rollbacks are trivial, rolling back state (if you even can) is not.

by ljm 3y ago

The problem is when your deployment depends on external state like a database. Code rollbacks are trivial, rolling back state (if you even can) is not.

mcny 3y ago

I always say this - there is no such thing as a rollback. Anyone who pretends you can simply "roll back" a stateful application is out of their mind. If you can roll back and forward events, good for you but for most of us rollbacks are actually a new build, deployment, and a new set of tests. Every single time.

perrygeo 3y ago
The system can only move forwards. Just like a `git revert`, it's a new commit to an immutable history. This is why I've never used or understood the value of the "downgrade" feature in some database migration tools. If you need to revert, make a _new_ migration that fixes the problem. Your tooling/code/logs should reflect the true history of the system without cooking the books.
- OJFord 3y ago
  In what sense is 'downgrade' not just that prepared ahead of time? Or do you just not like the name? Of course usually it would be followed by something that fixes it 'properly' - applying the original 'upgrade' again but with some correction, but typically that'll take longer than reverting to the previous version, and given it's gone wrong you'll probably want to take longer with it, test more, restore confidence.
  - perrygeo 3y ago
    downgrade implies that such a thing is possible. In a stateful system though, you cannot take back that which you have committed. It's quite literally impossible by definition.
    - OJFord 3y ago
      No it isn't? Forward migration: add date column, set = date(day, month, year) for existing records, drop is valid date constraint, drop day/month/year columns Down migration: add day/month/year columns, add is valid date constraint, set day/month/year = appropriate datepart of date, drop date column for trivial example.
- tianzhou 3y ago
  >> This is why I've never used or understood the value of the "downgrade" feature in some database migration tools It's technically a roll forward, while to most users it's more nature to say rollback/revert
  - grecy 3y ago
    It’s moving forward to get back to where you started
- nicoburns 3y ago
  I’ve found “downgrade” mostly useful for quick iteration during local development (where you may well not care about lost data). Realise your initial cut of the migration wasn’t quite right? Run “down” then fix it, then “up” again.

chaxor 3y ago

If a git commit hash was directly tied to a data hash at that state (IPFS), that would be trivial.

tbrownaw 3y ago
Or have the SAN make a snapshot of your db before the deploy, no fancy solution-looking-for-a-problem tech needed. Completely reverting all the state to some point in the past is easy and have been a solved problem for quite a while. The actual problem is undoing states changes related to the code change, while not losing state changes related to normal system activity. If I add some new fields to how my web forum records posts, and then find out that it's eating every tenth post and need to revert, it'd be good to not lose and posts made in the meantime.
- chaxor 3y ago
  Good point, this system is much more useful for scientists and ML researchers, etc. wherein the code may generate datasets over several weeks or months of compute time, but produce some output that should be directly tied to the code. You're right though. Systems in the social media space dealing with users appending to a database typically don't have this requirement to be easily reproducible, so the local snapshots are the right choice.
baq 3y ago
And useless. Unless customers don’t care about their data.
- chaxor 3y ago
  I don't understand your statement. If a git clone of a specific commit also pulls the data that is generated from that commit hash via IPFS simultaneously, then you can always have the data at whatever state you choose. How might this have a negative effect on conserving the data? Do you mean that there are additional structures required to manage keys if the data is encrypted?
rjmunro 3y ago

dmoy 3y ago

Sure, that is still a problem. But that problem exists whether or not your flags/configs are pushed atomically with your code/binary changes. Having flags push separately adds another problem on top of data corruption.

mrweasel 3y ago

One of the very best development teams I worked with had an interesting take, they always did database migrations first. Any new state that was to be added to the system could only be done so by first adding the new database fields or tables. This ensure that version 1 of the code would work with version 1 and 2 of the database. They would then roll out version 2 of the code, but have the new features hidden behind a feature flags (in the database), ensuring that version 2 could run, without using the new database schema. Once they where confident that everything was still running on version 2 of the code and database they'd enable the new feature. Later the feature flag could be migrated from the database to a properties file, saving the database lookup. I wouldn't necessarily call this approach simple, but it was incredibly safe and rollbacks was always a none event.

steffres 3y ago
That sounds reasonable. But what about the case where the DB migration of version 2 would be incompatbile with code version 1, e.g. a column was dropped?
- jaimebuelta 3y ago
  You NEVER do that in one go, you need to split it in several deployments. Dropping a column is relatively straightforward, in two steps. First deploy a version of the code that doesn’t use the column, then release the migration dropping the column. The typical example is the renaming of a column, which needs to be done in several steps: 1. Create the new column, copying the data from the old column (DB migration) both columns exist, but only the old one is used 2. Deploy new code that works with both columns, reading from old and writing new and old 3. Deploy data migration (DB migration) that ensures old and new columns has the same values (to ensure data consistency). At this point, there are no “old column only” writes by the code deployed in previous step 4. Deploy new code using only new column. Old column is deprecated 5. Delete old column At any given point, code versions N (current) and N-1 (previous) are compatible with the DB. Any change on the DB is done in advance in a backwards compatible way.
  - 3y ago