28 ms·

rust-query manages migrations and reads the schema from the database to check that it matches what was defined in the application. If at any point the database

by Hytak 2y ago

rust-query manages migrations and reads the schema from the database to check that it matches what was defined in the application. If at any point the database schema doesn't match the expected schema, then rust-query will panic with an error message explaining the difference (currently this error is not very pretty). Furthermore, at the start of every transaction, rust-query will check that the `schema_version` (sqlite pragma) did not change. (source: I am the author)

mjr00 2y ago
> rust-query manages migrations and reads the schema from the database to check that it matches what was defined in the application. If at any point the database schema doesn't match the expected schema, then rust-query will panic with an error message explaining the difference (currently this error is not very pretty). IMO - this sounds like "tell me you've never operated a real production system before without telling me you've never operated a real production system before." Shit happens in real life. Even if you have a great deployment pipeline, at some point, you'll need to add a missing index in production fast because a wave of users came in and revealed a shit query. Or your on-call DBA will need to modify a table over the weekend from i32 -> i64 because you ran out of primary key values, and you can't spend the time updating all your code. (in Rust this is dicier, of course, but with something like Python shouldn't cause issues in general.) Or you'll just need to run some operation out of band -- that is, not relying on a migration -- because it what makes sense. Great example is using something like pt-osc[0] to create a temporary table copy and add temporary triggers to an existing table in order to do a zero-downtime copy. Or maybe you just need to drop and recreate an index because it got corrupted. Shit happens! Anyway, I really wouldn't recommend a design that relies on your database always agreeing with your codebase 100% of the time. What you should strive for is your codebase being compatible with the database 100% of the time -- that means new columns get added with a default value (or NULL) so inserts work, you don't drop or rename columns or tables without a strict deprecation process (i.e. a rename is really add in db -> add writes to code -> backfill values in db -> remove from code -> remove from db), etc... But fundamentally panicking because a table has an extra column is crazy. How else would you add a column to a running production system? [0] https://docs.percona.com/percona-toolkit/pt-online-schema-change.html https://docs.percona.com/percona-toolkit/pt-online-schema-ch...
- threeseed 2y ago
  > Even if you have a great deployment pipeline, at some point, you'll need to add a missing index in production fast because a wave of users came in and revealed a shit query. This sounds more like a CI/CD and process issue. There is no reason why adding a new index in code and deploying it into Production should be more complex or error prone than modifying it on the database itself.
  - mjr00 2y ago
    Direct execution of `CREATE INDEX...` on a database table is always going to be faster than going through a normal deployment pipeline. Even if we assume your pipeline is really fast, which is probably not the case at most orgs, you are still comparing a single SQL statement execution, to a single SQL statement execution + git push + code reviews + merge + running through Jenkins/Circle/whatever. How long does that overhead take? How much money have you lost because your website won't load when your post is on the frontpage of HN? Seconds and minutes count. I don't want my code crashing because an unexpected index exists in this scenario.
    - threeseed 2y ago
      You should be able to deploy end to end to Production in less than a minute. Companies should be focused on solving that problem first before doing insanely short-sighted workarounds like skipping pushing to Git and code reviews.
      - mjr00 2y ago
        > You should be able to deploy end to end to Production in less than a minute. When I was at AWS (RDS) our end-to-end production deployment process was 7 days. We were also pulling $25million/day or so in profit. I'm sure that number is much higher now. There's a large difference between what the theoretical "right" thing is from a textbook perspective, and what successful engineering teams do in reality. edit: besides, it doesn't even make sense in this context. I have 100 servers talking to the database. I need to create an index, ok, add it to the code. Deploy to server 1. Server 1 adds the index as part of the migration process, and let's say it's instant-ish (not realistic but whatever). Do the other 99 servers now panic because there's an unexpected index on the table?
kelnos 2y ago
In addition to the deployment-time issues and other stuff I and others have commented downthread, I thought of another problem with this. I can't see how this would even work for trivial, quick, on-line schema changes. Let's say I have 10 servers running the same service that talks to the database (that is, the service fronting the database is scaled out horizontally). How would I do a migration? Obviously I can't deploy new code to all 10 servers simultaneously that will do the schema migration; only one server can run the migration. So one server runs the migration, and... what, the other 9 servers immediately panic because their idea of the schema is out of date? Or I deploy code to all 10 servers but somehow designate that only one of them will actually do the schema migration. Well, now the other 9 servers are expecting the new schema, and will panic before that 1 server can finish doing the migration. It seems to me that rust-query is only suitable for applications where you have to schedule downtime in order to do schema changes. That's just unacceptable for any business I've worked at.
- dayjah 2y ago
  This isn’t unique to rust-query; this problem also exists with ActiveRecord, for example. At Twitch we just had to really think about our migrations and write code to handle differences. Basically no free lunch!
- rendaw 2y ago
  I think first and foremost, if you're going to use a tool like this, you need to do everything through the tool. That said, for zero downtime migrations there are a number of techniques, but it typically boils down to splitting the migration into two steps where each step is rolled out to each server before starting the next: https://teamplify.com/blog/zero-downtime-DB-migrations/ https://teamplify.com/blog/zero-downtime-DB-migrations/ https://johnnymetz.com/posts/multistep-database-changes/ https://johnnymetz.com/posts/multistep-database-changes/ etc I'm not sure if there's anything that automates this, but it'd probably need to involve the infrastructure layer (like terraform) too. Edit: There's one other approach I've heard of for zero downtime deployments: Start running the new version in new instances/services parallel to the old version, but pause it before doing any database stuff. Drain client connections to the old version and queue them. Once drained, stop the old version, perform database migrations, and start the new version, then start consuming the queue. This is (I think) more general but you could get client timeouts or need to kill long requests to the old version, and requires coordination between infrastructure (load balancer?) and software versions.
Merad 2y ago
Unless you have some tricks up your sleeve that I'm not thinking of, an immediate consequence of this is that zero downtime deployments and blue/green deployments become impossible. Those both rely on your app being able to run in a state where the schema is not an exact match for what the app expects - but it's compatible so the app can still function.
- ComputerGuru 2y ago
  Semantic versioning?
  - Merad 2y ago
    If I understand the GP correctly, there's no notion of semver involved. Any difference in the schema results in a runtime error.
    - ComputerGuru 2y ago
      Yes, that’s what I gathered from the code. But I was proposing it as an “easy” solution that doesn’t involve throwing away OP’s main idea.
- Filligree 2y ago
  And that's okay. Most applications don't need zero-downtime deployments, and there are already plenty of APIs that support that use case. I'd rather have more like this one.
Filligree 2y ago
I'm so glad you made this. I've been searching for a decent Rust database library for half a year already, and this ticks all the boxes. I haven't tried it yet, so I might have to eat my words later, but- great job! It's going to save tons of effort.
theptip 2y ago
In all the systems I’ve built (mostly Django) you need to tolerate vN and vN+1 simultaneously; you are not going to turn off your app to upgrade the DB. You’ll have some Pods on the old application version while you do your gradual upgrade. How do you envision rolling upgrades working here?

by Hytak 2y ago

mjr00 2y ago

> rust-query manages migrations and reads the schema from the database to check that it matches what was defined in the application. If at any point the database schema doesn't match the expected schema, then rust-query will panic with an error message explaining the difference (currently this error is not very pretty). IMO - this sounds like "tell me you've never operated a real production system before without telling me you've never operated a real production system before." Shit happens in real life. Even if you have a great deployment pipeline, at some point, you'll need to add a missing index in production fast because a wave of users came in and revealed a shit query. Or your on-call DBA will need to modify a table over the weekend from i32 -> i64 because you ran out of primary key values, and you can't spend the time updating all your code. (in Rust this is dicier, of course, but with something like Python shouldn't cause issues in general.) Or you'll just need to run some operation out of band -- that is, not relying on a migration -- because it what makes sense. Great example is using something like pt-osc[0] to create a temporary table copy and add temporary triggers to an existing table in order to do a zero-downtime copy. Or maybe you just need to drop and recreate an index because it got corrupted. Shit happens! Anyway, I really wouldn't recommend a design that relies on your database always agreeing with your codebase 100% of the time. What you should strive for is your codebase being compatible with the database 100% of the time -- that means new columns get added with a default value (or NULL) so inserts work, you don't drop or rename columns or tables without a strict deprecation process (i.e. a rename is really add in db -> add writes to code -> backfill values in db -> remove from code -> remove from db), etc... But fundamentally panicking because a table has an extra column is crazy. How else would you add a column to a running production system? [0] https://docs.percona.com/percona-toolkit/pt-online-schema-change.html https://docs.percona.com/percona-toolkit/pt-online-schema-ch...

threeseed 2y ago
> Even if you have a great deployment pipeline, at some point, you'll need to add a missing index in production fast because a wave of users came in and revealed a shit query. This sounds more like a CI/CD and process issue. There is no reason why adding a new index in code and deploying it into Production should be more complex or error prone than modifying it on the database itself.
- mjr00 2y ago
  Direct execution of `CREATE INDEX...` on a database table is always going to be faster than going through a normal deployment pipeline. Even if we assume your pipeline is really fast, which is probably not the case at most orgs, you are still comparing a single SQL statement execution, to a single SQL statement execution + git push + code reviews + merge + running through Jenkins/Circle/whatever. How long does that overhead take? How much money have you lost because your website won't load when your post is on the frontpage of HN? Seconds and minutes count. I don't want my code crashing because an unexpected index exists in this scenario.
  - threeseed 2y ago
    You should be able to deploy end to end to Production in less than a minute. Companies should be focused on solving that problem first before doing insanely short-sighted workarounds like skipping pushing to Git and code reviews.
    - mjr00 2y ago
      > You should be able to deploy end to end to Production in less than a minute. When I was at AWS (RDS) our end-to-end production deployment process was 7 days. We were also pulling $25million/day or so in profit. I'm sure that number is much higher now. There's a large difference between what the theoretical "right" thing is from a textbook perspective, and what successful engineering teams do in reality. edit: besides, it doesn't even make sense in this context. I have 100 servers talking to the database. I need to create an index, ok, add it to the code. Deploy to server 1. Server 1 adds the index as part of the migration process, and let's say it's instant-ish (not realistic but whatever). Do the other 99 servers now panic because there's an unexpected index on the table?

kelnos 2y ago

In addition to the deployment-time issues and other stuff I and others have commented downthread, I thought of another problem with this. I can't see how this would even work for trivial, quick, on-line schema changes. Let's say I have 10 servers running the same service that talks to the database (that is, the service fronting the database is scaled out horizontally). How would I do a migration? Obviously I can't deploy new code to all 10 servers simultaneously that will do the schema migration; only one server can run the migration. So one server runs the migration, and... what, the other 9 servers immediately panic because their idea of the schema is out of date? Or I deploy code to all 10 servers but somehow designate that only one of them will actually do the schema migration. Well, now the other 9 servers are expecting the new schema, and will panic before that 1 server can finish doing the migration. It seems to me that rust-query is only suitable for applications where you have to schedule downtime in order to do schema changes. That's just unacceptable for any business I've worked at.

dayjah 2y ago
This isn’t unique to rust-query; this problem also exists with ActiveRecord, for example. At Twitch we just had to really think about our migrations and write code to handle differences. Basically no free lunch!
rendaw 2y ago
I think first and foremost, if you're going to use a tool like this, you need to do everything through the tool. That said, for zero downtime migrations there are a number of techniques, but it typically boils down to splitting the migration into two steps where each step is rolled out to each server before starting the next: https://teamplify.com/blog/zero-downtime-DB-migrations/ https://teamplify.com/blog/zero-downtime-DB-migrations/ https://johnnymetz.com/posts/multistep-database-changes/ https://johnnymetz.com/posts/multistep-database-changes/ etc I'm not sure if there's anything that automates this, but it'd probably need to involve the infrastructure layer (like terraform) too. Edit: There's one other approach I've heard of for zero downtime deployments: Start running the new version in new instances/services parallel to the old version, but pause it before doing any database stuff. Drain client connections to the old version and queue them. Once drained, stop the old version, perform database migrations, and start the new version, then start consuming the queue. This is (I think) more general but you could get client timeouts or need to kill long requests to the old version, and requires coordination between infrastructure (load balancer?) and software versions.

Merad 2y ago

Unless you have some tricks up your sleeve that I'm not thinking of, an immediate consequence of this is that zero downtime deployments and blue/green deployments become impossible. Those both rely on your app being able to run in a state where the schema is not an exact match for what the app expects - but it's compatible so the app can still function.

ComputerGuru 2y ago
Semantic versioning?
- Merad 2y ago
  If I understand the GP correctly, there's no notion of semver involved. Any difference in the schema results in a runtime error.
  - ComputerGuru 2y ago
    Yes, that’s what I gathered from the code. But I was proposing it as an “easy” solution that doesn’t involve throwing away OP’s main idea.
Filligree 2y ago
And that's okay. Most applications don't need zero-downtime deployments, and there are already plenty of APIs that support that use case. I'd rather have more like this one.

Filligree 2y ago

I'm so glad you made this. I've been searching for a decent Rust database library for half a year already, and this ticks all the boxes. I haven't tried it yet, so I might have to eat my words later, but- great job! It's going to save tons of effort.

theptip 2y ago

In all the systems I’ve built (mostly Django) you need to tolerate vN and vN+1 simultaneously; you are not going to turn off your app to upgrade the DB. You’ll have some Pods on the old application version while you do your gradual upgrade. How do you envision rolling upgrades working here?