4 ms·

"Also be wary of the formerly South, now built-in migrations stuff. It's built around a fragile model (perfect history representation), so has lots of foot guns

"Also be wary of the formerly South, now built-in migrations stuff. It's built around a fragile model (perfect history representation), so has lots of foot guns." My experience has been opposite so I'd be interested in hearing your experiences if you are willing. I have been using the built in migrations since day 1 on a medium sized Django project with 350+ migrations and migration issues for our project have been exceedingly rare. edit: We have a small team of developers, so merge migrations are very rare for us, which might be a contributing factor.

rowanseymour 4y ago
Same here. If I was starting a non-Python project tomorrow I'd consider using Django to manage the database schema - especially now that we can describe custom indexes and constraints in migrations. Our project has gone thru over 1000 migrations so far tho we squash them down to 50 or so about once a year.
- jbellew 4y ago
  Thats interesting, if I was using TypeScript to access the data, how would I keep the schemas in sync between TS and python?
  - rowanseymour 4y ago
    I've never used TypeScript to talk to a database but there might be tooling to generate classes from tables. Even if there isn't, manually keeping some TypeScript classes in sync might still be worth the effort, for being able to manage schema migrations easily elsewhere.
  - dnadler 4y ago
    We do this manually along with a pydantic as a middleman between Django and TS. Works pretty well and is not a major inconvenience to keep things aligned.
  - anileated 4y ago
    Can introspect and export Django models to JSON schema or similar, then in TS read it and use compiler low-level API to generate types. There may be libraries for either stage…
  - replygirl 4y ago
    if using DRF you export openapi to json and use openapi-typescript-codegen if using graphene or strawberry you export sdl to json and use @graphql-codegen/typescript
  - crucialfelix 4y ago
    Prisma works very well and has usable migrations. Not as solid as Django's. Tiangolo of FastAPI fame is working on https://sqlmodel.tiangolo.com/ https://sqlmodel.tiangolo.com/ Which is pydantic models, SQL alchemy. Migrations are coming soon. This will probably be an excellent way to build APIs with db. Then you can generate a typescript client from the built in OpenAPI schema.
theptip 4y ago
Diamond-dependencies are hard to work with, and so we forbade them. Specifically, you can't revert an individual migration, you just specify your desired "target" to roll back to, and therefore you can't unapply a single branch of a diamond dependency. This means if your second branch of a diamond dependency breaks the DB, but your app depends on the first branch, you're SOL and are now manually running SQL to fix your production DB. (Can you tell I'm speaking from experience here? :) ) Your migration code uses the model classes, but the migrations are using a rehydrated version of the model that doesn't include any methods; another footgun. Basically you need to copy in any logic that you're using into your migration file, or else that migration's logic will change as your code is refactored. You might naively think that because `model.do_the_thing()` works now, the migration is somehow pickling a snapshot of your model class. It's not. Because of the above, you should really squash migrations frequently, but it's a big pain to do so -- particularly if you have dependency cycles between apps. ("Just Say No to Apps" is my default advice for new Django developers. If you have a real need for making a bit of library code sharable between multiple different Django projects then you know enough to break this rule. Within a single monolithic project, apps just cause pain.) Squashing migrations quickly gets to some extremely gnarly and hard-to-work-with logic in the docs. Moving models between apps isn't supported by the migration machinery; it's an involved and dangerous process. One idea here that might save Apps is if you manually remove the App Prefix from your "owned" / internal apps; if I have Customer and Ledger apps, I don't really need to namespace the tables; `user`, `information`, `ledger_entries` are fine table names instead of `customer_user`, `customer_information`, `ledger_ledger_entries`, a normal DB admin would not namespace the table names. You neeed the app namespacing to make it safe to use someone else's apps, but I think namespacing for your own apps inside a single repo is harmful. I find the migration framework to be worth using, but it's definitely got some sharp edges.
- randlet 4y ago
  Great answer thank you!
- rowanseymour 4y ago
  Maybe you need to turn this into an article because over the last decade of working with Django, we've learnt the same lessons, sometimes the hard way. Learning to never import real models into data migrations was a big one. I recently wanted to move a model between apps and ended up going the route of create new table, copy all rows over, delete old table. It was annoying, but the only way to make it work with regular migrations. We ended up writing our own script[1] to squash migrations, and I'd love to know if there's a better way. We needed something that works for clean installs or existing installs that already have the current migrations installed - so it generates empty migrations which get applied on existing installs, and then they get replaced with real initial migrations on clean installs starting from a new release. 1. https://github.com/nyaruka/rapidpro/blob/main/tools/squash_migrations.py https://github.com/nyaruka/rapidpro/blob/main/tools/squash_m...

rowanseymour 4y ago

Same here. If I was starting a non-Python project tomorrow I'd consider using Django to manage the database schema - especially now that we can describe custom indexes and constraints in migrations. Our project has gone thru over 1000 migrations so far tho we squash them down to 50 or so about once a year.

jbellew 4y ago
Thats interesting, if I was using TypeScript to access the data, how would I keep the schemas in sync between TS and python?
- rowanseymour 4y ago
  I've never used TypeScript to talk to a database but there might be tooling to generate classes from tables. Even if there isn't, manually keeping some TypeScript classes in sync might still be worth the effort, for being able to manage schema migrations easily elsewhere.
- dnadler 4y ago
  We do this manually along with a pydantic as a middleman between Django and TS. Works pretty well and is not a major inconvenience to keep things aligned.
- anileated 4y ago
  Can introspect and export Django models to JSON schema or similar, then in TS read it and use compiler low-level API to generate types. There may be libraries for either stage…
- replygirl 4y ago
  if using DRF you export openapi to json and use openapi-typescript-codegen if using graphene or strawberry you export sdl to json and use @graphql-codegen/typescript
- crucialfelix 4y ago
  Prisma works very well and has usable migrations. Not as solid as Django's. Tiangolo of FastAPI fame is working on https://sqlmodel.tiangolo.com/ https://sqlmodel.tiangolo.com/ Which is pydantic models, SQL alchemy. Migrations are coming soon. This will probably be an excellent way to build APIs with db. Then you can generate a typescript client from the built in OpenAPI schema.

theptip 4y ago

Diamond-dependencies are hard to work with, and so we forbade them. Specifically, you can't revert an individual migration, you just specify your desired "target" to roll back to, and therefore you can't unapply a single branch of a diamond dependency. This means if your second branch of a diamond dependency breaks the DB, but your app depends on the first branch, you're SOL and are now manually running SQL to fix your production DB. (Can you tell I'm speaking from experience here? :) ) Your migration code uses the model classes, but the migrations are using a rehydrated version of the model that doesn't include any methods; another footgun. Basically you need to copy in any logic that you're using into your migration file, or else that migration's logic will change as your code is refactored. You might naively think that because `model.do_the_thing()` works now, the migration is somehow pickling a snapshot of your model class. It's not. Because of the above, you should really squash migrations frequently, but it's a big pain to do so -- particularly if you have dependency cycles between apps. ("Just Say No to Apps" is my default advice for new Django developers. If you have a real need for making a bit of library code sharable between multiple different Django projects then you know enough to break this rule. Within a single monolithic project, apps just cause pain.) Squashing migrations quickly gets to some extremely gnarly and hard-to-work-with logic in the docs. Moving models between apps isn't supported by the migration machinery; it's an involved and dangerous process. One idea here that might save Apps is if you manually remove the App Prefix from your "owned" / internal apps; if I have Customer and Ledger apps, I don't really need to namespace the tables; `user`, `information`, `ledger_entries` are fine table names instead of `customer_user`, `customer_information`, `ledger_ledger_entries`, a normal DB admin would not namespace the table names. You neeed the app namespacing to make it safe to use someone else's apps, but I think namespacing for your own apps inside a single repo is harmful. I find the migration framework to be worth using, but it's definitely got some sharp edges.

randlet 4y ago
Great answer thank you!
rowanseymour 4y ago
Maybe you need to turn this into an article because over the last decade of working with Django, we've learnt the same lessons, sometimes the hard way. Learning to never import real models into data migrations was a big one. I recently wanted to move a model between apps and ended up going the route of create new table, copy all rows over, delete old table. It was annoying, but the only way to make it work with regular migrations. We ended up writing our own script[1] to squash migrations, and I'd love to know if there's a better way. We needed something that works for clean installs or existing installs that already have the current migrations installed - so it generates empty migrations which get applied on existing installs, and then they get replaced with real initial migrations on clean installs starting from a new release. 1. https://github.com/nyaruka/rapidpro/blob/main/tools/squash_migrations.py https://github.com/nyaruka/rapidpro/blob/main/tools/squash_m...