4 ms·
I agree Django's migration system has a few unfortunate quirks (most notably the help-text-change-migrations), but it's also worked great for Zulip across over
by tabbott 6y ago
I agree Django's migration system has a few unfortunate quirks (most notably the help-text-change-migrations), but it's also worked great for Zulip across over 300 database migrations, and I wouldn't recommend anyone do their own migration framework on raw SQL files by hand.
The issues mentioned here in this article are real (E.g. we have a lint rule blocking importing other code inside migration files), but the author is also ignoring the benefits of Django's system, namely that it's a well-documented system that does most of the work of managing migrations for you, and does a good enough job that the vast majority of developers working on a large software project (like Zulip, with 96 people who've contributed changes to database migrations) will do it correctly without special training just from the Django documentation.
And the downsides are certainly minor compared to the obvious downsides with rolling one's own system with raw SQL (E.g. the fact that nothing validates that the model definitions match the database schema in the author's proposal!). I wouldn't recommend anyone take the author's advice for what to do instead, but this article is helpful reading for anyone curious about what issues one will encounter after a few years working with Django.
> How many times have you worked on a project and, after 1-2 years of development, you have changed to a different database?
FWIW, Zulip was originally developed with MySQL, and then about a year into developed we switched to Postgres. It also had a phase where we ran SQLite in development environments. Building on Django made our life a lot easier doing this.
> Once the migrations have been applied to all your testing and production systems, and all your developers' databases, just remove them.
This is only true if you don't care about being able to use tools like `git bisect` where you might want to switch a development environment to older versions and back again (I do this often as part of).
> If you do it in a loop, it results in a "ripple load", meaning you'll run 50 queries, each one fetching one record, while you could have run a single query fetching 50 records, or just adding an extra JOIN to another query.
This is my biggest complaint about Django's ORM -- lazy fetching of foreign keys is great in a management shell but rarely desirable in production, and it results in inexperienced developers doing database queries in loops all the time. We address this with a bit of middleware that formats log lines like this e.g. `41ms (db: 10ms/2q)` (showing how much time was spend doing database queries and how many queries were done) -- which lets one easily identify problematic cases of this both in development and in production.
But if Django had a setting that just disables the lazy fetching of foreign keys feature (making accessing an unfetched foreign key an error), I'd turn it on for Zulip.
See also https://zulip.readthedocs.io/en/latest/subsystems/schema-migrations.html https://zulip.readthedocs.io/en/latest/subsystems/schema-mig....
- airstrike 6y ago> This is my biggest complaint about Django's ORM -- lazy fetching of foreign keys is great in a management shell but rarely desirable in production, and it results in inexperienced developers doing database queries in loops all the time. I may be misunderstanding you, but isn't this what select_related and prefetch_related exist for?
- acdha 6y agoThose are how you optimize it, but first you have to recognize the need. The easiest way to make this visible is to increase logging or install Django-debug-toolbar, so people can see when the query count jumps, but the best way to prevent regressions is to use something like the testing framework’s assertNumQueries() method to prevent regressions: https://docs.djangoproject.com/en/3.0/topics/testing/tools/#django.test.TransactionTestCase.assertNumQueries https://docs.djangoproject.com/en/3.0/topics/testing/tools/#... I’ve used that with a fuzzy number class which will say it’s __eq__ to any number below a specified max so you can say e.g. my homepage should take less than 10 queries and not have to touch it for minor changes while catching when someone implements a new feature in a way which breaks your select_related usage.
- sinsterizme 6y agoI really appreciate your comment, I think I will add your log line idea to my project at work! I also wanted to disable lazy fetching of foreign keys, and I wrote something for it, then decided it was too risky to use in production (monkey patching) so I ended up just running nplusone [0] locally and fixed the issues it detected. Really would like Django to add this feature so it can be enforced properly though! [0] - https://github.com/jmcarp/nplusone https://github.com/jmcarp/nplusone