13 ms·
Schema migrations merged into Django master
The Kickstarter:
http://www.kickstarter.com/projects/andrewgodwin/schema-migrations-for-django
- gitaarik 13y agoSo what about data migrations? In the docs I couldn't find something about it. And it's pretty important for a complete migration tool, isn't it?
- teilo 13y agoNot sure what you mean. South has something called a "data migration". However, the only difference between "data migrations" and "schema migrations" is that data migrations do not include a dry run. Dry runs require transaction support. Not all databases have it (MySQL). Other than that, what is your definition of a "data migration"?
- metaphorm 13y agodata migration involves mutating data in the table. for example, consider a timestamp column that was originally recorded in some local timezone but that you need to convert to store everthing in UTC + offset format. No changes to the table itself, but a fairly significant change to the data. you could manage that operation using a Python script, but it would potentially be slow and might make it hard to take advantage of database specific features. A data migration tool allows you to describe this operation in succinct declarative code and then the migration tool will figure out how to get the database to do that with maximum safety and efficiency. or you could do it in raw SQL as well, but thats a bit uncomfortable if your project has used the ORM interface for all database operations. you'd also have to write different SQL for each backend. a data migration tool can generate the correct SQL for any backend you plug in.
- jsmeaton 13y agoI've always done data migrations like that, by hand, with straight up SQL. It would seem to me that there'd be way too many combinations of changes to properly implement in a generic way. Although, after reading your comment, I'd never taken 3rd party libraries into account. Adding/changing timezone support to a datetime field is a really good example of when that might be necessary. The groundwork has been laid with an API and supporting infrastructure for schema migrations though. It looks like it should be possible to implement data migrations by adding Operations to django.db.migrations.operations, and suitable sql and methods to each of the backends. You could then hand-craft the migration files. I'm assuming that the logic behind migrations wouldn't verify the database/model structure before running such a data migration which is probably flat out wrong though.
- teilo 13y agoUnderstood. But I cannot imagine a data migration framework that would be anything more than a scaffold for custom functions. Imagine the difficulty involving forward and backward data migrations. I cannot imagine a way to automate this. In any case, you can't have it both ways. Either you do it in SQL, or you do it in Python using the ORM. There isn't a third option.
- deleted 13y ago[deleted]
- inglesp 13y agoThis is exciting for Django, and it's exciting for OSS in general, as it demonstrates the viability of crowdfunding (certain) OSS development. Congratulations to Andrew and to all involved.
- vanni 13y agoFor reference: http://www.kickstarter.com/projects/andrewgodwin/schema-migrations-for-django http://www.kickstarter.com/projects/andrewgodwin/schema-migr... http://www.aeracode.org/2013/8/23/plan-comes-together/ http://www.aeracode.org/2013/8/23/plan-comes-together/
- Daviey 13y agoNeat. Once this hits a stable release, does it mean that South should be considered deprecated for new projects?
- andrewgodwin 13y agoFor anything on Django 1.7 (the version it will come out in), yes, South will no longer be needed. Older versions of Django should get a South 2 in the next few months with a lot of this code backported.
- nilved 13y agoAwesome, Django might be usable out of the box now. :)
- fotcorn 13y agodjango-extensions is still missing, especially the shell_plus and runscript commands.
- targusman 13y agoI don't use most of django-extensions, but shell_plus would be awesome.
- nkuttler 13y agoYes, for people who couldn't type pip install south
- nilved 13y agoThat's why I specified "out of the box." It doesn't make sense for there to be completely mandatory extensions like South -- they should be bundled in, fitting with the "batteries included" culture of Python.
- nkuttler 13y agoI don't really feel out of the box when I type pip install django south
- nilved 13y agoRegrettably, how you feel doesn't change the well-understood and long-standing definition of "out of the box."
- WayneDB 13y agoYou have to A) know about this ridiculously named package called "south" and B) remember to install it everytime though. That's why it's not "out of the box". Here, Google even backs us up on this - https://www.google.com/search?q=define%3A+out+of+the+box https://www.google.com/search?q=define%3A+out+of+the+box "Out of the box is the term used to denote items, functionalities, or features that do not require any additional installation. ..." - via http://en.wikipedia.org/wiki/Out_of_the_box http://en.wikipedia.org/wiki/Out_of_the_box
- j4mie 13y agoAnd here are the docs: https://docs.djangoproject.com/en/dev/topics/migrations/ https://docs.djangoproject.com/en/dev/topics/migrations/
- japhyr 13y agoMigrations are per-app, but the command is shown as python manage.py makemigrations Using South, it was python manage.py schemamigration app_name How does django know which app to build migrations for?
- andrewgodwin 13y agoIt does them all at once. The new system knows about interdependencies, so if you have ForeignKeys between apps it'll make all of them in the right order.
- deleted 13y ago[deleted]
- arocks 13y agoSo if I understand correctly, "makemigrations" will convert all apps to use migrations except apps which were created using previous versions of Django?
- andrewgodwin 13y agoIt will prompt you if an app doesn't yet have migrations and ask if you want it to. That prompt is currently a bit aggressive - asking you every time - but that will be fixed soon.
- antihero 13y agoHas it solved that issue where say I have a TinyMCE field, then I ditch TinyMCE and use CKEditor, that I'd still need TinyMCE installed in order to do the old migration, despite it just being a TextField database-wise?
- andrewgodwin 13y agoI'd like to point out that this is just a first working version, and there's still some work left to do, but I'm very relieved this is finally merged!
- mattdeboard 13y agoIs this pluggable? Can users stick with South (or whatever migration backend) and still use the same API?
- andrewgodwin 13y agoNo, this is a complete rewrite and a different way of doing migrations than South currently does. There should also be a South 2 eventually that is mostly compatible with this new migration design, for those stuck on older Django.
- deleted 13y ago[deleted]
- famousactress 13y agoAndrew, is there info somewhere comparing the new stuff to current South in terms of what features differ etc? I'm particularly curious if there have been any changes to the way the orm freezing infra works.
- simonw 13y agoORM freezing is basically gone, in favour of a smarter approach where the previous state is derived from reading all of the previous migrations in to an in-memory data structure. Andrew's blog has some great write-ups of how the new system has evolved: http://www.aeracode.org/category/django-diaries/ http://www.aeracode.org/category/django-diaries/
- famousactress 13y agowow. I peeked, and that's gonna take me some time to wrap my head around. Really interesting approach. Does this mean though that if you ever fall out of favor w/ the db.migrations API and for instance make a manual change to a piece of schema, that the faked orm will be forever broken because it won't know about that change?
- slig 13y agoI briefly read the doc in the changeset, but couldn't find if the migration files from South are compatible with this new bultin migrations. Does anyone know that?
- kingkilr 13y agoNo, south migrations are not compatible, you'll have to collapse your south migrations down and start with these.
- workhere-io 13y agoSomewhat related: Are there any plans to integrate Alembic into SQLAlchemy?
- mattdeboard 13y agoI hope not. Not everything has to be a kitchen sink.
- lucian1900 13y agoIt's about as integrated as it could get while still being a separate package, since it's written by the author of Alchemy. Most users of SQLAlchemy tend to use several libraries anyway, as opposed to one framework (the Django style).
- ris 13y agoBut does it work with custom ("initial") sql? This was one of the failing points of south in my opinion.
- ciokan 13y agoI absolutely love python and I'm using it daily but Django seems so....outdated?! Biggest web framework needs someone to raise funding via kickstarter to build a schema migration? 2013 and you still can't build a REST application without installing 3rd party code? I mean c'moon there's angularjs, emberjs, knockout, extjs etc etc etc out for years don't you see a light at the end of the tunnel or you're the type of team that build more tunnel? Sorry to say but news like this just make me sad. Still happy I chose flask + sqlalchemy for my website. Django is somewhere in 2004-2005 still. I lose the admin of course but I hate general things anyway (one size fits all type of thing).
- oinksoft 13y agoAnd the rest of the world who see Django as an amazing RAD tool with a huge app ecosystem keep plugging along ... don't get me wrong, I'm a huge advocate frameworks like AngularJS, but that approach is just not suitable for most meat-and-potatoes public-facing web applications.
- wbond 13y agoI recently built (more or less) a REST API and a Backbone front-end using Python. Why would I want to use Django to do so when there are so many other (more appropriate) options? Does Django really need to be a framework that does everything? I started before Flask supported Python 3, so I used bottle, psycopg2 and a bunch of other python packages. Knowing what I was interested in accomplishing, I was pretty easily able to stitch together these libraries to create a solid, to-the-point codebase. Having spent a bunch of time working on various backbone-heavy apps with Rails I don't have much interest in using a big, full-stack, opinionated framework to act as a REST API. There is so much overhead and so many opinions to work against.
- ciokan 13y agoI especially avoided the word "rails". You know why
- taude 13y agoCouldn't agree more, especially when you're basically speaking JSON with some authentication/authorization. I even question myself when using Flask-Restful vs just making my views return straight up JSON....but...that's a different discussion.
- est 13y agoNow let's get rid of WSGI and make async signals possible!
- jgroszko 13y agoAre you trying to combine WebSockets or realtime communication with Django? I'd look at Cody Soyland's excellent work on this. http://codysoyland.com/2011/feb/6/evented-django-part-one-socketio-and-gevent/ http://codysoyland.com/2011/feb/6/evented-django-part-one-so... or look at the Django example in gevent-socketio https://github.com/abourget/gevent-socketio https://github.com/abourget/gevent-socketio
- Ensorceled 13y agoYes! My £50.00 was well spent :-)
- groundCode 13y agoThis is great news for Django - congratulations to all involved!
- gamegoblin 13y agoI started coding Django a couple of months ago and was blown away when I attempted to add a field to an existing model only to get yelled at. StackOverflow comments told me that South was the only way to go. I guess I just couldn't believe it wasn't an already existing feature. It seems that adding fields is common enough to be needed early on... Happy about the news!
- grumps 13y agoI'd be curious to read that question... I have also been working Django on the side and have encountered migration issues.
- mbesto 13y agoI too was amazed when I first tried out Django to find out that there was no way to change my model, migrate and test. I guess that's one of the things they don't teach you upon experiencing ORM for the first time.
- glesica 13y agoIn the past (not sure about now) the Django docs (and thus, I have to assume, the devs) were somewhat against automated migrations. The rationale was that the developer should understand his or her data models and do migrations manually to avoid problems caused by mistakes made by the migration machinery. Obviously this attitude has changed, perhaps because of the success of South and the fact that most people would rather have the process automated, even if it causes occasional weirdness.
- chrismorgan 13y agoIt was documented as something that was desired, but not something that would go into Django core until the dust settled as to what the best approach was; it's not something to hurry into. What's been implemented now is significantly better than what South had. If they'd just decided to adopt South, that wouldn't have happened. Thus, it's good that it happened. :-)
- hcarvalhoalves 13y ago
- jsmeaton 13y agoFantastic work! I've been following the blog posts and it was interesting to understand the decisions you made along the way. Also, thanks for supporting Oracle straight away. When you first mentioned the kickstarter I noted the possibility that oracle support would either lag or be missed entirely, effectively making oracle support in django a second-class citizen. I'm starting a new project in a few weeks (on Oracle RAC), so I'll try to test as much as possible.
- bmelton 13y agoI'm also using Oracle, and have long lamented the lack of South (or anything like it). I will be a happy test user for this as well.
- caioariede 13y agoA bit OT, but how do you guys handle database changes between multiple branches? Say I added some fields in branch A through multiple apps and then I need to go back to the master to do a fix. How do you revert the changes before checkout master?
- sdfjkl 13y agoIf you write backwards migrations as well as forwards, South will do that for you, so presumably the Django migration system will do as well.
- caioariede 13y agoI know this. But I'm wondering if there is an easy way to do that, like a checkpoint that I can rollback all migrations to a specific point.
- kanzure 13y agoSome people just choose to use separate database names for each branch.
- defrex 13y ago"Migrations specify which other migrations they depend on - including earlier migrations in the same app - in the file, so it’s possible to detect when there’s two new migrations for the same app that aren’t ordered." This is a fantastic feature. I've worked on projects that use a South fork that only uses a migration number, eg 0003.py, specifically to cause version control to trigger a merge conflict.
- falcolas 13y agoIf any devs read this, for MySQL, you may want to consider using the update method adopted by many larger companies - something like pt-online-schema-change (or make the alter method pluggable so someone else can). Basically, create a table as a copy of the old table, set up triggers to update the new table as data pours into the old table, alter the new table, and then do a rename. This buys you some rollback capabilities, but more importantly it limits the impact on production traffic as the alters run. Of course, this is only really an issue on tables with hundreds of thousands of rows, but it's better than the naïve approach.
- famousactress 13y agoI've come to think this is something that I'll do earlier rather than later on projects moving forward. The last few projects I've worked on have put this off "until we get bigger", which inevitably ends up affecting agility and the types of migrations you're willing to do and when you're willing to release them. As complex as the approach is, I think I'd rather bite it off early in future projects. Or, use postgres!
- falcolas 13y agoPersonally, I don't start worrying about it until it actually becomes a problem (i.e. when an alter actually takes more than a few seconds to run). At which point it's easy to migrate to using pt-online-schema-change. Then again, I do most of my DB maintenance manually (I have DBA experience), so while I do use Django in production, I have less need to rely on generic automation tools. Or use PostgreSQL, Oracle, MSSQL, or... lots of options there. [EDIT] As a final thought on the topic, never be afraid of making schema changes, even in MySQL. It may require a bit more work to implement, but there are a bevy of solutions (most of which can be automated) which can limit or eliminate the "pain" caused by such alters.
- thezilch 13y agoOr, assuming you already run, at least, one secondary write or read-only server, use Percona XtraDB Cluster / Galera. There are caveats for pt-online-schema-change that make it not usable in all cases, and they are not necessarily edge cases.
- tocomment 13y agoDumb question here. Instead of keeping all these migration files why can the migration just compare the models to the database directly and make he appropriate changes?
- jsmeaton 13y agoFor a couple of reasons that I can think of. You might want to do a rollback. You may have multiple copies of the database (various dev versions, staging, production), and multiple changesets might build up in one environment before progressing to another which must be specifically ordered. There may be changes that need to be manually coded or tweaked, and you need some intermediate structure for that to happen.
- lectrick 13y agoWelcome to Rails circa 5 years ago! :)
- kayoone 13y agoHeck, even PHP frameworks have had migrations for ages!
- matthewrudy 13y agoRails has had decent migrations for at least 7 years, probably longer. (I only started 7 years ago, and it was already pretty standard)
- lectrick 13y agoMan, time flies...
- moe 13y agoExcept no, Rails doesn't have this. Not even remotely close. Rails still makes you write your migrations by hand. It also forces you to distribute your critical constraints, validations and dependencies over a usually incomplete model class, and incremental migration files. In Rails/AR there is no place in the filesystem to find out about the currently declared schema. It's a fundamentally broken design.
- Demiurge 13y agoSo is this slated for 1.7 now? Or is it still on track for 1.6?
- tomchristie 13y agoIt's for 1.7, always has been.
- Demiurge 13y agoOk. But it hasn't always been, according kickstarter (http://www.kickstarter.com/projects/andrewgodwin/schema-migrations-for-django http://www.kickstarter.com/projects/andrewgodwin/schema-migr...) 1.6 is mentioned, and "1.7 at latest". Considering 1.6 isn't out, I was hoping for it!
- tomchristie 13y agoNoted, apologies. 1.6 is in beta and feature frozen, and it's certainly been the plan for a while to target 1.7. I wouldn't sweat it tho - the 1.5-1.6 cycle has been pretty quick, and I'm sure as soon as the migrations work is ready and tested the Django core team will be pretty eager to push a 1.7 release.
- Demiurge 13y agoI'll work on tempering my excitement :)
- kdazzle 13y agoWhy wouldn't Django just use South instead of creating ever more dependencies on itself?
- simonw 13y agoThe migrations support for Django is being written by the author of South, based on everything he learnt on that project. It's essentially South 2 but baked in to Django.
- bb0wn 13y agoFinally, but too little too late in my opinion. I love python, but I would rather use Rails for web stuff at this point. So many 3rd party libraries are needed to do what Rails can do right out of the box.
- nkuttler 13y agoThat's fine. Some people consider Django bloated. Some prefer flask. Others will prefer something with more stuff included than Django. It sounds like you never were really the target audience for Django?
- bb0wn 13y agoWhat exactly is the target audience for Django? I actually love using Python and the Python design philosophy. If I'm not part of the target audience for Django, who is?
- d0m 13y agoWell django is already very big.. so the target audience is people using Django ;-) It's a very stable, python-based, web framework with a huge community and widely supported. Some python hackers find it too big with too much bloated features. As far as I'm concerned, anytime I start with a smaller web-framework I need to add mostly all django functionaly the next few days so meh. IMHO, for new projects, DJango is clearly one of the top contestant.
- andybak 13y agoYep. If half the world complains it does too much and the other half that it does too little then maybe it's got it about right ;-)
- bobbyi_settv 13y agoMost Rails projects that I've seen rely on way more third party "gems" than the average Django project. Regardless, using 3rd party libraries anywhere is pretty straightforward these days, so I don't see how the inconvenience of adding a library to your requirements list could outweigh the fact that Rails means you are using Ruby when you could be using Python.
- DrJ 13y agoPeople still argue over Flask vs Django. Django is a working car, the comes with a lot of features, and is more difficult to customize (and you eventually have to start hacking at it for custom features). Flask is the assembled chassis, engine, steering (no frame, body). It's up to you to build it into a car of your choice. Eventually though you will have to look at tuning the base system for your special special system.