4 ms·

Not sure what you mean. South has something called a "data migration". However, the only difference between "data migrations" and "schema migrations" is that da

by teilo 13y ago

Not sure what you mean. South has something called a "data migration". However, the only difference between "data migrations" and "schema migrations" is that data migrations do not include a dry run. Dry runs require transaction support. Not all databases have it (MySQL). Other than that, what is your definition of a "data migration"?

metaphorm 13y ago
data migration involves mutating data in the table. for example, consider a timestamp column that was originally recorded in some local timezone but that you need to convert to store everthing in UTC + offset format. No changes to the table itself, but a fairly significant change to the data. you could manage that operation using a Python script, but it would potentially be slow and might make it hard to take advantage of database specific features. A data migration tool allows you to describe this operation in succinct declarative code and then the migration tool will figure out how to get the database to do that with maximum safety and efficiency. or you could do it in raw SQL as well, but thats a bit uncomfortable if your project has used the ORM interface for all database operations. you'd also have to write different SQL for each backend. a data migration tool can generate the correct SQL for any backend you plug in.
- jsmeaton 13y ago
  I've always done data migrations like that, by hand, with straight up SQL. It would seem to me that there'd be way too many combinations of changes to properly implement in a generic way. Although, after reading your comment, I'd never taken 3rd party libraries into account. Adding/changing timezone support to a datetime field is a really good example of when that might be necessary. The groundwork has been laid with an API and supporting infrastructure for schema migrations though. It looks like it should be possible to implement data migrations by adding Operations to django.db.migrations.operations, and suitable sql and methods to each of the backends. You could then hand-craft the migration files. I'm assuming that the logic behind migrations wouldn't verify the database/model structure before running such a data migration which is probably flat out wrong though.
- teilo 13y ago
  Understood. But I cannot imagine a data migration framework that would be anything more than a scaffold for custom functions. Imagine the difficulty involving forward and backward data migrations. I cannot imagine a way to automate this. In any case, you can't have it both ways. Either you do it in SQL, or you do it in Python using the ORM. There isn't a third option.
  - deleted 13y ago
    [deleted]

by teilo 13y ago

metaphorm 13y ago

data migration involves mutating data in the table. for example, consider a timestamp column that was originally recorded in some local timezone but that you need to convert to store everthing in UTC + offset format. No changes to the table itself, but a fairly significant change to the data. you could manage that operation using a Python script, but it would potentially be slow and might make it hard to take advantage of database specific features. A data migration tool allows you to describe this operation in succinct declarative code and then the migration tool will figure out how to get the database to do that with maximum safety and efficiency. or you could do it in raw SQL as well, but thats a bit uncomfortable if your project has used the ORM interface for all database operations. you'd also have to write different SQL for each backend. a data migration tool can generate the correct SQL for any backend you plug in.

jsmeaton 13y ago
I've always done data migrations like that, by hand, with straight up SQL. It would seem to me that there'd be way too many combinations of changes to properly implement in a generic way. Although, after reading your comment, I'd never taken 3rd party libraries into account. Adding/changing timezone support to a datetime field is a really good example of when that might be necessary. The groundwork has been laid with an API and supporting infrastructure for schema migrations though. It looks like it should be possible to implement data migrations by adding Operations to django.db.migrations.operations, and suitable sql and methods to each of the backends. You could then hand-craft the migration files. I'm assuming that the logic behind migrations wouldn't verify the database/model structure before running such a data migration which is probably flat out wrong though.
teilo 13y ago
Understood. But I cannot imagine a data migration framework that would be anything more than a scaffold for custom functions. Imagine the difficulty involving forward and backward data migrations. I cannot imagine a way to automate this. In any case, you can't have it both ways. Either you do it in SQL, or you do it in Python using the ORM. There isn't a third option.
- deleted 13y ago
  [deleted]