4 ms·

I’ve never used Ecto outside of a Phoenix app, so yes, in my mind the line between them was blurry. Thanks for pointing out my mistake! A very recent real-worl

I’ve never used Ecto outside of a Phoenix app, so yes, in my mind the line between them was blurry. Thanks for pointing out my mistake! A very recent real-world challenge I ran into was having to coordinate updates and diffs to all the files involved and keeping them in sync while doing local development (before deployment but also while working on fixing a very subtle bug). Every time I changed the migration file (even to add an index), I had to remember to checkout the previous version of “structure.sql” or the migration wouldn’t be run, even during a complete DB reset (structure.sql is used to know which migrations have been run and synced with the SQL dump). Also, the schema in Ecto does not have a complete validation API, so you’ll need to do those in “changeset” functions. Overall I like Ecto, and part of me regrets letting myself get spoiled by Django’s model layer and DB tools.

josevalim 4y ago
It is totally fine to get spoiled if you consider it is better :D Your case about structure.sql is interesting. It would be nice if we could automate it somehow but, if we simply tried to re-run a changed migration, then the migration would likely fail because the other operations in it (such as adding fields), you already exist, no? Do you have any suggestions/ideas on how to tackle this? We could have a "mix ecto.migration.fix" that reverts the last migration, wait until you close/save the file, and migrate again, but I am not sure how useful it would be. How would Django approach this? If you auto-migrate and then do further changes, does it change the existing migration or does it generate new ones? PS: the changeset functions are pretty much on purpose though. It is important to decouple the validation from the schema because a single schema can have several validations rules (and they can diverge overtime!).
- ralmidani 4y ago
  Django’s approach is to run all the migrations, from the migration files, if the database gets wiped clean. In other words, if you’re developing on your local machine, you can just drop the DB and start new (or if using SQLite, just delete a file), and after running migrations, you never have to worry about whether all the necessary files are in sync with the database. As Neo might say:“there is no spoon…” seriously, structure.sql is simply not needed, so you won’t find yourself making creative use of git reset/checkout on a specific file as you’re iterating. To answer an earlier question you had, Django’s “makemigrations” command tries its best to help you avoid having to manually edit migration files; if you rename a field in models.py, it’s usually smart enough to ask you, at the command line, if you renamed it, and proceed to make the migration file for you. Of course, for advanced migrations the command can only take you so far, but again, if you’re trying to iterate quickly, you really need to experience Django’s ORM and DB tools to be able to appreciate them fully. Regarding changesets, I love the functional API, and at the same time I wish we could automatically derive a “default_changeset” function from the schema itself for straightforward cases, and then let devs run diverging validations in their own changeset functions. Maybe an interesting idea to explore with a macro? By the way, I’m a huge fan of your work, and hope I get the opportunity to work with Elixir for a long time. Any criticisms or comparisons are to try to bring improvements to the ecosystem I love. :)
  - josevalim 4y ago
    Thanks! FWIW, structure.sql is not needed in Ecto either. You must do that when: 1. You want to start discarding migrations (because you have too many) 2. You had an existing database Are those never a "trigger" on Django side? I appreciate the opportunity to learn!
    - maxmorlocke 4y ago
      Generally speaking, it's inadvisable to 'discard' migrations with django. If the migration set is getting too large, the general practice is to 'squash' migrations, which is a django-supported function for merging the migrations down to a single app. You can do the less blessed, but more simple function of nuking the migrations, updating the migrations table, and remaking your migrations... but you have to coordinate that in every environment. For an existing database, you can easily create django models for each table, etc. There's a --fake option to update the migration table to make it think you've applied these migrations, but not actually apply them. This convinces django you've brought the database in sync. May your deity or deities help you if you did not actually bring it in sync. I've used this quite a bit in some java ee->python migrations I've done in the past.
    - ralmidani 4y ago
      TIL you actually don’t need structure.sql for Ecto! In that case, discarding migrations on a large running app seems like a not-always-good practice, right? FWIW, my experience with Ecto has always been in joining existing codebases that are already big. I do also want to add that Django can kind of do the reverse of auto-generating migrations: it can “inspect” an existing database and generate Python classes that allow you to use the Django API as if you wrote those classes yourself: https://docs.djangoproject.com/en/4.1/howto/legacy-databases/ https://docs.djangoproject.com/en/4.1/howto/legacy-databases... Of course, the feature is not going to be perfect, especially if a team has been using odd naming conventions in the legacy database, but it seems helpful at least in theory.
      - bostonvaulter2 4y ago
        Inspecting a database and coming up with models is a frequent pain point that I see new elixir devs experiencing. I’d love to see someone take a stab at generators for that. I wonder if someone would do an integration for https://ash-hq.org/ https://ash-hq.org/ that does that.
- bostonvaulter2 4y ago
  Hmmm, if we’re willing to accept a dependency on git than it might be interesting to write a `mix ecto.migration.rerun` that rolled back any migrations that are untracked or dirty according to git, then re-ran them. That would simplify a pain point I encounter when I’m iterating on a migration because sometimes I rollback too far accidentally and finding the right argument to rollback feels like a chore. The nice thing is that could be prototypes outside of ecto quite easily.

josevalim 4y ago

It is totally fine to get spoiled if you consider it is better :D Your case about structure.sql is interesting. It would be nice if we could automate it somehow but, if we simply tried to re-run a changed migration, then the migration would likely fail because the other operations in it (such as adding fields), you already exist, no? Do you have any suggestions/ideas on how to tackle this? We could have a "mix ecto.migration.fix" that reverts the last migration, wait until you close/save the file, and migrate again, but I am not sure how useful it would be. How would Django approach this? If you auto-migrate and then do further changes, does it change the existing migration or does it generate new ones? PS: the changeset functions are pretty much on purpose though. It is important to decouple the validation from the schema because a single schema can have several validations rules (and they can diverge overtime!).

ralmidani 4y ago
Django’s approach is to run all the migrations, from the migration files, if the database gets wiped clean. In other words, if you’re developing on your local machine, you can just drop the DB and start new (or if using SQLite, just delete a file), and after running migrations, you never have to worry about whether all the necessary files are in sync with the database. As Neo might say:“there is no spoon…” seriously, structure.sql is simply not needed, so you won’t find yourself making creative use of git reset/checkout on a specific file as you’re iterating. To answer an earlier question you had, Django’s “makemigrations” command tries its best to help you avoid having to manually edit migration files; if you rename a field in models.py, it’s usually smart enough to ask you, at the command line, if you renamed it, and proceed to make the migration file for you. Of course, for advanced migrations the command can only take you so far, but again, if you’re trying to iterate quickly, you really need to experience Django’s ORM and DB tools to be able to appreciate them fully. Regarding changesets, I love the functional API, and at the same time I wish we could automatically derive a “default_changeset” function from the schema itself for straightforward cases, and then let devs run diverging validations in their own changeset functions. Maybe an interesting idea to explore with a macro? By the way, I’m a huge fan of your work, and hope I get the opportunity to work with Elixir for a long time. Any criticisms or comparisons are to try to bring improvements to the ecosystem I love. :)
- josevalim 4y ago
  Thanks! FWIW, structure.sql is not needed in Ecto either. You must do that when: 1. You want to start discarding migrations (because you have too many) 2. You had an existing database Are those never a "trigger" on Django side? I appreciate the opportunity to learn!
  - maxmorlocke 4y ago
    Generally speaking, it's inadvisable to 'discard' migrations with django. If the migration set is getting too large, the general practice is to 'squash' migrations, which is a django-supported function for merging the migrations down to a single app. You can do the less blessed, but more simple function of nuking the migrations, updating the migrations table, and remaking your migrations... but you have to coordinate that in every environment. For an existing database, you can easily create django models for each table, etc. There's a --fake option to update the migration table to make it think you've applied these migrations, but not actually apply them. This convinces django you've brought the database in sync. May your deity or deities help you if you did not actually bring it in sync. I've used this quite a bit in some java ee->python migrations I've done in the past.
  - ralmidani 4y ago
    TIL you actually don’t need structure.sql for Ecto! In that case, discarding migrations on a large running app seems like a not-always-good practice, right? FWIW, my experience with Ecto has always been in joining existing codebases that are already big. I do also want to add that Django can kind of do the reverse of auto-generating migrations: it can “inspect” an existing database and generate Python classes that allow you to use the Django API as if you wrote those classes yourself: https://docs.djangoproject.com/en/4.1/howto/legacy-databases/ https://docs.djangoproject.com/en/4.1/howto/legacy-databases... Of course, the feature is not going to be perfect, especially if a team has been using odd naming conventions in the legacy database, but it seems helpful at least in theory.
    - bostonvaulter2 4y ago
      Inspecting a database and coming up with models is a frequent pain point that I see new elixir devs experiencing. I’d love to see someone take a stab at generators for that. I wonder if someone would do an integration for https://ash-hq.org/ https://ash-hq.org/ that does that.
bostonvaulter2 4y ago
Hmmm, if we’re willing to accept a dependency on git than it might be interesting to write a `mix ecto.migration.rerun` that rolled back any migrations that are untracked or dirty according to git, then re-ran them. That would simplify a pain point I encounter when I’m iterating on a migration because sometimes I rollback too far accidentally and finding the right argument to rollback feels like a chore. The nice thing is that could be prototypes outside of ecto quite easily.