6 ms·
Dynamic Django Models with Model Models
- Walkman 8y agoThis is extremely clever! Something like this needs a deep understanding of Django and Python internals. The bullet points are written like they put it together under 10 minutes or so :D I love it!
- thecatspaw 8y agoTo me it looks like a walkthrough how to set it up, rather than a guide. If I googled "Dynamic Django models", _this_ is what I would like to see.
- fleetfox 8y agoThis seems to solve all my EAV use cases.
- collyw 8y agoIt is essentially EAV isn't it? Well integrated into Django so it works with the admin and migrations.
- spapas82 8y agoNo it's not EAV. EAV is an sql (anti) pattern where you have a table with varchar fields such as name, type and value. So you'll have tuples like (id, int, 99), (name, string, sera) or (enable, boolean, true). This makes it very difficult to do aggregates and you can't do any consistency checks from the db. What the article proposes is much better than this: A way to create models (ie tables in the database) dynamically. To make it more clear if you are not familiar with django, it'd be the same as if the CREATE TABLE sql was generated dynamically depending on the user selections. So in this case each model will be in its own table and the fields will have proper types. You may even be able to have referential consistency using a ForeignKey field!
- babayega2 8y ago> You may even be able to have referential consistency using a ForeignKey field Since you say "you may" ... , do you happen to see any problem that may arise with this implementation ?
- spapas82 8y agoWell, the original article doesn't actually mention ForeignKeys that's why I written may (it only has TextField and IntegerField). Of course nothing stops you from extending them; you'll need to of course add the model where you'll want to create an FK to. So the MakeField model will have an optional field to denote the class to which you want to create the FK; probably an FK to ContentType would suffice for this field (https://docs.djangoproject.com/en/2.1/ref/contrib/contenttypes/#the-contenttype-model https://docs.djangoproject.com/en/2.1/ref/contrib/contenttyp...). I know I've written field and FK too many times, I hope it makes sense; if it doesn't tell me and I'll provide a more thorough example.
- jonatron 8y agoI know it's possible to take the ModelModel approach a bit further and add ForeignKeys to it. JSONField is like a better EAV, but it doesn't do dynamic relationships.
- collyw 8y agoYou are right, I had only quickly skimmed the article on my phone earlier. Its kind of similar, but moving it from the SQL layer to the Django layer in my opinion, but it does actually create database tables. The models are dynamic and in memory, (rather than in a file) which is at least as bad an idea as EAV in my opinion. But cool as a blog post.
- pmart123 8y agoI've used EAV before after putting some time into thinking of a solution and I'd be curious if there was a better way. Basically, we were building a restricted stock database depending on either a client preference or a customizable list. For instance, a religious institution may not want to invest in a company that derives revenue from tobacco. So, we might have: 1. Is the company in an industry or sector we have flagged as tobacco? 2. Upon review, did one of our analyst create an exception list for companies partially involved in selling tobacco i.e. CVS, LVMH with duty-free, etc.? 3. Did the religious foundation also provide its own list of companies associated with tobacco usually through a consultant or several. Therefore, we used a schema like: table: exclusion_lists columns: company_id, exclusion_list_id, source_id An account would register a restriction, which would group the baseline restricted companies from say sector to along with any customized lists that the account subscribed to, and a view would output all the restricted stocks and reasons why. It didn't seem like an anti-pattern at the time. The benefit seemed to be if a restriction code was changed, it would all happen in one place versus updated in a bunch of JSON objects. Adding a column for a client for some special restriction with three companies would create a very sparse table so that didn't seem like an elegant solution either.
- orf 8y agoAn alternative to this is to use a Form. You can have a model that produces the form configuration (relations for each field in the form for example) and produce this Form on request. Then, when the form is saved, take the resulting dictionary and save it to a JSONField in a 'result' model. If you want to re-display the form with the results, just create the form from the configuration model and add the data from the result model. I've had great results with this technique, and you can model any field except a foreign key. Django forms provides the validation, and the backing storage is dynamic so you can change fields easily. It seems less fragile than creating models like this article proposes.
- shezi 8y agoThat's about what the author says right at the top of the article. However, this is only more flexible in the front end. If you want to do anything with your data in the back end, a JSONField will help you less than fully structured data. While you can do some queries on JSON fields, most of the richer queries will have to be done in python, slowing processing down. If you're looking for runtime-changeable, structured data that you can run complex queries on, ModelModels sound like a good alternative. Most enterprise applications are in this area.
- spapas82 8y agoI'd written a blog post describing more or less this 5 years ago: https://spapas.github.io/2013/12/24/django-dynamic-forms/ https://spapas.github.io/2013/12/24/django-dynamic-forms/
- shimon 8y agoI did something like this in an app for art collectors once. My client wanted very different forms and styles for different objects -- say a coin vs. a stamp. The templates to render the forms were thus very complex and specified every field in the form, so rather than define fields in both the Form class and the template, I made a dynamic Form-subclass generator that would read the template file and create the class to process and store it. Typing that out now, I'm both enjoying the cleverness of that solution and wondering if it was at all justifiable. It probably would have made a good blog post.
- deleted 8y ago
- tomtomtom25 8y agoOn a slightly different topic, I was wondering as to what is the best approach to implement(or use a library) a progress bar in file upload (large files) forms. I've been looking around for quite a while but no success.
- jonatron 8y agoIt's suspiciously like you're asking me to shamelessly self promote https://github.com/jonatron/django-admin-resumable-js/ https://github.com/jonatron/django-admin-resumable-js/
- rheide 8y agoThis is the kind of thing that I love to see as a blogpost, and I admire the ingenuity of the author, but I would hate to ever see this in a production codebase.
- shezi 8y agoUnfortunately, most enterprise applications will have something like this, because they'll need to be customisable for individual companies. I've worked with a CA product years ago that had written their own SQL engine for doing exactly that. I would have much preferred the Django solution.
- Alex3917 8y agoYeah anyone who did this in a codebase I manage would be on their way to looking for a new job. Trying to save a couple lines of code by relying on undocumented private API functions is seriously not acceptable, especially when there are multiple other ways of doing this. Good blog post though.
- Walkman 8y agoThis is not about "trying to save a couple lines of code" at all. Sometimes you want to gives the customer the ability to customize the app however they want and sometimes this is needed in order to do that. E.g. custom profile fields, building custom forms themselves (like Google Forms), etc.
- TeeWEE 8y agoThis is basically what MetaModelling is about. A model of your model. Boom https://en.wikipedia.org/wiki/Metamodeling https://en.wikipedia.org/wiki/Metamodeling Its used a lot in defining DLS's. Or Programming languages in general. Also editors often need to work one level higher in the abstraction chain. Abstractions around sourcecode. Lisp is the master is this.
- ppeetteerr 8y ago"Warning: You probably don't want to do this because it's unsupported and very hacky." The opening line should have stopped here. Dynamic data structures in untyped languages are risky, hard to debug, and prone to breaking One can argue that document stores (e.g. CouchDB and MangoDB) have been around for a while but their strength (and that of JSONField), is the ability to store hierarchy, not arbitrary structure (which they also do). A dynamic model like the one in the article serves as a database-stored deserializer, which is more of the domain of the business logic, rather than data. Having said all this, the article is dope :o
- t-richards 8y ago>Dynamic data structures in untyped languages are risky What exactly do you mean by "untyped languages"? Python and every other high level language like it definitely has types. Maybe you meant "dynamically typed"?
- ppeetteerr 8y agoYeah, exactly.
- anonuser123456 8y ago'Dynamically typed' is a colloquial term for an untyped language.
- dangerbird2 8y agoIt's a snarl word for dynamically typed languages, to conflate strong, dynamic type systems like Python's with true untyped languages like Assembly, Forth, and BCPL.
- digitaLandscape 8y agoIt has types... which are rarely at runtime, when you typically just use an unchecked bag of key-value pairs. It's closer to getting a bag of bytes in assembly than it is to the behaviour you'd get in any actually-typed language. Nobody should be using such crap in 2018.
- imperio59 8y agoOne big downside with this approach no one is talking about is that your model is no longer in sync with the rest of your code via commits. This is generally what causes headaches later on, when you accidentally delete a field in production or forget to add a new one to mimic your dev environment. But we had something like that where I last worked and it was okay. There was also no concept of dev db which was a different topic altogether...
- Walkman 8y agoThis is not about generating code instead of writing it at all. It is about making customizations possible.
- candiodari 8y agoGiven that the whole point is to allow the customer to have their own schemas that is not a problem, but exactly the intention of the schema.
- adpirz 8y agoA real-world answer to the problem presented here seems to be Django CMS: https://www.django-cms.org/en/ https://www.django-cms.org/en/
- dfsegoat 8y agoIt's not really realistic to bring this into a large, pre-existing application (REST API) from our experience - which is admittedly limited. Re-building the app around django cms would be the way to go - but being able to do this from scratch in an existing application is pretty nice.
- danpalmer 8y agoI work on a large Django codebase, and while this is a cool blog post, it's quite important once you reach a certain size that these things aren't actually done in practice. If you wanted to _fundamentally_ build your product around this concept that's fine, but Django probably isn't the right framework, however no feature is worth the pain of this. For what it's worth, Django models and doing good relational design solve a huge number of problems. We've got ~380 Django apps and ~350 models and haven't yet needed dynamic models.
- aldoushuxley001 8y agoHow is it possible to have 380 Django apps in a single codebase? That sounds insane. Is that not absurdly complex to manage? I can't even imagine.
- danpalmer 8y agoIt actually works pretty nicely. Most of our apps are small and focused, and we nest apps within each other heavily. It's fairly common for example to see an app introduce some models, Python APIs to work with them, and business logic, and then to have a sub-app for the web API and bits relating to that, and a sub-app for the admin interfaces (we don't use the Django admin). Example: - orders - refunds - shipping - royal_mail - click_and_collect - fraud - fraud_blacklists - returns This means that working on our Click and Collect integration, one only needs to hold in their head the current app, and maybe some APIs provided by the apps above it. It's relatively uncommon to import across apps on different branches of the tree, and when we do that's typically only to top-level apps containing the most core models, or for very explicit public APIs provided to the whole site (which might be re-exported from an __init__.py up the tree anyway). I find it a pretty nice codebase to work on. We're hiring too https://www.thread.com/jobs/software-engineer https://www.thread.com/jobs/software-engineer
- toyg 8y ago> We encourage applications from people who aren’t cis white men, who are currently over-represented on the engineering team at Thread. ...
- egeozcan 8y agoGreat hack! A sane person would just use something that suits much better for this use-case though. Processwire[1] comes to mind, for example. [1]: https://processwire.com/ https://processwire.com/
- harel 8y agoI've done something similar as an experiment a few years back but I've used Meta Classes which define themselves, to support a schema-less model backed by MongoDB. It felt like a more flexible and correct approach at the time. In the end after my experiment ended, I've scrapped it decided to wait for PostgreSql 9.4 to be released with the BJSON field support. Then JSONField was my new friend.
- pjungwir 8y agoThere's this application I've been writing and re-writing my whole career: I call it "some users design forms, other users fill out forms." It is hard to design the db as anything other than EAV. (You can push more or less of it into JSON columns, but there are tradeoffs.) I have often wondered about pushing the customizability down a level though, like what the author does here, where each form creates a separate database table. It's too crazy for me, but it's tempting. And I have actually done it before where users' forms automatically generate individual tables in a separate flattened reporting database. It works really well for giving enterprise customers SQL access to an OLAP view of their data, even if I wouldn't use it for my source-of-truth OLTP structure. (Some day I'm going to write a blog post about that.) Even just for derived reporting tables, you quickly discover all kinds of limits in your database: reserved words, max columns per table, max columns per query, number of foreign keys pointing at a central table, etc. I haven't hit anything fatal though (with Postgres). Anyway, I'm excited to see that someone actually pulled this off, complete with Django models and migrations, even if it's just for fun.
- TylerE 8y agoPostgres JSONB columns could give you a lot of that without actually creating seperate tables - for instance indexes over embedded keys.
- ppeetteerr 8y agoThis. The best of both worlds + you get querying inside of the JSON.
- Keyframe 8y agoJSON Schema Form might be of interest to you.
- jackbravo 8y agoSomething like https://form.io/ https://form.io/ ?
- ubernostrum 8y agoI once wrote something very similar to this as a literal proof-of-concept hack for a form-oriented problem. It's fun to get it to the point where it works for the simplified test cases such projects always start out with. But I never used it beyond that proof-of-concept; I still reach for EAV and/or JSON columns when I need to do this sort of thing for production software.
- coleifer 8y agoAnother way of describing this is: * Django has facilities for mapping Python classes to database tables * Author uses metaprogramming to construct classes to match table definitions decided at runtime * Author uses built-in schema migration tools to synchronize backend There's nothing about this that wows me, although it does demonstrate knowledge of Django internals and Python metaprogramming. It doesn't impress me because this is the bread-and-butter of what Django does.
- jonatron 8y agoThat pretty much sums it up.
- Fradow 8y agoMy first instinct was to scream "just use a CMS", but there are obvious advantages to this technique, so here are my thoughts to make it less hacky if someone ever intends to run that in production. - the most obvious one I don't see mentionned is that you should use a separate database than your main one for all the generated tables. Pretty sure it would save on operational headaches. - it would probably be useful to generate all the Python code that could be used for making migrations, if only for debugging purposes - understand early on which Fields you want to use with which options. I wonder if ForeignKeyField would be possible. That would be a great feature, but probably bring headaches. - obviously, try to submit a PR to Django for the migrations loader hack, maintaining Monkey-patchs is troublesome All in all, it's a pretty cool hack, but even with those modifications, I'd still be very wary to use that in production instead of battle-tested alternatives, unless you have a use-case that really warrants having full-blown SQL schema for your dynamic data. Edit: on second thought, I can think of a use case where using that in a CMS would have saved me troubles (at least different tables for different Models). When you have a production site receiving Model A data (for example, your clients orders) AND you want to edit Model B data (for example, your site dynamic pages), you can't do that with EAV all in a single table, but with using dynamic models, you could just copy the Model B data, or even go with more complicated scripts.