14 ms·

Tips for Building High-Quality Django Apps at Scale

191 points by stanleytang 9y ago

randomsearch 9y ago
I'm so glad to see the mention of "app" directories here. I've only dabbled in Django development, but I've always thought the desire to divide things into apps didn't really make any sense. It felt like the developers of Django had said "well, intuitively, there must be some unit of reuse at this level" and then stuck the notion of apps in there in an attempt to provide that reuse. This seems to me to be somewhat unpythonic, unnecessary complexity introduced right from the start - why not make the reuse optional?. The file system layout becomes confusing as a result. More generally, I would expect that the most minimal Django site would be a single file, that would seem most Pythonic, but it isn't designed like that. Django doesn't "feel" like most Python projects do. I've always assumed this must be because I'm a total noob/idiot when it comes to real-world development, and that projects like Django must be doing things right and I just don't get it (although it remains entirely possible that this is the case). It's heartening, however, to see I'm at least not _entirely_ wrong for having a gut unease about this, and it makes me wonder how on earth I can judge whether it's me or a given framework/language in the general case. Do I have reverse Dunning–Kruger? Or did I just guess right in this case? I have no idea.
- mi100hael 9y ago
  Part of it is that Django has been around for quite a while now. It's maybe _the_ most succesful Python framework out there, so there are some paradigms that aren't common anymore but are hold-overs from years past. In particular, it pre-dates the current "microservice" trend and assumes a fairly monolithic environment. And I think "apps" _do_ make sense in certain contexts. Consider this situation: - I start a "polls" app like the Django docs example. - I get a lot of users and they like my functionality, so I want to engage the community. - I decide to start publishing blog posts, but I'm too lazy to spin up a whole new Wordpress install - I find a "blog" app and drop that into my project with the '/blog' URL prefix. Now I can start writing blog posts right away and anyone can go to my site '/blog' and see it. There are no cross-app database dependencies to worry about. The same would be true for adding a basic "social" app that shows a user-profile page for django Users at '/users/<id>' or something like that.
  - randomsearch 9y ago
    The polls app example is given in the introductory Django tutorial...
    - mi100hael 9y ago
      Yes, that's what I said...?
      - randomsearch 9y ago
        Sorry, misread that line. Point being, how many examples do you have where someone writing a Django site is going to reuse their own apps?
        soperj 9y ago
        I reuse my own apps constantly, whenever I do new sites.
        randomsearch 9y ago
        Excellent - could you give some examples of what you reuse? I'm interested in whether this is a worthwhile abstraction in the general case, or whether for most people (like me and the post) it's an unhelpful complication.
        mi100hael 9y ago
        It's a worthwhile abstraction for any generic feature that isn't specific to the one project you're working on: - User profiles - Content management/publishing - Messaging and chat - Searching your database - Admin - Analytics on your database Segmenting into apps makes the most sense when there are no cross-app dependencies on migrations: additional read-only capabilities on your existing models in another app, or new capability using entirely new, stand-alone models.
    - deleted 9y ago
- crdoconnor 9y ago
  >I'm so glad to see the mention of "app" directories here. I've only dabbled in Django development, but I've always thought the desire to divide things into apps didn't really make any sense. It felt like the developers of Django had said "well, intuitively, there must be some unit of reuse at this level" and then stuck the notion of apps in there in an attempt to provide that reuse. Yes, let's say you install a module that has a specific Django functionality from pypi, it will typically be installed as an app. It confuses newbies sometimes who think that they need to worry about how to divide up their application into different applications. 99% of the time they don't and shouldn't (putting everything in a single app called 'core' in your project is a pretty common pattern). Unfortunately the docs were never very clear on this. >This seems to me to be somewhat unpythonic, unnecessary complexity introduced right from the start - why not make the reuse optional?. The file system layout becomes confusing as a result. More generally, I would expect that the most minimal Django site would be a single file, that would seem most Pythonic, but it isn't designed like that. It can be made to behave like that if you really want: http://olifante.blogs.com/covil/2010/04/minimal-django.html http://olifante.blogs.com/covil/2010/04/minimal-django.html Most people aren't using Django for that though, so it's not widely advertised nor known. >It's heartening, however, to see I'm at least not _entirely_ wrong for having a gut unease about this, It's not like the framework is perfect. There's plenty wrong with it, but it largely managed not to fuck everything up, which is pretty rare.
- dangayle 9y ago
  Have you checkout out O'Reilly's Lightweight Django book? It basically goes through all of this, starting with a single file Django app.
  - randomsearch 9y ago
    No! I read a lot of books, but missed that. Thanks!
  - jwarren116 9y ago
    +1 for Lightweight Django. I had the pleasure of working with one of the authors. The book is well laid out and covers some really great topics that aren't always covered well elsewhere in the context of Django (i.e. Websocket services with Django). It's a great companion for Two Scoops of Django.
    - dangayle 9y ago
      In my opinion that book should be the starting point. The official tutorials have you way deep in the weeds with database migrations and stuff before you even have a basic understanding of where you're at and why. Worst "hello world" ever.
- Osmose 9y ago
  Where app directories are incredibly useful is in Django library development. They're a really convenient way to hook in models, static files, template helpers, etc. to an existing Django site, without forcing users to do a bunch of setup work when integrating your library. It's not a coincidence that most Django libraries have a very similar setup: 1. Install the package 2. Add the package to settings.INSTALLED_APPS (and middleware, etc). 3. Profit! For an instance of a Django site, apps are less useful, and the article's recommendation to be more careful when splitting your site into multiple apps is great advice.
  - randomsearch 9y ago
    Yeah, I agree. Makes sense for libraries. So keep that separate for library writers, don't expose me to the complexity of that.
    - andybak 9y ago
      The benchmark I use is "Is it probable that this app might become shared between projects?" Using a Django 'app' for this makes sense. If it's generic enough that I might consider moving it into it's own repo then it's probably not going to be heavily intertwined with the rest of the project.
- marcosdumay 9y ago
  > why not make the reuse optional? Well, it is optional. You can just not divide you stuff and place everything inside the main app. But notice the article recreates the same layer of reuse inside the main app, and got some extra problems because of it (renaming the tables is probably caused by this). Django may need an extra paragraph at the tutorial, telling how it is ok to place everything on the main app when everything is application specific software, and how you should break generic functionality on other apps. By the way, what would be reverse Dunning-Kruger?
  - randomsearch 9y ago
    > Well, it is optional What I mean is, you still have to have a "main" app. Why? I don't want to reuse it. So I don't want an "app". I just want to build my website. Stop making me put things in an unnecessarily complicated directory hierarchy. If it were Java, I wouldn't think twice about it, but it doesn't seem Pythonic. > reverse Dunning-Kruger Good question! Probably... illusory inferiority AKA imposter syndrome. I assumed Django knows best, but maybe it doesn't.
    - bmelton 9y ago
      Sure. One of the Djangoesque opinions is that if you're building a site, you'd like it to be maintainable, and that you'll probably add on to it. If you've been working in the Python ecosystem long enough to know that you're building a site that doesn't have those assumptions, you'll probably pick something different to start out with, like Flask or Bottle.
    - marcosdumay 9y ago
      > you still have to have a "main" app Well, it has to be organized some way. It happens to have the organization that gives you more flexibility, instead of forcing something that only fits very small sites onto you. That means you'll have an extra directory on your tree. Anyway, nobody uses only the main app. The article creates a big amount of problems by trying to only write on it, but I bet that even them use the Django's builtin apps. > AKA imposter syndrome Oh. Dunning-Kruger study detected that one too. It's within the effect: people tend to self evaluate into a small band near (but often under) the top, whatever their real evaluation is.
- arctangent 9y ago
  A good rule of thumb is that a Django "app" should be (1) standalone and (2) reusable. If you were to write a Django project to automate some complicated internal process at your particular company, it's likely that you'd break up the code into separate Python modules to help give it an easy to understand structure. But it's unlikely that any other company would necessarily benefit from any of those separate modules by themselves. However, let's say you come up with a clever idea about how to log errors in your project. Now, that's the sort of thing that other people might want to use in their own projects. If you were to restructure your code a bit and make it a bit more abstract then you could create a separate "app" that could be dropped into other Django projects.
- tlp8800 9y ago
  You could write a Django site in a single file if you wanted to. See this for an example: https://www.safaribooksonline.com/library/view/lightweight-django/9781491946275/ch01.html https://www.safaribooksonline.com/library/view/lightweight-d...
- jasonthevillain 9y ago
  It depends what you're doing. It shines on low level features like authentication and image cropping, where I need ways to do these things, and I can use them across the board. It breaks down when you have a lot of views and urls that are trying to stay close to their related models, but are too interconnected to have any real kind of separation of concerns. I find myself using all of the above. There's nothing wrong with building a small site in a single app, or nesting apps to provide a little more structure for how they're connected or separated.
- neves 9y ago
  I'm not affiliated with them, but you think seriously about Django, you should take a look at the book https://www.twoscoopspress.com/products/two-scoops-of-django-1-11https://www.twoscoopspress.com/products/two-scoops-of-django-1-11 https://www.twoscoopspress.com/products/two-scoops-of-django... I still have the 1.5 edition, but it helped me a lot. I'm think about upgrading. Do you recommend?
deleted 9y ago
[deleted]
cwisecarver 9y ago
I've got more than 5 years of experience with Django on a number of teams and at a couple of companies and in my experience almost everything in this article is completely incorrect. The only things I would agree with is the point about project layout and avoiding django's squashmigrations for the truncate the migrations table, delete the migrations, and create a new initial migration. Practically everything else in this article is wrong, in my opinion.
- adamors 9y ago
  Can you expand on what do you think is incorrect in the post?
  - deleted 9y ago
    [deleted]
  - cwisecarver 9y ago
    - FK/M2M across reusable, packaged apps is only bad if you don't match the interfaces correctly. See: almost every third-party Django app that is built to integrate with another application's models. - Sometimes you want the app concept just for organization. There's nothing bad about that and it makes sharing things inside of a project easier. - Explicitly naming your database tables doesn't make any sense. You're using an ORM. Accept it or don't use it. - Explicitly declaring through on a m2m field if you're not adding metadata to the relationship is pointless, but, if you're not using the table naming from Django I could see why they'd be invisible because there's no pattern to follow. - GenericForeignKeys are dangerous but not for any of the reasons listed. It's because they implicitly force a two way join which seems magic until it becomes debilitatingly slow. - The entire section on migrations leads me to believe that the first time the migrations are being run is on a production deploy. If you don't know SQL and you don't test your migrations prior to deployment, yes, it will be fairly difficult to determine what kind of performance/locking they're going to have. - No to Fat Models? This breaks down to "The framework we chose to use suggests a pattern, we also chose not to follow that pattern." That's fine if you want to do that but I wouldn't suggest it to others. - It's not hard to get signals to not fire in certain circumstances, you put the conditional in the signal callback like almost every other event-driven pattern. Also, bulk updating models doesn't fire signals in Django because save isn't called. Read the documentation. - Avoid using the ORM? Why choose a framework as complete as Django where 80% of the features are built around or on top of the ORM and then don't use it? - Caching complex objects makes cache invalidation hard. Well, yes, yes it does.
deleted 9y ago
[deleted]
tinix 9y ago
This reads way more like a list of Django caveats and anti-patterns, than a guide to running any kind of Python application at "scale". Maybe a sprinkling of good hygiene, but that has nothing to do with scalability (unless you're talking about developer scalability and cognitive overhead). Further, to suggest NOT using the ORM and to build out a middle layer on top of the ORM just for CRUD, is, well, insanity. On one hand the author is saying to cut down on bloat, and then on the other hand they are saying to add needless abstraction on an already bloated ORM. I don't use Django, but I do use Python for application development nearly every day, and I have to say... this article reads more like a list of (predictable and basic) problems their team had than a guide to scaling an application. Maybe they mean, how to scale their team, workflow, and their efficiency, NOT the application itself. The only thing I agree with here is the avoidance of some Django features... However, they should take it a step further, and just avoid Django in the first place. Also, am I the only one that has never needed to use "migrations" or any similar sorts of features? You're doing something very wrong and overcomplicating things if you seriously need to lean on migrations that heavily.
- Alex3917 9y ago
  > This reads way more like a list of Django caveats and anti-patterns, than a guide to running any kind of Python application at "scale". This isn't about scaling in the sense of handling more page views, but rather about scaling in the sense of having a codebase where the cost of adding each new feature and developer is O(n). The issue is that a lot of the 'recommended' Django patterns are terrible, and using all the features that are built-in would lead to a completely unmaintainable app. I don't understand the authors point about not using the ORM, in terms of why creating multiple tables inside a transaction doesn't solve the issue he's talking about. But aside from that all this advice is dead on, and these are exactly the same practices that I recommend to all consulting clients.
- orf 9y ago
  I agree with all of your points about the article, but... > However, they should take it a step further, and just avoid Django in the first place. Django is a tool, and like most tools it has a use. I find Django indispensable for writing specific kinds of applications, and it's admin interface is by far the best thing since sliced bread for internal/backoffice applications. It's not perfect by any means but it's amazing for quickly getting a CRUD app with an awesome administration interface up and running with minimal effort. Django rest framework, migrations and the amazing ecosystem (reversion, django-mptt, django-polymorphic, debug toolbar, django-currencies to name very few) are also what makes Django appealing and rather awesome. Making a blanket statement like 'Avoid Django' is rather silly, given all that. > Also, am I the only one that has never needed to use "migrations" or any similar sorts of features? I'm not sure why you put quotes around migrations as if it's some alien, obscure or weird feature. If you've never written a web application that needs migrations then you're not writing the kinds of applications (or indeed any 'serious' application) that would benefit from Django IMO.
johnthealy3 9y ago
I was feeling okay about this article until seeing this colossal punt: > That said, the real intention behind this pattern is to keep the API/view/controller lightweight and free of excessive logic, which is something we would strongly advocate. Having logic inside model methods is a lesser evil, but you may want to consider keeping models lightweight and focused on the data layer. To make this work, you will need to figure out a new pattern and put your business logic in some layer that is in between the data layer and the API/presentational layer. Having fat models is definitely a problem I'm having, and it's nice to see it's a problem for the author too, but the advice "figure it out" is presented without any explicit suggestions.
- iamsomewalrus 9y ago
  My team has run into the same problem in a large Rails app. We decided on a combination of service objects and data access objects (DAO) where appropriate. In a lot of cases this has really helped with the grep-ability of our codebase. Service objects allow us to encapsulate complex operations that touch several domains and tables through a simple API. They're just plain objects. DAOs fulfill a specific case of providing a simple CRUD API to a domain. We use a lot of DynamoDB so the DAO allows us to hide the complexity of reaching out to several tables in order to return a result. We also make sure that the DAO returns an immutable object that can't reach back into the datastore like an ActiveRecord object can (#save! methods come to mind). The difference between a service object and a DAO is a little blurry but is enforced through quick design discussion and code reviews.
- sbov 9y ago
  Procedural/functional code. Splitting your models up more by features than high level things (e.g. having a separate CustomerBilling rather than putting it in Customer). Component based architectures. Service layer to coordinate models. Etc. Also, this is/was a common problem (at least, the god class aspect) in games that the industry has slowly solved over the past 20 years, so that's another place you can look for inspiration. Further, unless you know you're going to scale in the beginning, I would recommend refactoring/evolving over time. It doesn't do anyone any favors by having 10 models to represent your Customer if your Customer is already relatively thin.
linkmotif 9y ago
Django is not meant to be run at scale. At least not horizontal scale. If you're not going to use the ORM, which prevents you from scaling horizontally, why are you using Django? Django can be scaled by getting large instances with lots of RAM, which is quite affordable these days. It can also be scaled with caching. But if you want horizontal scaling, don't use Django. That's not what it's for.
- benwilber0 9y ago
  There's nothing really inherent to Django that prevents it from scaling horizontally, not even the ORM. For the typical SQL read-heavy workload it will scale as well as anything else depending on the scalability of the backing RDBMS. Should always use proper HTTP response caching with something like Varnish or via 3rd-party CDN where possible.
  - linkmotif 9y ago
    > depending on the scalability of the backing RDBMS As far as I know, none of the backing RDBMSs that the ORM supports are horizontally distributable. At least not officially, right? CockroachDB with its Postgres interface sounds good, or Citus? But as of today, are any of the supported DBs distributed?
    - benwilber0 9y ago
      MySQL Cluster [1] has been around a long time for applications that need linear scalability with ACID guarantees [1] https://www.mysql.com/products/cluster/ https://www.mysql.com/products/cluster/
- ensignavenger 9y ago
  How does the ORM prevent horizontal scaling?
  - linkmotif 9y ago
    The ORM works against a relational database. The relational databases that Django supports are not distributed, right?
Osmose 9y ago
I appreciate this article for talking about scale in terms of project size and cognitive overhead, instead of in terms of traffic/computation/storage. Writing maintainable services isn't easy, and tips like these are painful to learn on your own.
goblin89 9y ago
I find that the safest way to migrate DB schema is gradually, spreading out intermediate steps across a few deploys. Migrate the schema creating the (initially unused) fields in advance. Step by step change your logic where it talks to the storage—start querying new field values with a fallback, use new fields for incoming data. Migrate the existing data. Remove the fallbacks. Ensure the old fields are not used at this point. Migrate the schema, finally deleting the old fields. Don’t rush, let every step reach production. In deployment sequence, apply migrations after the new code is already running. Goes without saying, monitor error rates for spikes. Yes, at some points in time your production DB schema won’t be fully normalized. In return for prolonged messiness you get smoother flow, no single fateful and stressful deployment that attempts to get it right all in one go. Migration handling is the only part of the article I find a bit foggy and debatable. The rest roughly resonates with what I’ve arrived at through my years of experience with Django web apps.
bradmwalker 9y ago
https://mobile.twitter.com/HackerNewsOnion/status/476835980863733761 https://mobile.twitter.com/HackerNewsOnion/status/4768359808...
joepour 9y ago
It's interesting that they're still using the default ORM. Stranger still they're letting the ORM scaffold tables (?!?) now they're at this size?
msmm 9y ago
It seems like author has problems with correctly architecting his apps and blames Django for his inability to do that. Only thing that makes sense is a tip for migrations.
St-Clock 9y ago
"Don’t cache Django models" This is too broad to be good advice. A typical solution to stale models is to increment a cache version key, effectively invalidating the old cache. Version keys can be coarse- or fine-grained. For example, you may have one version key for the entire application (supported by Django by default) and one version key per model or application (you have to roll your own solution). If your app's models change, you can increment the app version key and the next time you try to fetch a model instance from the cache, you will miss and instead fetch the instance from the DB.
nezo 9y ago
About ContentType : "If you ever move models between apps or rename your apps, you’re going to have to do some manual patching on this table" Thanks for this, just had a kind of struggle recently

191 points by stanleytang 9y ago

I'm so glad to see the mention of "app" directories here. I've only dabbled in Django development, but I've always thought the desire to divide things into apps didn't really make any sense. It felt like the developers of Django had said "well, intuitively, there must be some unit of reuse at this level" and then stuck the notion of apps in there in an attempt to provide that reuse. This seems to me to be somewhat unpythonic, unnecessary complexity introduced right from the start - why not make the reuse optional?. The file system layout becomes confusing as a result. More generally, I would expect that the most minimal Django site would be a single file, that would seem most Pythonic, but it isn't designed like that. Django doesn't "feel" like most Python projects do. I've always assumed this must be because I'm a total noob/idiot when it comes to real-world development, and that projects like Django must be doing things right and I just don't get it (although it remains entirely possible that this is the case). It's heartening, however, to see I'm at least not _entirely_ wrong for having a gut unease about this, and it makes me wonder how on earth I can judge whether it's me or a given framework/language in the general case. Do I have reverse Dunning–Kruger? Or did I just guess right in this case? I have no idea.

mi100hael 9y ago
Part of it is that Django has been around for quite a while now. It's maybe _the_ most succesful Python framework out there, so there are some paradigms that aren't common anymore but are hold-overs from years past. In particular, it pre-dates the current "microservice" trend and assumes a fairly monolithic environment. And I think "apps" _do_ make sense in certain contexts. Consider this situation: - I start a "polls" app like the Django docs example. - I get a lot of users and they like my functionality, so I want to engage the community. - I decide to start publishing blog posts, but I'm too lazy to spin up a whole new Wordpress install - I find a "blog" app and drop that into my project with the '/blog' URL prefix. Now I can start writing blog posts right away and anyone can go to my site '/blog' and see it. There are no cross-app database dependencies to worry about. The same would be true for adding a basic "social" app that shows a user-profile page for django Users at '/users/<id>' or something like that.
- randomsearch 9y ago
  The polls app example is given in the introductory Django tutorial...
  - mi100hael 9y ago
    Yes, that's what I said...?
    - randomsearch 9y ago
      Sorry, misread that line. Point being, how many examples do you have where someone writing a Django site is going to reuse their own apps?
      - soperj 9y ago
        I reuse my own apps constantly, whenever I do new sites.
        randomsearch 9y ago
        Excellent - could you give some examples of what you reuse? I'm interested in whether this is a worthwhile abstraction in the general case, or whether for most people (like me and the post) it's an unhelpful complication.
        mi100hael 9y ago
        It's a worthwhile abstraction for any generic feature that isn't specific to the one project you're working on: - User profiles - Content management/publishing - Messaging and chat - Searching your database - Admin - Analytics on your database Segmenting into apps makes the most sense when there are no cross-app dependencies on migrations: additional read-only capabilities on your existing models in another app, or new capability using entirely new, stand-alone models.
  - deleted 9y ago
crdoconnor 9y ago
>I'm so glad to see the mention of "app" directories here. I've only dabbled in Django development, but I've always thought the desire to divide things into apps didn't really make any sense. It felt like the developers of Django had said "well, intuitively, there must be some unit of reuse at this level" and then stuck the notion of apps in there in an attempt to provide that reuse. Yes, let's say you install a module that has a specific Django functionality from pypi, it will typically be installed as an app. It confuses newbies sometimes who think that they need to worry about how to divide up their application into different applications. 99% of the time they don't and shouldn't (putting everything in a single app called 'core' in your project is a pretty common pattern). Unfortunately the docs were never very clear on this. >This seems to me to be somewhat unpythonic, unnecessary complexity introduced right from the start - why not make the reuse optional?. The file system layout becomes confusing as a result. More generally, I would expect that the most minimal Django site would be a single file, that would seem most Pythonic, but it isn't designed like that. It can be made to behave like that if you really want: http://olifante.blogs.com/covil/2010/04/minimal-django.html http://olifante.blogs.com/covil/2010/04/minimal-django.html Most people aren't using Django for that though, so it's not widely advertised nor known. >It's heartening, however, to see I'm at least not _entirely_ wrong for having a gut unease about this, It's not like the framework is perfect. There's plenty wrong with it, but it largely managed not to fuck everything up, which is pretty rare.
dangayle 9y ago
Have you checkout out O'Reilly's Lightweight Django book? It basically goes through all of this, starting with a single file Django app.
- randomsearch 9y ago
  No! I read a lot of books, but missed that. Thanks!
- jwarren116 9y ago
  +1 for Lightweight Django. I had the pleasure of working with one of the authors. The book is well laid out and covers some really great topics that aren't always covered well elsewhere in the context of Django (i.e. Websocket services with Django). It's a great companion for Two Scoops of Django.
  - dangayle 9y ago
    In my opinion that book should be the starting point. The official tutorials have you way deep in the weeds with database migrations and stuff before you even have a basic understanding of where you're at and why. Worst "hello world" ever.
Osmose 9y ago
Where app directories are incredibly useful is in Django library development. They're a really convenient way to hook in models, static files, template helpers, etc. to an existing Django site, without forcing users to do a bunch of setup work when integrating your library. It's not a coincidence that most Django libraries have a very similar setup: 1. Install the package 2. Add the package to settings.INSTALLED_APPS (and middleware, etc). 3. Profit! For an instance of a Django site, apps are less useful, and the article's recommendation to be more careful when splitting your site into multiple apps is great advice.
- randomsearch 9y ago
  Yeah, I agree. Makes sense for libraries. So keep that separate for library writers, don't expose me to the complexity of that.
  - andybak 9y ago
    The benchmark I use is "Is it probable that this app might become shared between projects?" Using a Django 'app' for this makes sense. If it's generic enough that I might consider moving it into it's own repo then it's probably not going to be heavily intertwined with the rest of the project.
marcosdumay 9y ago
> why not make the reuse optional? Well, it is optional. You can just not divide you stuff and place everything inside the main app. But notice the article recreates the same layer of reuse inside the main app, and got some extra problems because of it (renaming the tables is probably caused by this). Django may need an extra paragraph at the tutorial, telling how it is ok to place everything on the main app when everything is application specific software, and how you should break generic functionality on other apps. By the way, what would be reverse Dunning-Kruger?
- randomsearch 9y ago
  > Well, it is optional What I mean is, you still have to have a "main" app. Why? I don't want to reuse it. So I don't want an "app". I just want to build my website. Stop making me put things in an unnecessarily complicated directory hierarchy. If it were Java, I wouldn't think twice about it, but it doesn't seem Pythonic. > reverse Dunning-Kruger Good question! Probably... illusory inferiority AKA imposter syndrome. I assumed Django knows best, but maybe it doesn't.
  - bmelton 9y ago
    Sure. One of the Djangoesque opinions is that if you're building a site, you'd like it to be maintainable, and that you'll probably add on to it. If you've been working in the Python ecosystem long enough to know that you're building a site that doesn't have those assumptions, you'll probably pick something different to start out with, like Flask or Bottle.
  - marcosdumay 9y ago
    > you still have to have a "main" app Well, it has to be organized some way. It happens to have the organization that gives you more flexibility, instead of forcing something that only fits very small sites onto you. That means you'll have an extra directory on your tree. Anyway, nobody uses only the main app. The article creates a big amount of problems by trying to only write on it, but I bet that even them use the Django's builtin apps. > AKA imposter syndrome Oh. Dunning-Kruger study detected that one too. It's within the effect: people tend to self evaluate into a small band near (but often under) the top, whatever their real evaluation is.
arctangent 9y ago
A good rule of thumb is that a Django "app" should be (1) standalone and (2) reusable. If you were to write a Django project to automate some complicated internal process at your particular company, it's likely that you'd break up the code into separate Python modules to help give it an easy to understand structure. But it's unlikely that any other company would necessarily benefit from any of those separate modules by themselves. However, let's say you come up with a clever idea about how to log errors in your project. Now, that's the sort of thing that other people might want to use in their own projects. If you were to restructure your code a bit and make it a bit more abstract then you could create a separate "app" that could be dropped into other Django projects.
tlp8800 9y ago
You could write a Django site in a single file if you wanted to. See this for an example: https://www.safaribooksonline.com/library/view/lightweight-django/9781491946275/ch01.html https://www.safaribooksonline.com/library/view/lightweight-d...
jasonthevillain 9y ago
It depends what you're doing. It shines on low level features like authentication and image cropping, where I need ways to do these things, and I can use them across the board. It breaks down when you have a lot of views and urls that are trying to stay close to their related models, but are too interconnected to have any real kind of separation of concerns. I find myself using all of the above. There's nothing wrong with building a small site in a single app, or nesting apps to provide a little more structure for how they're connected or separated.
neves 9y ago
I'm not affiliated with them, but you think seriously about Django, you should take a look at the book https://www.twoscoopspress.com/products/two-scoops-of-django-1-11https://www.twoscoopspress.com/products/two-scoops-of-django-1-11 https://www.twoscoopspress.com/products/two-scoops-of-django... I still have the 1.5 edition, but it helped me a lot. I'm think about upgrading. Do you recommend?

deleted 9y ago

[deleted]

cwisecarver 9y ago

I've got more than 5 years of experience with Django on a number of teams and at a couple of companies and in my experience almost everything in this article is completely incorrect. The only things I would agree with is the point about project layout and avoiding django's squashmigrations for the truncate the migrations table, delete the migrations, and create a new initial migration. Practically everything else in this article is wrong, in my opinion.

adamors 9y ago
Can you expand on what do you think is incorrect in the post?
- deleted 9y ago
  [deleted]
- cwisecarver 9y ago
  - FK/M2M across reusable, packaged apps is only bad if you don't match the interfaces correctly. See: almost every third-party Django app that is built to integrate with another application's models. - Sometimes you want the app concept just for organization. There's nothing bad about that and it makes sharing things inside of a project easier. - Explicitly naming your database tables doesn't make any sense. You're using an ORM. Accept it or don't use it. - Explicitly declaring through on a m2m field if you're not adding metadata to the relationship is pointless, but, if you're not using the table naming from Django I could see why they'd be invisible because there's no pattern to follow. - GenericForeignKeys are dangerous but not for any of the reasons listed. It's because they implicitly force a two way join which seems magic until it becomes debilitatingly slow. - The entire section on migrations leads me to believe that the first time the migrations are being run is on a production deploy. If you don't know SQL and you don't test your migrations prior to deployment, yes, it will be fairly difficult to determine what kind of performance/locking they're going to have. - No to Fat Models? This breaks down to "The framework we chose to use suggests a pattern, we also chose not to follow that pattern." That's fine if you want to do that but I wouldn't suggest it to others. - It's not hard to get signals to not fire in certain circumstances, you put the conditional in the signal callback like almost every other event-driven pattern. Also, bulk updating models doesn't fire signals in Django because save isn't called. Read the documentation. - Avoid using the ORM? Why choose a framework as complete as Django where 80% of the features are built around or on top of the ORM and then don't use it? - Caching complex objects makes cache invalidation hard. Well, yes, yes it does.

deleted 9y ago

[deleted]

tinix 9y ago

This reads way more like a list of Django caveats and anti-patterns, than a guide to running any kind of Python application at "scale". Maybe a sprinkling of good hygiene, but that has nothing to do with scalability (unless you're talking about developer scalability and cognitive overhead). Further, to suggest NOT using the ORM and to build out a middle layer on top of the ORM just for CRUD, is, well, insanity. On one hand the author is saying to cut down on bloat, and then on the other hand they are saying to add needless abstraction on an already bloated ORM. I don't use Django, but I do use Python for application development nearly every day, and I have to say... this article reads more like a list of (predictable and basic) problems their team had than a guide to scaling an application. Maybe they mean, how to scale their team, workflow, and their efficiency, NOT the application itself. The only thing I agree with here is the avoidance of some Django features... However, they should take it a step further, and just avoid Django in the first place. Also, am I the only one that has never needed to use "migrations" or any similar sorts of features? You're doing something very wrong and overcomplicating things if you seriously need to lean on migrations that heavily.

Alex3917 9y ago
> This reads way more like a list of Django caveats and anti-patterns, than a guide to running any kind of Python application at "scale". This isn't about scaling in the sense of handling more page views, but rather about scaling in the sense of having a codebase where the cost of adding each new feature and developer is O(n). The issue is that a lot of the 'recommended' Django patterns are terrible, and using all the features that are built-in would lead to a completely unmaintainable app. I don't understand the authors point about not using the ORM, in terms of why creating multiple tables inside a transaction doesn't solve the issue he's talking about. But aside from that all this advice is dead on, and these are exactly the same practices that I recommend to all consulting clients.
orf 9y ago
I agree with all of your points about the article, but... > However, they should take it a step further, and just avoid Django in the first place. Django is a tool, and like most tools it has a use. I find Django indispensable for writing specific kinds of applications, and it's admin interface is by far the best thing since sliced bread for internal/backoffice applications. It's not perfect by any means but it's amazing for quickly getting a CRUD app with an awesome administration interface up and running with minimal effort. Django rest framework, migrations and the amazing ecosystem (reversion, django-mptt, django-polymorphic, debug toolbar, django-currencies to name very few) are also what makes Django appealing and rather awesome. Making a blanket statement like 'Avoid Django' is rather silly, given all that. > Also, am I the only one that has never needed to use "migrations" or any similar sorts of features? I'm not sure why you put quotes around migrations as if it's some alien, obscure or weird feature. If you've never written a web application that needs migrations then you're not writing the kinds of applications (or indeed any 'serious' application) that would benefit from Django IMO.

johnthealy3 9y ago

I was feeling okay about this article until seeing this colossal punt: > That said, the real intention behind this pattern is to keep the API/view/controller lightweight and free of excessive logic, which is something we would strongly advocate. Having logic inside model methods is a lesser evil, but you may want to consider keeping models lightweight and focused on the data layer. To make this work, you will need to figure out a new pattern and put your business logic in some layer that is in between the data layer and the API/presentational layer. Having fat models is definitely a problem I'm having, and it's nice to see it's a problem for the author too, but the advice "figure it out" is presented without any explicit suggestions.

iamsomewalrus 9y ago
My team has run into the same problem in a large Rails app. We decided on a combination of service objects and data access objects (DAO) where appropriate. In a lot of cases this has really helped with the grep-ability of our codebase. Service objects allow us to encapsulate complex operations that touch several domains and tables through a simple API. They're just plain objects. DAOs fulfill a specific case of providing a simple CRUD API to a domain. We use a lot of DynamoDB so the DAO allows us to hide the complexity of reaching out to several tables in order to return a result. We also make sure that the DAO returns an immutable object that can't reach back into the datastore like an ActiveRecord object can (#save! methods come to mind). The difference between a service object and a DAO is a little blurry but is enforced through quick design discussion and code reviews.
sbov 9y ago
Procedural/functional code. Splitting your models up more by features than high level things (e.g. having a separate CustomerBilling rather than putting it in Customer). Component based architectures. Service layer to coordinate models. Etc. Also, this is/was a common problem (at least, the god class aspect) in games that the industry has slowly solved over the past 20 years, so that's another place you can look for inspiration. Further, unless you know you're going to scale in the beginning, I would recommend refactoring/evolving over time. It doesn't do anyone any favors by having 10 models to represent your Customer if your Customer is already relatively thin.

linkmotif 9y ago

Django is not meant to be run at scale. At least not horizontal scale. If you're not going to use the ORM, which prevents you from scaling horizontally, why are you using Django? Django can be scaled by getting large instances with lots of RAM, which is quite affordable these days. It can also be scaled with caching. But if you want horizontal scaling, don't use Django. That's not what it's for.

benwilber0 9y ago
There's nothing really inherent to Django that prevents it from scaling horizontally, not even the ORM. For the typical SQL read-heavy workload it will scale as well as anything else depending on the scalability of the backing RDBMS. Should always use proper HTTP response caching with something like Varnish or via 3rd-party CDN where possible.
- linkmotif 9y ago
  > depending on the scalability of the backing RDBMS As far as I know, none of the backing RDBMSs that the ORM supports are horizontally distributable. At least not officially, right? CockroachDB with its Postgres interface sounds good, or Citus? But as of today, are any of the supported DBs distributed?
  - benwilber0 9y ago
    MySQL Cluster [1] has been around a long time for applications that need linear scalability with ACID guarantees [1] https://www.mysql.com/products/cluster/ https://www.mysql.com/products/cluster/
ensignavenger 9y ago
How does the ORM prevent horizontal scaling?
- linkmotif 9y ago
  The ORM works against a relational database. The relational databases that Django supports are not distributed, right?

Osmose 9y ago

I appreciate this article for talking about scale in terms of project size and cognitive overhead, instead of in terms of traffic/computation/storage. Writing maintainable services isn't easy, and tips like these are painful to learn on your own.

goblin89 9y ago

I find that the safest way to migrate DB schema is gradually, spreading out intermediate steps across a few deploys. Migrate the schema creating the (initially unused) fields in advance. Step by step change your logic where it talks to the storage—start querying new field values with a fallback, use new fields for incoming data. Migrate the existing data. Remove the fallbacks. Ensure the old fields are not used at this point. Migrate the schema, finally deleting the old fields. Don’t rush, let every step reach production. In deployment sequence, apply migrations after the new code is already running. Goes without saying, monitor error rates for spikes. Yes, at some points in time your production DB schema won’t be fully normalized. In return for prolonged messiness you get smoother flow, no single fateful and stressful deployment that attempts to get it right all in one go. Migration handling is the only part of the article I find a bit foggy and debatable. The rest roughly resonates with what I’ve arrived at through my years of experience with Django web apps.

bradmwalker 9y ago

https://mobile.twitter.com/HackerNewsOnion/status/476835980863733761 https://mobile.twitter.com/HackerNewsOnion/status/4768359808...

joepour 9y ago

It's interesting that they're still using the default ORM. Stranger still they're letting the ORM scaffold tables (?!?) now they're at this size?

msmm 9y ago

It seems like author has problems with correctly architecting his apps and blames Django for his inability to do that. Only thing that makes sense is a tip for migrations.

St-Clock 9y ago

"Don’t cache Django models" This is too broad to be good advice. A typical solution to stale models is to increment a cache version key, effectively invalidating the old cache. Version keys can be coarse- or fine-grained. For example, you may have one version key for the entire application (supported by Django by default) and one version key per model or application (you have to roll your own solution). If your app's models change, you can increment the app version key and the next time you try to fetch a model instance from the cache, you will miss and instead fetch the instance from the DB.

nezo 9y ago

About ContentType : "If you ever move models between apps or rename your apps, you’re going to have to do some manual patching on this table" Thanks for this, just had a kind of struggle recently