5 ms·
The Practical Guide to Scaling Django
- feydaykyn 2y agoThe snippets are not false, but there's so much context missing it's easy to worsen the situation, especially for beginners which seem to be the target audience. First, this guide should emphasize the need to measure before doing anything : django silk, django debug toolbarsm, etc. Of course, measure after the optimizations too, and measure in production with an apm. Second, some only work sometimes : select_related / prefetch_related / iterator will lead to giga SQL queries with nested joins all over the place, and ends by exploding ram usage. It will help at first, but soon enough one will pay any missing sql knowledge or naive relationships. Third, caching without taking the context into account will probably lead to data corruption one way or another. Debugging stale cache issues is not fun, since you cannot reproduce them easily. Fourth, celery is a whole new world, which requires workers, retry and idempotent logic, etc. Finally, scaling is also about code: architecture, good practices, basic algorithm, etc I'll end by linking to more complete resources : - https://docs.djangoproject.com/en/5.1/topics/performance/ https://docs.djangoproject.com/en/5.1/topics/performance/ - https://loadforge.com/guides/the-ultimate-guide-to-django-performance-best-practices-for-scaling-and-optimization https://loadforge.com/guides/the-ultimate-guide-to-django-pe... - https://medium.com/django-unleashed/django-application-performance-optimization-a-checklist-63e2c6d69e4e https://medium.com/django-unleashed/django-application-perfo...
- pastage 2y ago> scaling is also about code Which is darn hard if you are a beginner in a framework, loops in loops still bites me after reality does the integration test for me. This is especially true when you try to do a simple thing as a beginner. By scaling I am just talking about normal production, going from 2 developers to a couple of thousand customers.
- feydaykyn 2y agoTo mind it's a part where the Django guide could be expanded a bit, in order to help scaffold a simple but "open to the future" code architecture. For instance I would warn against fat models and propose a very light "service pattern" architecture
- sjducb 2y ago100% Always measure before you performance optimise. Lots of times the “fast” solution is slower. If you need a fast solution then add an integration test so that the system stays fast.
- bdzr 2y ago> Third, caching without taking the context into account will probably lead to data corruption one way or another. One can only hope it's data corruption and not a sensitive data leak.
- vldmrs 2y agoAt first I wanted to criticize the post, buuut after finishing reading it I actually liked it. Very concise and practical ps - I didn’t know about template “cache” directive
- danpalmer 2y agoFWIW, I'd advise against template caching. It's awkward to cache bust, and a network round trip to your cache will almost certainly be more expensive than the Python operations to render the template, even with stock Django templating which is slow. The only place it's possible worth it is if you do a lot of database queries from your template rendering, and you're therefore caching database results (as rendered text). In that case, it's an easy patch. However a much better solution is to fetch all database results up front. In my previous company we had a very significant Django codebase with plenty of templating, and found that using the templating system for (lazy loaded) database queries or caching was more hassle than it was worth and avoided it as much as possible. Treating template rendering as a pure CPU bound function was always better.
- nprateem 2y agoIt'd be faster to retrieve from a cache than to make a round-trip to a DB to get the data needed for templating.
- telgareith 2y ago[assumption]
- danpalmer 2y agoMy point was that you shouldn't be doing DB queries in the template. If you're doing the DB queries before templating then you should also be doing the cache queries before templating too.
- bdzr 2y agoI haven't used it, but I think this is well targeted by https://github.com/dabapps/django-zen-queries https://github.com/dabapps/django-zen-queries.
- vundercind 2y agoProbably 80% of notable performance problems I’ve seen in the kinds of systems that things like Django and Ruby get used for have been terrible queries or patterns of use for databases (I’ve seen 1,000x or worse costs for this versus something more-correct) and nearly all of the other 20% has been areas that plainly just needed some pretty straightforward caching. The nice thing about that is that spotting those, and the basic approach to fixing them, if not the exact implementation details, are cross-platform skills that apply basically anywhere. I actually can’t recall any other notable performance problems in those sorts of systems, over the years. Those are so common and the fixes so effective I guess the rest has just never rated attention. I’ve seen different problems in long-lived worker processes though (“make it streaming—everything becomes streaming when scale gets big enough” is the usual platform-agnostic magic bullet in those cases) A bunch of TFA is basically about those things, so I’m not correcting it, more like nodding along. Oh wait I just thought of another I’ve seen: serving large files through a scripting language, as in, reading it in and writing it back out with a scripting language. You run into trouble at even modest scale. There’s a magic response header for that, make Nginx or Apache or whatever serve it for you, it’s a fix that’s typically deleting a bunch of code and replacing it with one or two lines. Or else just use s3 and maybe signed URLs like the rest of the world. Problem solved.
- megaman821 2y agoI have had to combine files into a zipped file on demand before. It is hard to avoid the inherent slowness of that.
- ch4s3 2y agoInteresting, was there a business reason to not do that in the background somewhere?
- megaman821 2y agoYeah, very non-technical users that won't check their email or click on a notification when the zip file is ready for them.
- almost 2y agoThis sort of article seems perfectly poised to be useless to beginners (no context, doesn't tell you how to use the things) and experts (no nuance, just listing basic features) alike. Who is it for? Why does it exist? Why is it posted here?
- jerrygenser 2y agoSeo and marketing to sell their product is the reason it exists.
- 8organicbits 2y agoDon't store secrets in settings.py. Typically you'd inject those from secrets management as environment variables.
- halfcat 2y agoAnd also, when possible, try to use a key manager over environment variables. Using a library like keyring [1] is a significant step up from a .env file sitting in your dev environment. In other words: - Store secrets in settings.py (bad) - Store secrets in .env file (better) - Store secrets in OS-level key vault (even better) When the secrets are in a plaintext .env file, that file can get leaked in many non-obvious ways. Your antivirus uploads a copy, your IT department runs backups, someone on the team clones your git repo to a OneDrive/Dropbox folder and puts the .env file there. Then any of those services that has a leak, or any of the services those services use has a leak (improperly configured S3 bucket, etc), your secrets are leaked. [1] https://github.com/jaraco/keyring https://github.com/jaraco/keyring
- slashnode 2y agoThe basic outline of this post isn’t bad, the problem is that’s all there is - a basic outline. If you haven’t dealt with these problems before the checklists are meaningless. If you HAVE dealt with these problems before the checklists are redundant
- kehrazy 2y agoAn amazing movie.
- sharmi 2y ago[dead]
- tmarice 2y agoIf you're on Postgres, StringAgg and ArrayAgg are nice alternatives to prefetch_related to avoid building Python model instances and waste memory. I wrote a short blog post on recent optimizations we did on our Django codebase: https://tmarice.dev/blog/better-living-through-optimized-django/ https://tmarice.dev/blog/better-living-through-optimized-dja...
- rudasn 2y agoNice post,thanks for sharing! It would be nice to include the generated sql queries along with the code samples though. I've been on a similar path recently and being able to see the queries was really helpful (even the ones that failed!).
- The_Amp_Walrus 2y agoI did a more detailed yt video on django query optimisation (mostly for ppl new to the framework) for those interested https://www.youtube.com/watch?v=9uoI6pvuvYs https://www.youtube.com/watch?v=9uoI6pvuvYs
- medo-bear 2y agoSo I shouldnt have my business logic done in django templates?
- deleted 2y ago[deleted]