7 ms·

Django project optimization guide – Part 1

158 points by dizballanze 9y ago

git-pull 9y ago
This is my first time hearing about Silk. Thanks for mentioning that, looks like a dream for profiling. GitHub for clicking: https://github.com/django-silk/silk https://github.com/django-silk/silk To add, if using Django Debug Toolbar, there is a line_profiler [1] extension for it: https://github.com/dmclain/django-debug-toolbar-line-profiler https://github.com/dmclain/django-debug-toolbar-line-profile... [1] https://github.com/rkern/line_profiler https://github.com/rkern/line_profiler
- avelis 9y ago
  I help maintain django-silk. I wasn't the original creator but that person has virtually no time to maintain it. I wanted to take a moment here and tell the HN community thanks for all the contributions/support the library has received to continue along. It can definitely use all the support it can get. Silk has so much potential but works as a free tool for many.
- Alex3917 9y ago
  Silk is great for making sure queries aren't run multiple times unnecessarily, but definitely don't run it is production. There are a lot of subtle threading issues that, as far as I know, haven't quite gotten resolved.
kennydude 9y ago
Excellent guide. Lots of things I didn't know about apart from Debug Toolbar! Previously I've hacked in printing out SQL statements, but the logging configuration is a lot nicer.
- dizballanze 9y ago
  Check out part 2! https://news.ycombinator.com/item?id=14694092 https://news.ycombinator.com/item?id=14694092
thehardsphere 9y ago
Simple stuff, but useful.
- dizballanze 9y ago
  Check out part 2! https://news.ycombinator.com/item?id=14694092 https://news.ycombinator.com/item?id=14694092
danpalmer 9y ago
I was quite excited to see this, but I feel like almost everything in this post was "table stakes" for production Django, and nothing particularly groundbreaking. Having said that, I work on a large Django codebase, maybe these things aren't as common in smaller projects. Looking forward to Part 2!
- Finnucane 9y ago
  As someone who is using Django for small projects and still learning my way around it, hints like removing unused middleware are appreciated.
- dizballanze 9y ago
  Hi, I have released part 2: https://news.ycombinator.com/item?id=14694092 https://news.ycombinator.com/item?id=14694092
danpalmer 9y ago
For the profiling, I'd also recommend `run_profile_server` from django-extensions, and Snakeviz as a visualiser for that data. It records cprofile reports per-request so can be useful for seeing what takes the time in a single request. Common hot-spots I've noticed (other than n+1 queries) is deserialisation of database rows into Django ORM objects - it's relatively expensive, and template rendering. I'd definitely choose Jinja2 on a new project for templating as it's more generally more capable and approximately 2x faster, the downside being the goal of "logic-less templates" is slightly harder to achieve.
- sharmi 9y ago
  Jinja2 has some crucial features missing in django templates, but we can add this to the django template as filters and functions. On the other hand, many django plugins are written to accommodate only django templates. By choosing jinja2, you will be severely limiting what plugins you can use. Jinja2 is more like a superset of django templates with some minor syntax modifications. So I would recommend you go with django templates and bring in jinja2 only if needed (which would be 0.001% of the time) This is from experience where I went with Jinja2 for one project and learned my lesson. Now for the next one I just went with the default. Some of django template's choices are bit of an hindrance but an affordable price for the interoperability it provides.
  - methodover 9y ago
    Curious comment. What Django plug-ins were you using that caused you problems with Jinja2 templates? We've been using Jinja2 at my startup since the beginning; in the four years I'be been working here I can't recall any issues with plug-ins using it.
    - elevensies 9y ago
      I'm not who you asked, but the first thing that comes to my mind is contrib.auth.views, which look for templates at the default locations or the locations you provide (as described here: https://docs.djangoproject.com/en/1.11/topics/auth/default/#module-django.contrib.auth.views https://docs.djangoproject.com/en/1.11/topics/auth/default/#... ) So using non-django templates would rule out this kind of thing right?
sgarg26 9y ago
Thank you for putting this document together
AshishGupta93 9y ago
I found this helpful for DB Optimization - https://medium.freecodecamp.com/django-performance-optimization-looking-for-the-bottlenecks-8583789e341b https://medium.freecodecamp.com/django-performance-optimizat...
mooneater 9y ago
Not open source, but I found newrelic useful to quickly profile and discover bottlenecks in Django code.

158 points by dizballanze 9y ago

This is my first time hearing about Silk. Thanks for mentioning that, looks like a dream for profiling. GitHub for clicking: https://github.com/django-silk/silk https://github.com/django-silk/silk To add, if using Django Debug Toolbar, there is a line_profiler [1] extension for it: https://github.com/dmclain/django-debug-toolbar-line-profiler https://github.com/dmclain/django-debug-toolbar-line-profile... [1] https://github.com/rkern/line_profiler https://github.com/rkern/line_profiler

avelis 9y ago
I help maintain django-silk. I wasn't the original creator but that person has virtually no time to maintain it. I wanted to take a moment here and tell the HN community thanks for all the contributions/support the library has received to continue along. It can definitely use all the support it can get. Silk has so much potential but works as a free tool for many.
Alex3917 9y ago
Silk is great for making sure queries aren't run multiple times unnecessarily, but definitely don't run it is production. There are a lot of subtle threading issues that, as far as I know, haven't quite gotten resolved.

kennydude 9y ago

Excellent guide. Lots of things I didn't know about apart from Debug Toolbar! Previously I've hacked in printing out SQL statements, but the logging configuration is a lot nicer.

dizballanze 9y ago
Check out part 2! https://news.ycombinator.com/item?id=14694092 https://news.ycombinator.com/item?id=14694092

thehardsphere 9y ago

Simple stuff, but useful.

dizballanze 9y ago
Check out part 2! https://news.ycombinator.com/item?id=14694092 https://news.ycombinator.com/item?id=14694092

danpalmer 9y ago

I was quite excited to see this, but I feel like almost everything in this post was "table stakes" for production Django, and nothing particularly groundbreaking. Having said that, I work on a large Django codebase, maybe these things aren't as common in smaller projects. Looking forward to Part 2!

Finnucane 9y ago
As someone who is using Django for small projects and still learning my way around it, hints like removing unused middleware are appreciated.
dizballanze 9y ago
Hi, I have released part 2: https://news.ycombinator.com/item?id=14694092 https://news.ycombinator.com/item?id=14694092

danpalmer 9y ago

For the profiling, I'd also recommend `run_profile_server` from django-extensions, and Snakeviz as a visualiser for that data. It records cprofile reports per-request so can be useful for seeing what takes the time in a single request. Common hot-spots I've noticed (other than n+1 queries) is deserialisation of database rows into Django ORM objects - it's relatively expensive, and template rendering. I'd definitely choose Jinja2 on a new project for templating as it's more generally more capable and approximately 2x faster, the downside being the goal of "logic-less templates" is slightly harder to achieve.

sharmi 9y ago
Jinja2 has some crucial features missing in django templates, but we can add this to the django template as filters and functions. On the other hand, many django plugins are written to accommodate only django templates. By choosing jinja2, you will be severely limiting what plugins you can use. Jinja2 is more like a superset of django templates with some minor syntax modifications. So I would recommend you go with django templates and bring in jinja2 only if needed (which would be 0.001% of the time) This is from experience where I went with Jinja2 for one project and learned my lesson. Now for the next one I just went with the default. Some of django template's choices are bit of an hindrance but an affordable price for the interoperability it provides.
- methodover 9y ago
  Curious comment. What Django plug-ins were you using that caused you problems with Jinja2 templates? We've been using Jinja2 at my startup since the beginning; in the four years I'be been working here I can't recall any issues with plug-ins using it.
  - elevensies 9y ago
    I'm not who you asked, but the first thing that comes to my mind is contrib.auth.views, which look for templates at the default locations or the locations you provide (as described here: https://docs.djangoproject.com/en/1.11/topics/auth/default/#module-django.contrib.auth.views https://docs.djangoproject.com/en/1.11/topics/auth/default/#... ) So using non-django templates would rule out this kind of thing right?

sgarg26 9y ago

Thank you for putting this document together

AshishGupta93 9y ago

I found this helpful for DB Optimization - https://medium.freecodecamp.com/django-performance-optimization-looking-for-the-bottlenecks-8583789e341b https://medium.freecodecamp.com/django-performance-optimizat...

mooneater 9y ago

Not open source, but I found newrelic useful to quickly profile and discover bottlenecks in Django code.