9 ms·

Django 4.0 will include a built-in Redis cache back end

130 points by stanulilic 5y ago

whoknowswhat11 5y ago
My understanding is Django 4.0 would include SUPPORT for a redis cache backend using a subclass of django.core.cache.backends.BaseCache Now they are shipping a built in Redis cache back-end? That would be so totally crazy that I am having trouble believing this headline. How about "Django 4.0 will include built-in Redis cache support"
- cat199 5y ago
  > Now they are shipping a built in Redis cache back-end? no. actual link as you surmised states: This PR aims at adding support for Redis to be used as a caching backend with Django. As redis is the most popular caching backend, adding it to django.core.cache module would be a great addition for developers who previously had to rely on the use of third party packages. so just a bad link title
  - tyingq 5y ago
    I was somewhat surprised to see that the django.core.cache module didn't support an mmap based cache. The default cache is per-process, so it's somewhat limited in effectiveness. Something like Redis solves that, but adds another running process to manage. In the PHP world, for example, acpu is a popular user-facing cache that uses mmap so that it works cross-process, then you can move to redis if you want to scale across more than one host.
AaronFriel 5y ago
Django is, of course, an open source project and the core team and contributors are largely free to work on whatever they prefer. However, as a user of Django I sincerely wish they would focus more on stability: Changing jsonb deserialization in a patch release, 3.1.1: https://code.djangoproject.com/ticket/31956#comment:16 https://code.djangoproject.com/ticket/31956#comment:16 Queryset .get() can cause OOM, ticket has gone back and forth, problem existed as recently as Django 2.X: https://code.djangoproject.com/ticket/6785 https://code.djangoproject.com/ticket/6785 Django's cascade options differ from the database's, are manually implemented: https://code.djangoproject.com/ticket/21961 https://code.djangoproject.com/ticket/21961 Can't update keys via ORM, related to the above: https://code.djangoproject.com/ticket/21265 https://code.djangoproject.com/ticket/21265 Inconsistent behavior around constraints checking in unit tests, can lead to inexplicable behavior and difficulty testing when a constraint is violated: https://code.djangoproject.com/ticket/11665 https://code.djangoproject.com/ticket/11665 In general, the ORM functionality and the inconsistency, bugs being introduced in patch releases has me looking elsewhere for all new development and recommending to other engineers the same. We have a running joke in our engineering team that every bug we hit in Django will have an open or "closed wontfix" issue 5-10 years old.
- ldiracdelta 5y ago
  I love Django, but version upgrades are a gigantic hassle.
  - hdjjhhvvhga 5y ago
    What would you suggest as an alternative for a new project?
    - ldiracdelta 5y ago
      Django is still my go-to. Specifically [Django-REST-Framework](https://www.django-rest-framework.org/ https://www.django-rest-framework.org/) with a front-end written with [react-static](https://github.com/react-static/react-static https://github.com/react-static/react-static). Django's ORM is so nice and the ecosystem around it rocks. Its biggest downside is painful upgrades. They don't really follow [semantic versioning](https://semver.org/ https://semver.org/)
      - matsemann 5y ago
        To me however the ORM is one of the most annoying things. Not a fan of ORMs in general, but Django's implementation promotes a usage pattern where django-models are passed around everywhere, leading to hard to untangle dependencies between services once they do literally anything with the objects.
        radus 5y ago
        I'm relatively new to Django (longtime Flask user), and what struck me about the Django ORM so far is that it seems much easier to write extremely low performance queries. I didn't really feel this way when using SQLAlchemy or peewee, but perhaps that's just my inexperience.
        brianwawok 5y ago
        I really enjoy the Django ORM, and it is perhaps the main reason I still use it today. Right mix of simplicity and power.
leetrout 5y ago
Interesting decision. The mailing list is usually full of "things are better served by third party packages" and divesting core of stuff like this. Ive not been following the mailing list, however, so I'll have to go back and see what the justification was...
spapas82 5y ago
This is a welcome addition but isn't such a big deal... There are great third-party packages (i.e https://github.com/jazzband/django-redis https://github.com/jazzband/django-redis) for whoever needs a redis cache backend in a project. Myself, I just use good ol' memcached for all my caching needs (https://docs.djangoproject.com/en/3.2/topics/cache/#memcached https://docs.djangoproject.com/en/3.2/topics/cache/#memcache...). It is rock solid and has never failed me till now!
JonathanBeuys 5y ago
What is being cached? I don't understand what there is to cache in a web framework. Is it to cache rendered pages without dynamic content? If so, I think it is better to let a CDN handle that. Instead of the server that runs the web application. What am I missing?
- TylerE 5y ago
  Generally the approach is fragment caching... so you're caching some block of HTML that would otherwise hit the DB (Think something like a block of configurable sidebar links, or a "latest posts" block. HTML, not assets. Some such setups (like one I wrote for a busy Drupal newspaper site way back when) can support clearing relevant cache entries on DB write, so you're essentially never serving stale data, but can save _substantial_ DB load - and possibly webserver load too, depending on how computationally intensive the HTML generation is.
  - jayd16 5y ago
    I guess this is easier than read replicas? I can't imagine there's too much work to be saved by skipping the component building.
    - TylerE 5y ago
      Yes, if the queries involve lots of fiddly calculation (for instance, the newspaper system did a lot of fancy stuff around calculating good line breaks and wrapping, or the queries themselves are complicated, involve subqueries, many joins, etc.
    - zepolen 5y ago
      DB lookup + rendering vs cache lookup + concatenating. Yes it adds up.
- BiteCode_dev 5y ago
  First, there are people that don't want to use a CDN (don't trust them with censorship), or can't (local network). But you can cache other things that are not the entire page itself: - database queries - user sessions - api calls - page fragment - function memoization What's more, cache backends have the ability to expire things, which can be helpful for all sort of stuff.
benatkin 5y ago
At surface level, Redis isn't currently on this page, but will be there when the 4.0 version comes out: https://docs.djangoproject.com/en/3.2/topics/cache/ https://docs.djangoproject.com/en/3.2/topics/cache/ However, it seems this will enable the community to establish using redis as a best practice. So it will be a bigger deal than adding just another cache backend. This is good because the default is local-memory caching, and redis can easily beat that. https://docs.djangoproject.com/en/3.2/topics/cache/#local-memory-caching https://docs.djangoproject.com/en/3.2/topics/cache/#local-me...
nicwolff 5y ago
Nuts, I was hoping that this was an evolution of https://github.com/Suor/django-cacheops https://github.com/Suor/django-cacheops which uses Redis to cache and invalidate ORM result sets by an abstracted representation of filter values. It works, but has a lot of magic (as is the Django way) and takes some tending-to in production.
- rufugee 5y ago
  Interesting...I recall that Django always set itself apart from Rails by not having a lot of magic. Has this changed?
  - hgamaral 5y ago
    Django has always had some magic, although much less than ruby/rails. That kind of magic is the reason why it is a pain to package a Django project with py2exe, for instance.
BiteCode_dev 5y ago
Sensible decision, although hardly a surprise or a killer feature for something that is a major version bump. I assume the async story is going to be more fun. By the way, if you are looking for a good key/value store with expiration without the need to setup redis, have a look at the excellent diskcache: https://pypi.org/project/diskcache/ https://pypi.org/project/diskcache/ It is based on sharded sqlite files, and it's surprisingly capable, even in multiprocessing based env like a django site. Sqlite is really showing up in all sort of use cases these days, like huey, a task queue that doesn't need any broker to be installed to be useful: https://pypi.org/project/diskcache/ https://pypi.org/project/diskcache/ Now things can get fun... With supervisord (supervisord.org) you can manage multiple processes, and with fastapi + starlette, you can serve your WSGI site without a proxy pass, including static sites: https://fastapi.tiangolo.com/advanced/wsgi/ https://fastapi.tiangolo.com/advanced/wsgi/ Then with shiv you can bunddle all that in a pyz file: shiv.readthedocs.io/ Meaning you can now just upload a pyz file to your server, run python on it, and that's it, you have a full stack deployed, with a web framework, db, caching, task queues, static file serving, which can handle around 100k visitor each day without a sweat. No more infra needed. Just Python code, all bundle in one file. That's really neat.
- zentrus 5y ago
  Totally agree on hoping there is at least some movement forward on async--particularly async ORM. I love Django and DRF. It is my go-to for a lot of things, but the lack of async support is increasingly a problem for more organizations. Honestly, it would not surprise me that if a solid, kitchen sink web framework emerges in golang, Django will be abandoned rather quickly. 2 cents.
  - arcturus17 5y ago
    I love Django but it seems like such a limitation for certain workloads. You can often work around some of these issues with Celery but still… .Net already offers a kitchensink with async, it’s growing a lot and is much more attractive than a decade ago.
  - midrus 5y ago
rafaelturk 5y ago
Redis everywhere!
wilsonfiifi 5y ago
I think Django needs to "fix" serving static assets in production without the need to muck about with witenoise or nginx settings.
crb002 5y ago
Django also needs to pin a default Postgres container image, and default NGINX image. The default stack should be extensively end-to end tested and optimized, including an optional patched kernel, cgroup2 settings, and untrusted nsjail worker process for handling media files.
apple4ever 5y ago
This is cool. Be nice if Wordpress did the same.

130 points by stanulilic 5y ago

My understanding is Django 4.0 would include SUPPORT for a redis cache backend using a subclass of django.core.cache.backends.BaseCache Now they are shipping a built in Redis cache back-end? That would be so totally crazy that I am having trouble believing this headline. How about "Django 4.0 will include built-in Redis cache support"

cat199 5y ago
> Now they are shipping a built in Redis cache back-end? no. actual link as you surmised states: This PR aims at adding support for Redis to be used as a caching backend with Django. As redis is the most popular caching backend, adding it to django.core.cache module would be a great addition for developers who previously had to rely on the use of third party packages. so just a bad link title
- tyingq 5y ago
  I was somewhat surprised to see that the django.core.cache module didn't support an mmap based cache. The default cache is per-process, so it's somewhat limited in effectiveness. Something like Redis solves that, but adds another running process to manage. In the PHP world, for example, acpu is a popular user-facing cache that uses mmap so that it works cross-process, then you can move to redis if you want to scale across more than one host.

AaronFriel 5y ago

Django is, of course, an open source project and the core team and contributors are largely free to work on whatever they prefer. However, as a user of Django I sincerely wish they would focus more on stability: Changing jsonb deserialization in a patch release, 3.1.1: https://code.djangoproject.com/ticket/31956#comment:16 https://code.djangoproject.com/ticket/31956#comment:16 Queryset .get() can cause OOM, ticket has gone back and forth, problem existed as recently as Django 2.X: https://code.djangoproject.com/ticket/6785 https://code.djangoproject.com/ticket/6785 Django's cascade options differ from the database's, are manually implemented: https://code.djangoproject.com/ticket/21961 https://code.djangoproject.com/ticket/21961 Can't update keys via ORM, related to the above: https://code.djangoproject.com/ticket/21265 https://code.djangoproject.com/ticket/21265 Inconsistent behavior around constraints checking in unit tests, can lead to inexplicable behavior and difficulty testing when a constraint is violated: https://code.djangoproject.com/ticket/11665 https://code.djangoproject.com/ticket/11665 In general, the ORM functionality and the inconsistency, bugs being introduced in patch releases has me looking elsewhere for all new development and recommending to other engineers the same. We have a running joke in our engineering team that every bug we hit in Django will have an open or "closed wontfix" issue 5-10 years old.

ldiracdelta 5y ago
I love Django, but version upgrades are a gigantic hassle.
- hdjjhhvvhga 5y ago
  What would you suggest as an alternative for a new project?
  - ldiracdelta 5y ago
    Django is still my go-to. Specifically [Django-REST-Framework](https://www.django-rest-framework.org/ https://www.django-rest-framework.org/) with a front-end written with [react-static](https://github.com/react-static/react-static https://github.com/react-static/react-static). Django's ORM is so nice and the ecosystem around it rocks. Its biggest downside is painful upgrades. They don't really follow [semantic versioning](https://semver.org/ https://semver.org/)
    - matsemann 5y ago
      To me however the ORM is one of the most annoying things. Not a fan of ORMs in general, but Django's implementation promotes a usage pattern where django-models are passed around everywhere, leading to hard to untangle dependencies between services once they do literally anything with the objects.
      - radus 5y ago
        I'm relatively new to Django (longtime Flask user), and what struck me about the Django ORM so far is that it seems much easier to write extremely low performance queries. I didn't really feel this way when using SQLAlchemy or peewee, but perhaps that's just my inexperience.
      - brianwawok 5y ago
        I really enjoy the Django ORM, and it is perhaps the main reason I still use it today. Right mix of simplicity and power.

leetrout 5y ago

Interesting decision. The mailing list is usually full of "things are better served by third party packages" and divesting core of stuff like this. Ive not been following the mailing list, however, so I'll have to go back and see what the justification was...

spapas82 5y ago

This is a welcome addition but isn't such a big deal... There are great third-party packages (i.e https://github.com/jazzband/django-redis https://github.com/jazzband/django-redis) for whoever needs a redis cache backend in a project. Myself, I just use good ol' memcached for all my caching needs (https://docs.djangoproject.com/en/3.2/topics/cache/#memcached https://docs.djangoproject.com/en/3.2/topics/cache/#memcache...). It is rock solid and has never failed me till now!

JonathanBeuys 5y ago

What is being cached? I don't understand what there is to cache in a web framework. Is it to cache rendered pages without dynamic content? If so, I think it is better to let a CDN handle that. Instead of the server that runs the web application. What am I missing?

TylerE 5y ago
Generally the approach is fragment caching... so you're caching some block of HTML that would otherwise hit the DB (Think something like a block of configurable sidebar links, or a "latest posts" block. HTML, not assets. Some such setups (like one I wrote for a busy Drupal newspaper site way back when) can support clearing relevant cache entries on DB write, so you're essentially never serving stale data, but can save _substantial_ DB load - and possibly webserver load too, depending on how computationally intensive the HTML generation is.
- jayd16 5y ago
  I guess this is easier than read replicas? I can't imagine there's too much work to be saved by skipping the component building.
  - TylerE 5y ago
    Yes, if the queries involve lots of fiddly calculation (for instance, the newspaper system did a lot of fancy stuff around calculating good line breaks and wrapping, or the queries themselves are complicated, involve subqueries, many joins, etc.
  - zepolen 5y ago
    DB lookup + rendering vs cache lookup + concatenating. Yes it adds up.
BiteCode_dev 5y ago
First, there are people that don't want to use a CDN (don't trust them with censorship), or can't (local network). But you can cache other things that are not the entire page itself: - database queries - user sessions - api calls - page fragment - function memoization What's more, cache backends have the ability to expire things, which can be helpful for all sort of stuff.

benatkin 5y ago

At surface level, Redis isn't currently on this page, but will be there when the 4.0 version comes out: https://docs.djangoproject.com/en/3.2/topics/cache/ https://docs.djangoproject.com/en/3.2/topics/cache/ However, it seems this will enable the community to establish using redis as a best practice. So it will be a bigger deal than adding just another cache backend. This is good because the default is local-memory caching, and redis can easily beat that. https://docs.djangoproject.com/en/3.2/topics/cache/#local-memory-caching https://docs.djangoproject.com/en/3.2/topics/cache/#local-me...

nicwolff 5y ago

Nuts, I was hoping that this was an evolution of https://github.com/Suor/django-cacheops https://github.com/Suor/django-cacheops which uses Redis to cache and invalidate ORM result sets by an abstracted representation of filter values. It works, but has a lot of magic (as is the Django way) and takes some tending-to in production.

rufugee 5y ago
Interesting...I recall that Django always set itself apart from Rails by not having a lot of magic. Has this changed?
- hgamaral 5y ago
  Django has always had some magic, although much less than ruby/rails. That kind of magic is the reason why it is a pain to package a Django project with py2exe, for instance.

BiteCode_dev 5y ago

Sensible decision, although hardly a surprise or a killer feature for something that is a major version bump. I assume the async story is going to be more fun. By the way, if you are looking for a good key/value store with expiration without the need to setup redis, have a look at the excellent diskcache: https://pypi.org/project/diskcache/ https://pypi.org/project/diskcache/ It is based on sharded sqlite files, and it's surprisingly capable, even in multiprocessing based env like a django site. Sqlite is really showing up in all sort of use cases these days, like huey, a task queue that doesn't need any broker to be installed to be useful: https://pypi.org/project/diskcache/ https://pypi.org/project/diskcache/ Now things can get fun... With supervisord (supervisord.org) you can manage multiple processes, and with fastapi + starlette, you can serve your WSGI site without a proxy pass, including static sites: https://fastapi.tiangolo.com/advanced/wsgi/ https://fastapi.tiangolo.com/advanced/wsgi/ Then with shiv you can bunddle all that in a pyz file: shiv.readthedocs.io/ Meaning you can now just upload a pyz file to your server, run python on it, and that's it, you have a full stack deployed, with a web framework, db, caching, task queues, static file serving, which can handle around 100k visitor each day without a sweat. No more infra needed. Just Python code, all bundle in one file. That's really neat.

zentrus 5y ago
Totally agree on hoping there is at least some movement forward on async--particularly async ORM. I love Django and DRF. It is my go-to for a lot of things, but the lack of async support is increasingly a problem for more organizations. Honestly, it would not surprise me that if a solid, kitchen sink web framework emerges in golang, Django will be abandoned rather quickly. 2 cents.
- arcturus17 5y ago
  I love Django but it seems like such a limitation for certain workloads. You can often work around some of these issues with Celery but still… .Net already offers a kitchensink with async, it’s growing a lot and is much more attractive than a decade ago.
- midrus 5y ago

rafaelturk 5y ago

Redis everywhere!

wilsonfiifi 5y ago

I think Django needs to "fix" serving static assets in production without the need to muck about with witenoise or nginx settings.

crb002 5y ago

Django also needs to pin a default Postgres container image, and default NGINX image. The default stack should be extensively end-to end tested and optimized, including an optional patched kernel, cgroup2 settings, and untrusted nsjail worker process for handling media files.

apple4ever 5y ago

This is cool. Be nice if Wordpress did the same.