6 ms·

Zulip has been powered by Django since the very early days of its development with Django 1.4, back in 2012. As a reasonably mature web application with signifi

Zulip has been powered by Django since the very early days of its development with Django 1.4, back in 2012. As a reasonably mature web application with significant scale, we're at the stage in many companies' development where one starts to rip out more and more of the web framework to optimize things or just make them work the way we want. (E.g. while I was at Dropbox in early 2016, we discovered we only had about 600 lines of code left from the original Pylons framework that actually ran). One of the things that has been really fantastic about Django is that we're still happily using it for the vast majority of code in the project, and every time Django comes out with a new release, I read the changelog and get excited about several improvements that actually make my life better. Overall I think we've gotten a ton of value out of Python and Django and would recommend it to anyone starting a new full-featured web application project today. My only frustration with this release announcement is dropping Python 3.5 support now. Because Python 3.5 is not EOL and used in LTS OS releases that have vendor support through 2021, this forces us to choose between our policy of supporting vendor OS releases until they reach EOL, not upgrading to Django 3 for the next year or more, or shipping our own Python on Ubuntu Xenial.

NickBusey 7y ago
Thanks for your work on Zulip, I tend to be logged into at least 3 instances at any given time. Love the treaded model.
georgyo 7y ago
Zulip is quite amazing and I really like the product, however I somewhat dislike that the installer is a complete machine take over with many hard-coded paths. Supporting only Ubuntu releases is one thing, but the code base actively fights installing it on anything else but the supported distributions (ubuntu). At which point, why even have a installer, just ship an image. Since Zulip has so many moving components, I would love to see it be installed with something like Nix. Doing so would allow it to be installed in a consistent way across many different OSes, and decouple your reliance on software provided by the OS. It would also make upgrades seamless. You can install nix on any single linux distribution, not just nixos, and since your installer already hard requires running as root, you can install nix and then install zulip in nix.
- tabbott 7y ago
  We've supported Debian in production since 2018, and we're close to supporting RHEL as well. And even if we only supported Ubuntu, there's a lot of benefit to providing an installer that works on every Ubuntu LTS release; most larger organizations have additional monitoring/security/backups/etc. software they want to run on every system they manage, and to make their own decisions about when to move their organization to newer major OS releases. You can also use our Docker image if you want that, but a lot of organizations don't :). I wouldn't say the installer is actually a complete machine takeover; we're pretty cautious about using the system versions of dependencies shipped by the OS vendor. But we do __document__ that you should treat it as one, mainly for support reasons. The support experience that we currently provide is that where possible we respond to, investigate, and debug all reports of issues installing Zulip, with the goal of making it Just Work (this includes working to give nice error messages for common user errors). Before we added those warnings in our documentation, we got tons of reports from users who were trying to do things like install Zulip in a shared hosting environment with other software that doesn't expect to share its database (etc.) with something else, and where they don't have root access. Also, investigation often determined the system in question had low RAM/disk resource limits from the shared hosting environment that meant running Zulip on that system wasn't going to work in any case. At one time, that sort of shared hosting setup accounted for the majority of reports from folks having issues installing Zulip. Investigating those reports was a huge waste of time for our development community. We solved that problem by making our documentation recommend installing Zulip on a dedicated system. This also provides a lot of benefit in terms of our being able to do things like tune the configuration for postgres and memcached to allocate resources in an appropriate way for a Zulip installation with the system's RAM allocation. There is occasionally discussion in the community of adjusting our approach here, but given how easy it is to get a VM or container these days, we're currently happy with guiding users to the dedicated VM/container path that we can guarantee works ~100% of the time and provide a great installation experience.
dylan-m 7y ago
One of my favourite things with Django has been their mature approach to changes. Want to be guaranteed a stable upgrade? Step 1: Fix deprecation notices. Step 2: Upgrade. (If you're upgrading over a few releases, return to step 1). Coming from other frameworks it's a delight :) At least that's been my experience with smaller projects. I'm curious, does that line up with your experience with Zulip, too?
- tabbott 7y ago
  Yeah, that matches my experience; their documentation on upgrades is excellent, and there's generally only issues not covered by their deprecation notices in places where you were doing something not supported by the documented API (E.g. monkey-patching, subclassing to override something, etc.), and even then, they often comment if it's a common type of modification.
- andrewingram 7y ago
  I've personally never had a painless Django upgrade (except for patch releases). I always seem to fall into that grey area where I'm relying on undocumented behaviour of documented features (e.g. not using framework internals per se). I remember one Django release which changed the behaviour of views that returned files to download (it was probably Django 1.4), but there was no mention of that in the release notes. Sometimes I've knowingly relied on framework internals in ways I expected to eventually break, but those situations don't represent the bulk of my experience with painful upgrades.
Znafon 7y ago
> My only frustration with this release announcement is dropping Python 3.5 support now. Because Python 3.5 is not EOL and used in LTS OS releases that have vendor support through 2021, this forces us to choose between our policy of supporting vendor OS releases until they reach EOL, not upgrading to Django 3 for the next year or more, or shipping our own Python on Ubuntu Xenial. To be fair, I think the next LTS release of Django will be the last minor release before 4.0, 3.0 is not LTS.
nine_k 7y ago
If you don't mind: what are the key things that make Django useful for a project like Zulip? How much does the framework help you, and where? What are the places where you have to go against the grain, if any?
- tabbott 7y ago
  I'd say the big thing is helping us avoid reinventing the wheel. There's a lot of problems that every webapp has, and Django provides a system for solving most of them. I particularly enjoy their migrations and automated testing systems. When we need to integrate something that a lot of other people have, whether it be 2FA or SAML or LDAP or webpack, there's generally a reasonable library for it (or often, a reasonable way to do it built in, and Django's documentation is great so it's easy to figure that out). While there are some land mines with any ORM, the Django ORM does allow us to write the vast majority of our database operations with clean Pythonic code, rather than going to raw SQL. We need SQL in a few performance-intensive code paths, but it's the exception, rather than the rule, and the ORM is constantly getting better. And in contrast with some other ecosystems I've worked with, the libraries generally are well-written, well-tested, and stable, so even if they're not well-maintained because the old maintainer changed jobs or got busy or whatever, it doesn't feel like a burden to fork or adopt the library. I'm not particularly happy with Django forms or Django REST framework with how they organization validation code. For the former, I think the problem is just kinda messy; we mostly solve it by only using Django forms for the authentication code paths where we can reuse its libraries. For the latter, we have a framework we built that I'm very happy with and we plan to spin out for other projects to use when we have a bit of time. There's some documentation here: https://zulip.readthedocs.io/en/latest/tutorials/writing-views.html#request-variables https://zulip.readthedocs.io/en/latest/tutorials/writing-vie... for those curious; though that doesn't cover cool details like the mypy integration we did more recently.
  - gshdg 7y ago
    Ooooh, I'd be very interested in seeing that spun out. I also find DRF borderline infuriating from a design standpoint.
nickjj 7y ago
Hey, if you ever want to do a podcast episode that goes into your tech stack (why you made certain choices, how it's been going, deploying, etc.) let me know. I'd love to have you on as a guest for a podcast I recently started called Running in Production. Dan from https://realpython.com https://realpython.com and I just did one recently. That site runs Django too. You can check that out here: https://runninginproduction.com/podcast/4-real-python-is-one-of-the-largest-python-learning-platforms-around https://runninginproduction.com/podcast/4-real-python-is-one... Click the "Become a guest" button to get started if you want. Note: If you're reading this and you're not from Zulip and you are running Django in production (or any stack) hit me up too.
k_sze 7y ago
> My only frustration with this release announcement is dropping Python 3.5 support now. Because Python 3.5 is not EOL and used in LTS OS releases that have vendor support through 2021, this forces us to choose between our policy of supporting vendor OS releases until they reach EOL, not upgrading to Django 3 for the next year or more, or shipping our own Python on Ubuntu Xenial. One of the things I have learned early on is to not depend on the system-bundled Python. I either just use pyenv to install whatever Python version I like, or I use a Python Docker image (https://hub.docker.com/_/python/ https://hub.docker.com/_/python/). That way, I'm decoupled from the system-bundled Python and it's easier to test out new Python versions.
agildehaus 7y ago
> My only frustration with this release announcement is dropping Python 3.5 support now. Because Python 3.5 is not EOL and used in LTS OS releases that have vendor support through 2021 ... Django has an LTS release (2.2) that supports Python 3.5 just fine. Doesn't it make sense to use the LTS Django on an LTS operating system? There are plenty of ways to use newer python, too. Docker, PPAs, etc.
takeda 7y ago
> My only frustration with this release announcement is dropping Python 3.5 support now. Because Python 3.5 is not EOL and used in LTS OS releases that have vendor support through 2021, this forces us to choose between our policy of supporting vendor OS releases until they reach EOL, not upgrading to Django 3 for the next year or more, or shipping our own Python on Ubuntu Xenial. Don't ever rely on python that is included with OS. It makes upgrades really difficult. This is especially true with RedHat and CentOS where they are way behind. There are other issues, like dependencies, you are bound then to only using the dependencies version they provide. If there is a bug in one, tough luck, you can try to upgrade but then you're risking that you will break system tools (yum etc). Just assume that the system python is for the system. Use the right version you need (if you use redhat/centos I highly recommend ius.us repo, they make sure that it can be installed side by side and doesn't conflict with anything in the system) then create virtualenv and install your application there. That way you have full control of the python version you use and all of your application dependencies. Also when ops team wants you to move from version 7 to 8 of the OS, your application should work the same without no or very small changes.
- antb123 7y ago
  And you can assume that no security issues have entered python no matter which version? People use the OS version for security audit reasons.
  - takeda 7y ago
    The ius.io repo that I mentioned patches the version of Python. What it lets you do is you can install any major version you want (in fact you can install multiple at the same time and they won't conflict). So you can chose whether you want python 3.5.x, 3.6.x, 3.7.x, 3.8.x or maybe still you want 2.7.x.

NickBusey 7y ago

Thanks for your work on Zulip, I tend to be logged into at least 3 instances at any given time. Love the treaded model.

georgyo 7y ago

Zulip is quite amazing and I really like the product, however I somewhat dislike that the installer is a complete machine take over with many hard-coded paths. Supporting only Ubuntu releases is one thing, but the code base actively fights installing it on anything else but the supported distributions (ubuntu). At which point, why even have a installer, just ship an image. Since Zulip has so many moving components, I would love to see it be installed with something like Nix. Doing so would allow it to be installed in a consistent way across many different OSes, and decouple your reliance on software provided by the OS. It would also make upgrades seamless. You can install nix on any single linux distribution, not just nixos, and since your installer already hard requires running as root, you can install nix and then install zulip in nix.

tabbott 7y ago
We've supported Debian in production since 2018, and we're close to supporting RHEL as well. And even if we only supported Ubuntu, there's a lot of benefit to providing an installer that works on every Ubuntu LTS release; most larger organizations have additional monitoring/security/backups/etc. software they want to run on every system they manage, and to make their own decisions about when to move their organization to newer major OS releases. You can also use our Docker image if you want that, but a lot of organizations don't :). I wouldn't say the installer is actually a complete machine takeover; we're pretty cautious about using the system versions of dependencies shipped by the OS vendor. But we do __document__ that you should treat it as one, mainly for support reasons. The support experience that we currently provide is that where possible we respond to, investigate, and debug all reports of issues installing Zulip, with the goal of making it Just Work (this includes working to give nice error messages for common user errors). Before we added those warnings in our documentation, we got tons of reports from users who were trying to do things like install Zulip in a shared hosting environment with other software that doesn't expect to share its database (etc.) with something else, and where they don't have root access. Also, investigation often determined the system in question had low RAM/disk resource limits from the shared hosting environment that meant running Zulip on that system wasn't going to work in any case. At one time, that sort of shared hosting setup accounted for the majority of reports from folks having issues installing Zulip. Investigating those reports was a huge waste of time for our development community. We solved that problem by making our documentation recommend installing Zulip on a dedicated system. This also provides a lot of benefit in terms of our being able to do things like tune the configuration for postgres and memcached to allocate resources in an appropriate way for a Zulip installation with the system's RAM allocation. There is occasionally discussion in the community of adjusting our approach here, but given how easy it is to get a VM or container these days, we're currently happy with guiding users to the dedicated VM/container path that we can guarantee works ~100% of the time and provide a great installation experience.

dylan-m 7y ago

One of my favourite things with Django has been their mature approach to changes. Want to be guaranteed a stable upgrade? Step 1: Fix deprecation notices. Step 2: Upgrade. (If you're upgrading over a few releases, return to step 1). Coming from other frameworks it's a delight :) At least that's been my experience with smaller projects. I'm curious, does that line up with your experience with Zulip, too?

tabbott 7y ago
Yeah, that matches my experience; their documentation on upgrades is excellent, and there's generally only issues not covered by their deprecation notices in places where you were doing something not supported by the documented API (E.g. monkey-patching, subclassing to override something, etc.), and even then, they often comment if it's a common type of modification.
andrewingram 7y ago
I've personally never had a painless Django upgrade (except for patch releases). I always seem to fall into that grey area where I'm relying on undocumented behaviour of documented features (e.g. not using framework internals per se). I remember one Django release which changed the behaviour of views that returned files to download (it was probably Django 1.4), but there was no mention of that in the release notes. Sometimes I've knowingly relied on framework internals in ways I expected to eventually break, but those situations don't represent the bulk of my experience with painful upgrades.

Znafon 7y ago

nine_k 7y ago

If you don't mind: what are the key things that make Django useful for a project like Zulip? How much does the framework help you, and where? What are the places where you have to go against the grain, if any?

tabbott 7y ago
I'd say the big thing is helping us avoid reinventing the wheel. There's a lot of problems that every webapp has, and Django provides a system for solving most of them. I particularly enjoy their migrations and automated testing systems. When we need to integrate something that a lot of other people have, whether it be 2FA or SAML or LDAP or webpack, there's generally a reasonable library for it (or often, a reasonable way to do it built in, and Django's documentation is great so it's easy to figure that out). While there are some land mines with any ORM, the Django ORM does allow us to write the vast majority of our database operations with clean Pythonic code, rather than going to raw SQL. We need SQL in a few performance-intensive code paths, but it's the exception, rather than the rule, and the ORM is constantly getting better. And in contrast with some other ecosystems I've worked with, the libraries generally are well-written, well-tested, and stable, so even if they're not well-maintained because the old maintainer changed jobs or got busy or whatever, it doesn't feel like a burden to fork or adopt the library. I'm not particularly happy with Django forms or Django REST framework with how they organization validation code. For the former, I think the problem is just kinda messy; we mostly solve it by only using Django forms for the authentication code paths where we can reuse its libraries. For the latter, we have a framework we built that I'm very happy with and we plan to spin out for other projects to use when we have a bit of time. There's some documentation here: https://zulip.readthedocs.io/en/latest/tutorials/writing-views.html#request-variables https://zulip.readthedocs.io/en/latest/tutorials/writing-vie... for those curious; though that doesn't cover cool details like the mypy integration we did more recently.
- gshdg 7y ago
  Ooooh, I'd be very interested in seeing that spun out. I also find DRF borderline infuriating from a design standpoint.

nickjj 7y ago

Hey, if you ever want to do a podcast episode that goes into your tech stack (why you made certain choices, how it's been going, deploying, etc.) let me know. I'd love to have you on as a guest for a podcast I recently started called Running in Production. Dan from https://realpython.com https://realpython.com and I just did one recently. That site runs Django too. You can check that out here: https://runninginproduction.com/podcast/4-real-python-is-one-of-the-largest-python-learning-platforms-around https://runninginproduction.com/podcast/4-real-python-is-one... Click the "Become a guest" button to get started if you want. Note: If you're reading this and you're not from Zulip and you are running Django in production (or any stack) hit me up too.

k_sze 7y ago

agildehaus 7y ago

> My only frustration with this release announcement is dropping Python 3.5 support now. Because Python 3.5 is not EOL and used in LTS OS releases that have vendor support through 2021 ... Django has an LTS release (2.2) that supports Python 3.5 just fine. Doesn't it make sense to use the LTS Django on an LTS operating system? There are plenty of ways to use newer python, too. Docker, PPAs, etc.

takeda 7y ago

> My only frustration with this release announcement is dropping Python 3.5 support now. Because Python 3.5 is not EOL and used in LTS OS releases that have vendor support through 2021, this forces us to choose between our policy of supporting vendor OS releases until they reach EOL, not upgrading to Django 3 for the next year or more, or shipping our own Python on Ubuntu Xenial. Don't ever rely on python that is included with OS. It makes upgrades really difficult. This is especially true with RedHat and CentOS where they are way behind. There are other issues, like dependencies, you are bound then to only using the dependencies version they provide. If there is a bug in one, tough luck, you can try to upgrade but then you're risking that you will break system tools (yum etc). Just assume that the system python is for the system. Use the right version you need (if you use redhat/centos I highly recommend ius.us repo, they make sure that it can be installed side by side and doesn't conflict with anything in the system) then create virtualenv and install your application there. That way you have full control of the python version you use and all of your application dependencies. Also when ops team wants you to move from version 7 to 8 of the OS, your application should work the same without no or very small changes.

antb123 7y ago
And you can assume that no security issues have entered python no matter which version? People use the OS version for security audit reasons.
- takeda 7y ago
  The ius.io repo that I mentioned patches the version of Python. What it lets you do is you can install any major version you want (in fact you can install multiple at the same time and they won't conflict). So you can chose whether you want python 3.5.x, 3.6.x, 3.7.x, 3.8.x or maybe still you want 2.7.x.