7 ms·
Handling 100 Requests per Second with Python and Django
- ericholscher 5y agoA blog post we recently published talking about scaling up our operations. I love using standard tools (Django, Python, Postgres) to achieve this. Definitely shows that you don't need fancy tools until you get huge -- and here is a longer version of that story.
- mattip 5y agoI get that python is under 30% of your profiled time, but just wondering if you tried PyPy?
- davidfischer 5y agoWe've run some tests with PyPy on Read the Docs itself but not for ads. For Sphinx documentation builds, builds took around ~50% of the time. It was especially pronounced on builds with large numbers of doc files (hundreds) and therefore complicated side navigation.
- mattip 5y agoRight, Sphinx itself is not JIT friendly, but I thing django should be faster
- jka 5y agoHas your experience led you to identify any areas for improvement within Django and/or Python? (enjoyed the post, and I like your philosophy of using simple widely available tools; I'm hopeful that any changes you're able to provide back will have a kind of community-benefit flywheel effect)
- ericholscher 5y agoHonestly, not really. A lot of the core issues that we'd hit were solved years ago (eg. multiple DB's for a read replica, JSON in the DB). I think it's one of the big benefits of using standard "boring" tools, that they do most everything we need them to do. We're not anxiously waiting on a new release, we're happily using new features implemented 5 years ago when we need them.
- jka 5y agoThanks - that's a good philosophy and position to be in :)
- SilurianWenlock 5y agoWhy dont use just use Java?
- zwarag 5y ago> There was also a strong desire to use a similar setup to Read the Docs as that's what our small team (of 5 at that time) knew well. Just read the article...
- rbanffy 5y agoI was asked that question a lot. It’s true that a month after the Java app is running, it’ll have served more requests than the Python app, but, by then, the Python app will be on its fifth feature release.
- hereforphone 5y ago"Why dont use just use Java?" - Every student who was forced to learn Java in college, ever.
- Zababa 5y agoWhat would using Java accomplish exactly?
- ldiracdelta 5y agoI see your "java", and I raise you "Why don't you just use Rust?"
- wil421 5y agoWhat framework in Rust would give you anything close to Django? Or even close to what Java has to offer?
- ldiracdelta 5y agoThat, my friend, was a joke.
- 5y ago
- rachelbythebay 5y agoNote: across multiple machines.
- ericholscher 5y agoI think we could probably do it on 1 machine. We have multiple mostly for availability and handling spikes. There's definitely no reason we couldn't use 1 large machine to do this, just not a great reason to run a production app this way.
- cryptos 5y ago100 requests per second? That is not too impressive. Yes, I know Python is their tool of choice, but it is probably the wrong tool for the problem at hand. There are frameworks that can handle over a million requests per second (simple json output) or at least several 10,000 requests per second if DB queries are performed (even though on different hardware, but just compare the scale). https://www.techempower.com/benchmarks/#section=data-r20&hw=ph&test=json https://www.techempower.com/benchmarks/#section=data-r20&hw=... https://www.techempower.com/benchmarks/#section=data-r20&hw=ph&test=query https://www.techempower.com/benchmarks/#section=data-r20&hw=... I think, if performance is a strong requirement, they were better off with another programming language.
- rbanffy 5y agoIn their case, it seems the request in question involves a write to a PostgreSQL database. Depending on the database, 100 can be a lot of work. They measured that the app spends about half its time waiting for the database. There seems to be a whole lot of things to do before thinking of changing basic plumbing.
- fabian2k 5y agoFor simple queries Postgres can do thousands of requests per second on desktop hardware without any serious tuning. Of course if those are complex queries or if they need to handle a lot of data this can be much, much slower. But Postgres is not slow for simple queries.
- puppet-master 5y agoPerhaps then the title of the post could be "handling 100 writes per second with Postgres"
- detaro 5y agoOr perhaps people could consider more than the title when commenting.
- Zababa 5y ago
- fabian2k 5y agoOne thing that jumps out at me is that it seems they didn't use any Postgres connection pooling. They mention that going above 100 concurrent connection would bump them into a more expensive Postgres plan, which really only makes sense if you don't use a pool. My first instinct is that the number of requests seems really low, but I have no idea about the complexity of each request. To me that is kind of a crucial information to actually evaluate anything in the blog post.
- ericholscher 5y agoThis is a great point. One of the ideas we were wanting to get across is that using a very simple tech stack you can get to a number of requests that most services will never meaningfully see. We do have our Django configured to pool connections, but it's configured per process, and not sharing them across all the various web processes that we're running: https://docs.djangoproject.com/en/3.2/ref/databases/#persistent-connections https://docs.djangoproject.com/en/3.2/ref/databases/#persist... I'd be curious to see if there is a meaningful gain from that approach, if anyone has done the transition before.
- fabian2k 5y agoThat's one part where having multiple completely independent processes is a disadvantage to just having threads. I never benchmarked this, but I always heard that Postgres processes are expensive and you should always use a connection pool. But this is probably more noticeable on the DB server side, where each process uses memory. I never used it myself, but PG bouncer might be useful to you in this case. The number of requests still feels slow, and it isn't clear to me from the blog post whether that is DB limited or CPU limited on the application servers. Even with writes on each access Postgres should still be bored at that kind of load.
- ericholscher 5y agoYea, our IO and CPU of our Postgres instance is around 20% currently. We haven't hit too many issues with Postgres, so haven't delved too deeply into the performance. I think people seeing performance posts are used to people pushing huge numbers. Our intent here was to show that using standard off the shelf tools with a tiny bit of architecting, you can hit huge performance numbers at a reasonable cost. We have lots of various ways to shave performance, but we're pretty sure we could scale 2-3x with the same stack without a ton of work.
- sbelskie 5y ago> A few months after rolling things out to production we encountered some issues with AppService. I’d be curious to hear more about this. Was is just degraded performance or something else?
- ericholscher 5y agoWe hit a few various issues. One of the worst was the instances becoming totally unavailable for random periods of time, and not being able to do anything really to debug or adjust them. We hacked some code to be able to eventually get into a shell on the machine, but that still didn't give us a ton of control. Generally we've found that having basic control over the hosting infrastructure is important. We still depend on Azure's LB's and other infra for keeping things available, but having the actual instances our code are running on accessible makes a lot of things easier.
- exdsq 5y agoDon't most major cloud providers offer multi-region replication to start solving the Europe/Asia latency issue?
- rbanffy 5y agoTheir data model seems very simple and appears to assume that there is a single consistent source of truth all the time.
- ericholscher 5y agoWe do a write on every ad view, so there is a bit more complexity here. Most of those writes should be atomic (eg. each ad will only be viewed once) -- but there is a case where an ad might get viewed, then clicked, and we need the view data to know if the click is valid.
- dillondoyle 5y agoI hate when HN'ers just chime in and say you're shit why didn't you do this. So hopefully I'm not that totally rude person, no-shade intended pointing out that if for some random reason you haven't seen ClickHouse you should check it out. Solves problems that I parsed from your blog post and originally built for that use case. 100 recs/s for an ad network is super duper mega tiny so I hope you're successful and get bigger!
- davidfischer 5y agoI did check out ClickHouse but I haven't gotten a chance to load more real data to give it the full test. It's definitely on the todo.
- listenallyall 5y agoJust an aside, but if the primary goal was to increase capacity (transactions per second), actually performing a write on every ad view would be the first thing I'd seek to eliminate. Is it possible to keep the details as a class/object in RAM, update an in-memory array or cache, and do a bulk write to the DB once every, say, 5 minutes? Even if you lost 0.5% of your data (and I would expect your actual losses would be much lower), we're talking ad clicks, not bank transactions. Eliminating the DB write and confirmation on every request, especially over the network, could easily speed up responses and therefore capacity by 5x or more.
- nickphx 5y agoI built a project in python using django that handles 11k/second. Requires a load balancer and a few machines of course.. :D
- axiosgunnar 5y agoComing from a node.js world, I thought they meant 100k and the title had a typo.
- dijit 5y agoIt would be nice if python could get the same amount of effort put in to performance as JavaScript did. But in this case the interpreter speed is largely irrelevant. This is a complex project and more than half the response time is spent waiting for the database (which is C++, not that it matters either)
- Zababa 5y agoIs the difference really that big? TechEmpower's benchmarks [1] shows express being a few times faster than Django, but it's an order of magnitude at most. You're talking about 4 orders of magnitude. Is it something that's not well covered by benchmarks? [1]: https://www.techempower.com/benchmarks/#section=data-r20&hw=ph&test=composite https://www.techempower.com/benchmarks/#section=data-r20&hw=...
- paulddraper 5y agoMore than likely there is some other factor disturbing the performance here. 100/s is slow
- heavyset_go 5y agoI wonder how much Django's async views and ASGI support would affect the request rate.
- davidfischer 5y agoI'm not an expert on async views at all but my hunch is it wouldn't do much. I do think async has some pretty interesting applications though on Read the Docs itself in our small proxy that serves docs which are just static files in s3. Currently it's a stripped down Django setup running in separate process from the main RTD app and it mostly sets headers picked up by nginx/sendfile.
- heavyset_go 5y agoAsync views alone might not do much, but ASGI support should mean that request handling is async, which might impact the request rate. Django's core is synchronous[1], though, so who knows. [1] https://arunrocks.com/a-guide-to-asgi-in-django-30-and-its-performance/ https://arunrocks.com/a-guide-to-asgi-in-django-30-and-its-p...
- zikani_03 5y agoInteresting read.. Have you tried using waitress[0]? We got some better performance for an internal service than Gunicorn after trying different gunicorn configurations to improve the performance of our workloads. We now mostly run (read depend) one instance of that service instead of 3 like we used to before. [0]: https://pypi.org/project/waitress/ https://pypi.org/project/waitress/
- davidfischer 5y agoMaybe I didn't run waitress through as full a test as I should have. In my (admittedly shallow) tests, it didn't offer any significant performance benefits over Gunicorn and I stuck with Gunicorn for the simple reason that the rest of our infra used it. Would you be willing to share a bit more about your configuration that made waitress significantly better for you?
- nickphx 5y agoSeems like it's behind gunicorn in terms of performance and features.
- no_time 5y agoForgive me for my ignorance (I'm not in the saas biz) but $300-$500/mo seems eye wateringly expensive for what is essentially static content + abuse detection and metering. less than $100/mo buys you an absolute beast of a VPS at some hosting provider that could alone handle way more than that.
- swuecho 5y ago5/mo digitalOcean instance should handle 100/req fine
- kofejnik 5y agoif you do blocking DB operations before serving ads, latency won't be great. Assuming you don't have to wait for transactions to commit, you can probably push required operations onto a queue (Redis would be fine for that) and serve the content immediately?