3 ms·

It has been long established that using Twisted's web capabilities is a bad idea. It has been broken from the usability point of view for a very long time and I

It has been long established that using Twisted's web capabilities is a bad idea. It has been broken from the usability point of view for a very long time and I don't think it will ever be fixed. Without reciting the entire history, Tornado got a lot of crap for not just fixing what Twisted had going on, but creating a new framework from scratch. Having worked with both, I see why they did it. Basically, if you have a bad experience with Twisted + web, you are very much not alone but the fault is with Twisted, not Python. Here's how I break it down: If you need async networking at the TCP or UDP level, use Twisted. If you want a web application, use Django. If you want a one page landing page type application, use Flask. If you are on a restricted budget and need really good performance for your web application that you tried and couldn't extract from Django, go with Tornado.

makmanalp 12y ago
I'll just add, if you want a simple REST API server, flask works well. Here's an example of async with flask and sqlalchemy: https://github.com/kljensen/async-flask-sqlalchemy-example https://github.com/kljensen/async-flask-sqlalchemy-example
- raverbashing 12y ago
  Correct, I was going to mention that If you need a "CMS style" web application go for Django, now if it breaks too much from that you might want to try Flask/Pylons/Pyramid
klibertp 12y ago
> that using Twisted's web capabilities is a bad idea Isn't async just a bad fit for a typical Web page? Two things which take most time per request (in my experience) are DB queries and rendering templates. While you can have async DB driver (only for postgres right now), all the template engines do their job synchronously, which - with async event loop - blocks everything until finished. There's deferToThread in Twisted, but if we're going to use a pool of threads anyway, what's the point? I also thought this problem was mostly solved with Nginx and uWSGI. This setup works extremely well in my experience, eliminating problems with handling too many sockets and such but allowing to write Django code as usual. Async is good if you mostly do things which can be asynchronous, like fetching things over the net, reading files from disk, communicating with Redis and DB. You really need pre-emptive scheduler for tasks (not necessarily threads, see Erlang) for anything that's going to be CPU-bound. And it's not true that rendering web pages is 100% IO-bound - not when you're using Python, Django and need consistently ~100ms response times.
- IgorPartola 12y ago
  Well, you answered your own question ("what's the point?"): because not every application is "app stack + RDBMS". There are many situations where your backing store is not an RDBMS. There are many situations where you are not rendering templates. There are many situations where queries take seconds, while rendering the result takes microseconds. While nginx is a very useful tool, it's not the application layer. What if you need your application layer to be fast and complex? What if your application layer works on streams, not rendered HTML pages? What if you want to support server-client notifications via WebSockets? There are so many different situations where a "block this request processor until the request is served" does not work. Having said that, I'll repeat again that async is not what you should reach for unless it makes perfect sense for your application. If you are building an RSS fetcher, sure go for it. If you are building a product for which you see peak usage of, say, 1m users, go for it. However, for most people, time to market is much more important than peak performance after you can't scale the hardware cheaply. That's where Django (Flask, Rails, etc.) make more sense.
  - klibertp 12y ago
    "What's the point" referred only to the use of deferToThread in Twisted. And I was specifically talking about "typical Web page", you know, login, logout, comments and such. Other than that we actually agree 100%. I wrote that "Async is good if you mostly do things which can be asynchronous", you in turn listed a couple of examples of such things (WebSockets, not RDBMS). We're violently in agreement here. One additional point I made was about Erlang. Really, if you're building " a product for which you see peak usage of, say, 1m users" go for Erlang (or rather about pre-emptive scheduling, but it boils down to Erlang anyway). In my experience it's the only environment which provides both concurrency and parallelism for both IO and CPU-bound tasks and is easily (transparently!) distributable to many nodes.
    - IgorPartola 12y ago
      That's a good and complete summary. I have been curious about Erlang for some time. The syntax keeps sending me running in the other direction, but perhaps I just haven't found a suitable project to work on where I could have an excuse to really dig in. any favorite resources you can recommend on learning Erlang?

makmanalp 12y ago

I'll just add, if you want a simple REST API server, flask works well. Here's an example of async with flask and sqlalchemy: https://github.com/kljensen/async-flask-sqlalchemy-example https://github.com/kljensen/async-flask-sqlalchemy-example

raverbashing 12y ago
Correct, I was going to mention that If you need a "CMS style" web application go for Django, now if it breaks too much from that you might want to try Flask/Pylons/Pyramid

klibertp 12y ago

> that using Twisted's web capabilities is a bad idea Isn't async just a bad fit for a typical Web page? Two things which take most time per request (in my experience) are DB queries and rendering templates. While you can have async DB driver (only for postgres right now), all the template engines do their job synchronously, which - with async event loop - blocks everything until finished. There's deferToThread in Twisted, but if we're going to use a pool of threads anyway, what's the point? I also thought this problem was mostly solved with Nginx and uWSGI. This setup works extremely well in my experience, eliminating problems with handling too many sockets and such but allowing to write Django code as usual. Async is good if you mostly do things which can be asynchronous, like fetching things over the net, reading files from disk, communicating with Redis and DB. You really need pre-emptive scheduler for tasks (not necessarily threads, see Erlang) for anything that's going to be CPU-bound. And it's not true that rendering web pages is 100% IO-bound - not when you're using Python, Django and need consistently ~100ms response times.

IgorPartola 12y ago
Well, you answered your own question ("what's the point?"): because not every application is "app stack + RDBMS". There are many situations where your backing store is not an RDBMS. There are many situations where you are not rendering templates. There are many situations where queries take seconds, while rendering the result takes microseconds. While nginx is a very useful tool, it's not the application layer. What if you need your application layer to be fast and complex? What if your application layer works on streams, not rendered HTML pages? What if you want to support server-client notifications via WebSockets? There are so many different situations where a "block this request processor until the request is served" does not work. Having said that, I'll repeat again that async is not what you should reach for unless it makes perfect sense for your application. If you are building an RSS fetcher, sure go for it. If you are building a product for which you see peak usage of, say, 1m users, go for it. However, for most people, time to market is much more important than peak performance after you can't scale the hardware cheaply. That's where Django (Flask, Rails, etc.) make more sense.
- klibertp 12y ago
  "What's the point" referred only to the use of deferToThread in Twisted. And I was specifically talking about "typical Web page", you know, login, logout, comments and such. Other than that we actually agree 100%. I wrote that "Async is good if you mostly do things which can be asynchronous", you in turn listed a couple of examples of such things (WebSockets, not RDBMS). We're violently in agreement here. One additional point I made was about Erlang. Really, if you're building " a product for which you see peak usage of, say, 1m users" go for Erlang (or rather about pre-emptive scheduling, but it boils down to Erlang anyway). In my experience it's the only environment which provides both concurrency and parallelism for both IO and CPU-bound tasks and is easily (transparently!) distributable to many nodes.
  - IgorPartola 12y ago
    That's a good and complete summary. I have been curious about Erlang for some time. The syntax keeps sending me running in the other direction, but perhaps I just haven't found a suitable project to work on where I could have an excuse to really dig in. any favorite resources you can recommend on learning Erlang?