5 ms·

How do you load balance a Flask application since it's single threaded? For optimal performance shouldn't there be a single Flask instance (process?) for each r

by nawitus 11y ago

How do you load balance a Flask application since it's single threaded? For optimal performance shouldn't there be a single Flask instance (process?) for each request being processed? What kind of load balancers are usually used with Flask and how do they decide to spawn new Flask instances? And why should Flask be used instead of asynchronous Tornado or Node.js?

matthewowen 11y ago
A common approach is to run Flask on uWSGI or gunicorn. You then put this behind nginx.
fuhrysteve 11y ago
You typically use something like uwsgi or gunicorn to serve requests, and it's uwsgi & gunicorn that decide how many processes or threads to start. These typically sit behind a load balancer and / or a reverse proxy. In terms of async Tornado / etc, it serves a different need. Like all wsgi servers, it is best to optimize Flask such that you serve a small number of concurrent requests that are fast to prevent a long queue waiting to serve web requests. With Tornado you have more flexibility in that particular area, but also another layer of complexity. Different tools for different use cases.
- nawitus 11y ago
  > Like all wsgi servers, it is best to optimize towards serving a small number of concurrent requests that are fast and end fast to free up the worker pool. Well, most (or all) web applications I've worked with make database calls to serve a response, which (in case of Flask) locks the whole thread, and there's always the possibility that the database request is slow. That's why I prefer asynchronous web servers where I don't need to worry about this particular problem. Modern web applications tend to be more like light proxies between the database and other (micro)services. Most logic has been transferred to the client, including templating. Serving static files should be done by Nginx in any case. Therefore I think it's rather critical that the web application does the serving of multiple requests well, and why I think the async architecture is the way going forward.
  - icebraining 11y ago
    Well, most (or all) web applications I've worked with make database calls to serve a response, which (in case of Flask) locks the whole thread I don't get this, when did Flask start connecting to databases? The only point of Flask is that it only provides an HTTP layer, nothing more. If you want to connect to a database, you have to import some other library (or use something from Python's standard library), which may or may not be blocking.
    - orf 11y ago
      Flask started talking to databases the moment people started using it. It's a web framework. OP's comment was saying it's single threaded nature is a detriment.
      - icebraining 11y ago
        Flask is just an HTTP layer, not an execution engine, so it doesn't really have a single- or multi-threaded nature. It's thread-safe, though. That said, the default development server built-in (which uses werkzeug's underneath) is multi-threaded - just pass threaded=True to app.run(...) - and in production, you can deploy a Flask app using an evented framework like Tornado or Gevent: http://flask.pocoo.org/docs/0.10/deploying/wsgi-standalone/#tornado http://flask.pocoo.org/docs/0.10/deploying/wsgi-standalone/#...
odonnellryan 11y ago
There really isn't a reason, I don't believe, to use Flask over X. It's just as good, and it comes down to your needs. To answer your first question: you use something like uWSGI. Flask, by default, stores sessions entirely client-side. You'll have to use something like Redis if you want to use more than one server (properly, anyway), but a Python app like this should easily be able to scale. It won't be Flask that stops you, anyway.
jperras 11y ago
Typically, one uses an application server container such as uWSGI for a production deployment of a Flask application. uWSGI allows for brokering requests across multiple application processes and/or threads, and also allows for more interesting and complex scenarios such as zero-downtime deploys, "automatic" scaling of processes/threads based on load characteristics, to name a few. > And why should Flask be used instead of asynchronous Tornado or Node.js? Tornado's asynchronous request handling is quite nice, but it also forces you to ensure that every 3rd party system you interact with has an equivalent async-compatible library; otherwise you end up blocking a request while waiting for a response, thus defeating the whole purpose of Tornado. Most systems (e.g. Postgresql[1]) have async-compatible drivers, but it's not always the case. As for why not node.js, well, perhaps a better question would be: How would Node be an improvement over Flask+uwsgi? What are the associated infrastructure/architecture/cultural costs? 1. http://momoko.61924.nl/en/latest/index.html http://momoko.61924.nl/en/latest/index.html
- nawitus 11y ago
  > "automatic" scaling of processes/threads based on load characteristics How is this done in detail? In the case of Flask does it create new processes or threads? How quickly does it create a new Flask process/thread to serve a new request? EDIT: This website documents configuration settings for autoscaling: http://uwsgi-docs.readthedocs.org/en/latest/Cheaper.html http://uwsgi-docs.readthedocs.org/en/latest/Cheaper.html - looks like it's based on various heuristics and algorithms, and in practice it's too slow to create a new process for each request, thus wasting resources I would think that ideally the number of processes/threads would exactly match the number of live requests, otherwise you're either having too few instances to handle all the requests as fast as possible or having too many instances which means that more resources are used than necessary, thus increasing costs. I think Node.js is an improvement because the architecture is better suited for scaling and all the dependencies are asynchronous by default. Of course, I think there are other reasons for Node.js (e.g. isomorphism, easier full-stack development etc.), but I guess that's another discussion, really.
  - icebraining 11y ago
    uWsgi does have an async mode, which works together with an async loop engine like gevent: http://uwsgi-docs.readthedocs.org/en/latest/Async.html http://uwsgi-docs.readthedocs.org/en/latest/Async.html Of course, this doesn't change your app and its dependencies to work in an async-friendly way, but then again, Node.js callbacks are hardly transparent either.
viperscape 11y ago
I don't know current status, but in the post you would use gevent or similar to async. Also you would put nginx and likely gunicorn in front of the app. https://www.digitalocean.com/community/tutorials/how-to-serve-flask-applications-with-gunicorn-and-nginx-on-ubuntu-14-04 https://www.digitalocean.com/community/tutorials/how-to-serv...
Gratsby 11y ago
I've built with Flask, Tornado, and with Node/Express. Flask/Python blows Node out of the water where performance is concerned. That shouldn't be a surprise to anyone. Flask has a significantly faster ramp up time than Tornado as it is designed to be simple. As far as load balancing/performance, you front end it with something like NginX. You can process things out of band easily enough with Flask, but I wouldn't evangelize it so much to say that it's the best solution for every problem. Personally, I like to keep things as simple and as logically separated as possible until it's absolutely necessary to change. Flask is one of those frameworks that doesn't present a lot of mystery. My biggest problem with node isn't performance - for that it's fine for most use cases that I deal with. It's that you can get perl-level convoluted and sloppy very quickly and easily.
- nostrademons 11y ago
  Curious about your performance experience with Node - the TechEmpower benchmarks have Node at about 2-4x the speed of Flask: https://www.techempower.com/benchmarks/#section=data-r11&hw=peak&test=fortune https://www.techempower.com/benchmarks/#section=data-r11&hw=... And my experience is generally also that V8 beats the pants off CPython in performance. I suspect that part of the problem is that the node ecosystem encourages you to just "npm install" whenever you have a problem, and so you have 3-4 layers of third-party libraries doing minimal work (but consuming lots of CPU cycles) on top of your problem space. But if that's the problem - don't do that. Flask is a microframework; it's trivially easy to build a small microframework on top of the Node.js stdlibs. Node was meant as a scriptable high-performance C++ server, after all, and there's nothing stopping you from using it as one.
  - Gratsby 11y ago
    The benchmark is a little strange. It has a range of 723 requests (CherryPy) for a plaintext hello world to 6.8Million (ulib) for plaintext. I looked at the Flask numbers and they don't compare with my own - I got between 3 and 4 times that when I benchmarked it myself on hardware that I'm sure was older and less powerful (and I didn't bother to turn off logging). I looked at the node code and it does not use express - which could very well be the difference I am experiencing. I'll tell you what though... I'm going to have a look at ulib for static files!
zenogais 11y ago
Original post author here. On the single server level I've mostly used Flask behind uWSGI for load balancing, My experience with this has been overall positive (but uwsgi can be a bit of a pain to script deployments for, maybe it's better now though, not sure). Then there's the further load balancing you can do behind something like an ELB instance in AWS to control load across multiple servers.

by nawitus 11y ago

matthewowen 11y ago

A common approach is to run Flask on uWSGI or gunicorn. You then put this behind nginx.

fuhrysteve 11y ago

You typically use something like uwsgi or gunicorn to serve requests, and it's uwsgi & gunicorn that decide how many processes or threads to start. These typically sit behind a load balancer and / or a reverse proxy. In terms of async Tornado / etc, it serves a different need. Like all wsgi servers, it is best to optimize Flask such that you serve a small number of concurrent requests that are fast to prevent a long queue waiting to serve web requests. With Tornado you have more flexibility in that particular area, but also another layer of complexity. Different tools for different use cases.

nawitus 11y ago
> Like all wsgi servers, it is best to optimize towards serving a small number of concurrent requests that are fast and end fast to free up the worker pool. Well, most (or all) web applications I've worked with make database calls to serve a response, which (in case of Flask) locks the whole thread, and there's always the possibility that the database request is slow. That's why I prefer asynchronous web servers where I don't need to worry about this particular problem. Modern web applications tend to be more like light proxies between the database and other (micro)services. Most logic has been transferred to the client, including templating. Serving static files should be done by Nginx in any case. Therefore I think it's rather critical that the web application does the serving of multiple requests well, and why I think the async architecture is the way going forward.
- icebraining 11y ago
  Well, most (or all) web applications I've worked with make database calls to serve a response, which (in case of Flask) locks the whole thread I don't get this, when did Flask start connecting to databases? The only point of Flask is that it only provides an HTTP layer, nothing more. If you want to connect to a database, you have to import some other library (or use something from Python's standard library), which may or may not be blocking.
  - orf 11y ago
    Flask started talking to databases the moment people started using it. It's a web framework. OP's comment was saying it's single threaded nature is a detriment.
    - icebraining 11y ago
      Flask is just an HTTP layer, not an execution engine, so it doesn't really have a single- or multi-threaded nature. It's thread-safe, though. That said, the default development server built-in (which uses werkzeug's underneath) is multi-threaded - just pass threaded=True to app.run(...) - and in production, you can deploy a Flask app using an evented framework like Tornado or Gevent: http://flask.pocoo.org/docs/0.10/deploying/wsgi-standalone/#tornado http://flask.pocoo.org/docs/0.10/deploying/wsgi-standalone/#...

odonnellryan 11y ago

There really isn't a reason, I don't believe, to use Flask over X. It's just as good, and it comes down to your needs. To answer your first question: you use something like uWSGI. Flask, by default, stores sessions entirely client-side. You'll have to use something like Redis if you want to use more than one server (properly, anyway), but a Python app like this should easily be able to scale. It won't be Flask that stops you, anyway.

jperras 11y ago

Typically, one uses an application server container such as uWSGI for a production deployment of a Flask application. uWSGI allows for brokering requests across multiple application processes and/or threads, and also allows for more interesting and complex scenarios such as zero-downtime deploys, "automatic" scaling of processes/threads based on load characteristics, to name a few. > And why should Flask be used instead of asynchronous Tornado or Node.js? Tornado's asynchronous request handling is quite nice, but it also forces you to ensure that every 3rd party system you interact with has an equivalent async-compatible library; otherwise you end up blocking a request while waiting for a response, thus defeating the whole purpose of Tornado. Most systems (e.g. Postgresql[1]) have async-compatible drivers, but it's not always the case. As for why not node.js, well, perhaps a better question would be: How would Node be an improvement over Flask+uwsgi? What are the associated infrastructure/architecture/cultural costs? 1. http://momoko.61924.nl/en/latest/index.html http://momoko.61924.nl/en/latest/index.html

nawitus 11y ago
> "automatic" scaling of processes/threads based on load characteristics How is this done in detail? In the case of Flask does it create new processes or threads? How quickly does it create a new Flask process/thread to serve a new request? EDIT: This website documents configuration settings for autoscaling: http://uwsgi-docs.readthedocs.org/en/latest/Cheaper.html http://uwsgi-docs.readthedocs.org/en/latest/Cheaper.html - looks like it's based on various heuristics and algorithms, and in practice it's too slow to create a new process for each request, thus wasting resources I would think that ideally the number of processes/threads would exactly match the number of live requests, otherwise you're either having too few instances to handle all the requests as fast as possible or having too many instances which means that more resources are used than necessary, thus increasing costs. I think Node.js is an improvement because the architecture is better suited for scaling and all the dependencies are asynchronous by default. Of course, I think there are other reasons for Node.js (e.g. isomorphism, easier full-stack development etc.), but I guess that's another discussion, really.
- icebraining 11y ago
  uWsgi does have an async mode, which works together with an async loop engine like gevent: http://uwsgi-docs.readthedocs.org/en/latest/Async.html http://uwsgi-docs.readthedocs.org/en/latest/Async.html Of course, this doesn't change your app and its dependencies to work in an async-friendly way, but then again, Node.js callbacks are hardly transparent either.

viperscape 11y ago

I don't know current status, but in the post you would use gevent or similar to async. Also you would put nginx and likely gunicorn in front of the app. https://www.digitalocean.com/community/tutorials/how-to-serve-flask-applications-with-gunicorn-and-nginx-on-ubuntu-14-04 https://www.digitalocean.com/community/tutorials/how-to-serv...

Gratsby 11y ago

I've built with Flask, Tornado, and with Node/Express. Flask/Python blows Node out of the water where performance is concerned. That shouldn't be a surprise to anyone. Flask has a significantly faster ramp up time than Tornado as it is designed to be simple. As far as load balancing/performance, you front end it with something like NginX. You can process things out of band easily enough with Flask, but I wouldn't evangelize it so much to say that it's the best solution for every problem. Personally, I like to keep things as simple and as logically separated as possible until it's absolutely necessary to change. Flask is one of those frameworks that doesn't present a lot of mystery. My biggest problem with node isn't performance - for that it's fine for most use cases that I deal with. It's that you can get perl-level convoluted and sloppy very quickly and easily.

nostrademons 11y ago
Curious about your performance experience with Node - the TechEmpower benchmarks have Node at about 2-4x the speed of Flask: https://www.techempower.com/benchmarks/#section=data-r11&hw=peak&test=fortune https://www.techempower.com/benchmarks/#section=data-r11&hw=... And my experience is generally also that V8 beats the pants off CPython in performance. I suspect that part of the problem is that the node ecosystem encourages you to just "npm install" whenever you have a problem, and so you have 3-4 layers of third-party libraries doing minimal work (but consuming lots of CPU cycles) on top of your problem space. But if that's the problem - don't do that. Flask is a microframework; it's trivially easy to build a small microframework on top of the Node.js stdlibs. Node was meant as a scriptable high-performance C++ server, after all, and there's nothing stopping you from using it as one.
- Gratsby 11y ago
  The benchmark is a little strange. It has a range of 723 requests (CherryPy) for a plaintext hello world to 6.8Million (ulib) for plaintext. I looked at the Flask numbers and they don't compare with my own - I got between 3 and 4 times that when I benchmarked it myself on hardware that I'm sure was older and less powerful (and I didn't bother to turn off logging). I looked at the node code and it does not use express - which could very well be the difference I am experiencing. I'll tell you what though... I'm going to have a look at ulib for static files!

zenogais 11y ago

Original post author here. On the single server level I've mostly used Flask behind uWSGI for load balancing, My experience with this has been overall positive (but uwsgi can be a bit of a pain to script deployments for, maybe it's better now though, not sure). Then there's the further load balancing you can do behind something like an ELB instance in AWS to control load across multiple servers.