3 ms·

We've been using Flask recently, and I do like the simple, no-nonsense approach. Our project is modest in its library needs, and it's nice to have a useful tool

We've been using Flask recently, and I do like the simple, no-nonsense approach. Our project is modest in its library needs, and it's nice to have a useful toolkit without imposing a heavyweight structure on our code. There are significant limitations with some of these frameworks today, though. For example, lack of byte range/partial content support is bad news if you're serving large files (think multimedia). We're still investigating options here, but WSGI doesn't seem ideally suited to working that way, and as far as we can tell quite a few of the "microframeworks" lack any kind of support for returning partial content automatically. The concern seems to be that WSGI would need the application to generate a complete response even if it was going to serve only part of it as a 206, because there's no standardised part of the routing set-up that says "I need these byte ranges from whatever full response you would have given me". A common philosophy seems to be that such files should be served statically by your front-end web server anyway, but that is not sufficient in all cases. For example, you might need to process the request through your framework to generate the response on demand, implement access controls, or integrate with a custom logging/analytics framework. [Edits: Trying to clarify key points.]

irahul 15y ago
A common philosophy seems to be that such files should be served statically by your front-end web server anyway, but that is not sufficient in all cases. For example, you might need to process the request through your framework to generate the response on demand, implement access controls, or integrate with a custom logging/analytics framework. Depending on your use case, you can either use X-sendfile header - validate request, then return a X-sendfile response for your front-end server; or use gevent to stream the response.
- Chris_Newton 15y ago
  Thanks for the suggestions. In case it helps anyone else: It's something about the way that Flask handles a direct send_file that seems to be causing problems in our tests. However, we've found reports of sendfile not working properly on certain platforms and we're still investigating, so I'm not convinced at this stage that the problem is with Flask itself.
  - irahul 15y ago
    > However, we've found reports of sendfile not working properly on certain platforms and we're still investigating, so I'm not convinced at this stage that the problem is with Flask itself. I am confused by your sendfile not working properly on certain platforms. If you set the X-Sendfile header, and the web server supports it, it should work fine. Flask send_file doesn't do much other than setting up the headers; and then either setting the X-Senfile header so that web server handles it, or sends it using WSGI file wrapper supports. https://github.com/mitsuhiko/flask/blob/master/flask/helpers.py#L401 https://github.com/mitsuhiko/flask/blob/master/flask/helpers...
    - Chris_Newton 15y ago
      I was referring to the sendfile system call there. We've seen reports that on some versions of Linux kernel, it doesn't work, for some value of "doesn't work" that we haven't yet properly determined. We've also seen web pages that suggest Flask's send_file and send_from_directory would ultimately rely on that system call in some circumstances, though again I haven't checked through the code in detail yet so I don't know how accurate those reports are. In any case, we haven't tried using the X-Sendfile functionality yet, and I'm making no comment about how well that works. We do know, without any doubt at this point, that serving our video files using Flask's send_from_directory (using the default file wrapping behaviour, not X-Sendfile) is not working reliably for us with many different clients, where serving the same file statically straight from a web server like Apache works fine. We're still trying to identify exactly what it is that doesn't work, but in light of our discussion here today and what I've learned since then, I'm now hoping that we can just punt the whole set-up over to the known-working Apache implementation by using X-Sendfile and avoid the problem in the first place. (Edit: Thanks again for the suggestions, BTW. I had a fairly long list of ideas to follow up in relation to this problem, including investigating X-Sendfile, but your mention of that here prompted me to look into it first and will probably have saved me quite a bit of time if it does work for us.)
      - irahul 15y ago
        I see. I wasn't aware wsgi file wrapper might use sendfile. http://www.python.org/dev/peps/pep-0333/#optional-platform-specific-file-handling http://www.python.org/dev/peps/pep-0333/#optional-platform-s... That's environment specific; the wsgi server should define a environ['wsgi.file_wrapper'] if it needs to provide specific file handling functionality. I don't think you have an issue with Linux kernel - 2.2 and beyond will always have sendfile. It's more likely that your wsgi server isn't defining environ['wsgi.file_wrapper'] to something that uses sendfile. If you see the flask send_file code, it's using werkzeug.wsgi.wrap_file, which in turn either uses environ['wsgi.file_wrapper'], or uses simple Python code to read the file and send the contents. https://github.com/mitsuhiko/werkzeug/blob/master/werkzeug/wsgi.py#L537 https://github.com/mitsuhiko/werkzeug/blob/master/werkzeug/w... Reading file and sending the contents in your python code is a bad idea. You can either set debugging points and see if wsgi server is defining environ['wsgi.file_wrapper'] properly. Or more conveniently, you can just set 'USE_X_SENDFILE = True' in your flask settings, and then directly use send_file.

irahul 15y ago

A common philosophy seems to be that such files should be served statically by your front-end web server anyway, but that is not sufficient in all cases. For example, you might need to process the request through your framework to generate the response on demand, implement access controls, or integrate with a custom logging/analytics framework. Depending on your use case, you can either use X-sendfile header - validate request, then return a X-sendfile response for your front-end server; or use gevent to stream the response.

Chris_Newton 15y ago
Thanks for the suggestions. In case it helps anyone else: It's something about the way that Flask handles a direct send_file that seems to be causing problems in our tests. However, we've found reports of sendfile not working properly on certain platforms and we're still investigating, so I'm not convinced at this stage that the problem is with Flask itself.
- irahul 15y ago
  > However, we've found reports of sendfile not working properly on certain platforms and we're still investigating, so I'm not convinced at this stage that the problem is with Flask itself. I am confused by your sendfile not working properly on certain platforms. If you set the X-Sendfile header, and the web server supports it, it should work fine. Flask send_file doesn't do much other than setting up the headers; and then either setting the X-Senfile header so that web server handles it, or sends it using WSGI file wrapper supports. https://github.com/mitsuhiko/flask/blob/master/flask/helpers.py#L401 https://github.com/mitsuhiko/flask/blob/master/flask/helpers...
  - Chris_Newton 15y ago
    I was referring to the sendfile system call there. We've seen reports that on some versions of Linux kernel, it doesn't work, for some value of "doesn't work" that we haven't yet properly determined. We've also seen web pages that suggest Flask's send_file and send_from_directory would ultimately rely on that system call in some circumstances, though again I haven't checked through the code in detail yet so I don't know how accurate those reports are. In any case, we haven't tried using the X-Sendfile functionality yet, and I'm making no comment about how well that works. We do know, without any doubt at this point, that serving our video files using Flask's send_from_directory (using the default file wrapping behaviour, not X-Sendfile) is not working reliably for us with many different clients, where serving the same file statically straight from a web server like Apache works fine. We're still trying to identify exactly what it is that doesn't work, but in light of our discussion here today and what I've learned since then, I'm now hoping that we can just punt the whole set-up over to the known-working Apache implementation by using X-Sendfile and avoid the problem in the first place. (Edit: Thanks again for the suggestions, BTW. I had a fairly long list of ideas to follow up in relation to this problem, including investigating X-Sendfile, but your mention of that here prompted me to look into it first and will probably have saved me quite a bit of time if it does work for us.)
    - irahul 15y ago
      I see. I wasn't aware wsgi file wrapper might use sendfile. http://www.python.org/dev/peps/pep-0333/#optional-platform-specific-file-handling http://www.python.org/dev/peps/pep-0333/#optional-platform-s... That's environment specific; the wsgi server should define a environ['wsgi.file_wrapper'] if it needs to provide specific file handling functionality. I don't think you have an issue with Linux kernel - 2.2 and beyond will always have sendfile. It's more likely that your wsgi server isn't defining environ['wsgi.file_wrapper'] to something that uses sendfile. If you see the flask send_file code, it's using werkzeug.wsgi.wrap_file, which in turn either uses environ['wsgi.file_wrapper'], or uses simple Python code to read the file and send the contents. https://github.com/mitsuhiko/werkzeug/blob/master/werkzeug/wsgi.py#L537 https://github.com/mitsuhiko/werkzeug/blob/master/werkzeug/w... Reading file and sending the contents in your python code is a bad idea. You can either set debugging points and see if wsgi server is defining environ['wsgi.file_wrapper'] properly. Or more conveniently, you can just set 'USE_X_SENDFILE = True' in your flask settings, and then directly use send_file.