5 ms·

Python HTTP library 'urllib3' now works in the browser

108 points by SethMLarson 3y ago

canadiantim 3y ago
Pyodide examples using urllib3 in browser still requires using fetch, so this still requires javascript. What does this practically mean? Does this ever point to a future where we can use urlib3 instead of fetch?
- deleted 3y ago
  [deleted]
- SethMLarson 3y ago
  Lead maintainer of urllib3 here: We're tracking all the options for how we can make the experience better for folks looking to use Python in the browser. Today our biggest blocker is that there's no socket or TLS APIs for Emscripten/WASI. If those were to become available we'd jump on them right away :) I think the biggest benefit of a foundational project like urllib3 getting browser support is that /many/ Python projects are built on top of urllib3 and many of them likely work in browsers now that urllib3 works in the browser. For example, Requests is built on top of urllib3 and now gets browser support for free! And this is recursive, how many projects use Requests (spoiler: many). Pretty exciting IMO :)
  - canadiantim 3y ago
    wow that's awesome. That we could even talk about potentially using just python to make requests in browser is pretty wild (of course, as you say still blocks in the way - lack of socket or TLS APIs for Emscripten/WASI). Really cool stuff tho. All the best eh and thank you VERY much for the incredible urllib3 library.
  - SushiHippie 3y ago
    Do you know if there are some plans for sockets or TLS APIs or are we far away from this?
    - SethMLarson 3y ago
      Browsers limit the ability for these platforms to use raw sockets, there simply is no API for it. The best that can be done /today/ is to use WebSockets, which are not the same thing and can't be used for HTTP requests without the server expecting a WebSocket connection: https://github.com/emscripten-core/emscripten/issues/5196#issuecomment-299442038 https://github.com/emscripten-core/emscripten/issues/5196#is...
      - SushiHippie 3y ago
        Sure, sorry if it wasn't clear from my question, it wasn't specifically about urllib3, but I meant if there are plans that browsers will make such an API available or if this is something that will never become real.
        sillysaurusx 3y ago
        I doubt browsers would allow this, though gamedevs have wanted it for decades now. Particularly UDP.
        plopz 3y ago
        isn't the udp problem fixed with http3 unreliable webtransport?
        sillysaurusx 3y ago
        Is it? That would be cool. Under the hood is it actual udp?
        SushiHippie 3y ago
        HTTP3 uses QUIC, it's not "pure" UDP, but it's a protocol based on it https://en.wikipedia.org/wiki/QUIC https://en.wikipedia.org/wiki/QUIC
        3y ago
  - at_a_remove 3y ago
    Thank you for your work on urllib3; I will probably never use Requests if I can help it. I'm glad that you're keeping up that "batteries included" tradition.
  - mistrial9 3y ago
    except the browser context is being sandboxed with lots of access restriction, individual profiling, geolocation reporting, device identification, and more Big Tech does not like independent protocol operation it seems; therefore your interesting technical advance is likely to be a pawn in other, larger changes to the social contracts on the open net IMHO
- msoad 3y ago
  Am I misunderstanding this? You need JS to load the WASM module anyways, what does "requiring JS" means here?
  - ploxiln 3y ago
    It means that urllib3 (python code) isn't actually making the http request (it's not making the tcp connection, tls auth, writing the request, or serializing the request headers ...) It may be better to describe this as a "shim". Any python code that depended on urllib3 is now transparently shimmed in browser context to use the fetch-api instead. Which is useful.
    - zarzavat 3y ago
      This is definitely out of scope for web browsers. The security implications of allowing arbitrary sockets are prohibitive. To do something like that then you’re going to need a different environment with a different set of security guarantees and expectations, more like a shell where the user is responsible for vouching for the code’s trustworthiness.
  - canadiantim 3y ago
    You’re right. It just reflected a momentary misunderstanding of mine. WASM still requires JS.
hpeter 3y ago
why would you want to run python in the browser? Anyone can ELI5?
- deleted 3y ago
  [deleted]
- rmnclmnt 3y ago
  Lots of use-cases but to name a few: - Interactive documentation (static website) - Jupyter Notebooks without a server, great for education (Jupyter-Lite) - Python app alternative deployment as static webapp
- cle 3y ago
  There's a huge volume of existing Python code.
- dharmab 3y ago
  Python in Excel currently requires a separate cloud server to run Python formulas. Imagine this could run in the Excel web view instead.
  - codingglass 3y ago
    Work keeping an eye on Chris Laffra's work in this area - he did a Python in Excel demo [1] a few months back, and while sheets aren't in his ltk framework [2] yet, the work he's done on reactivity-in-python-on-the-web there is really promising. [1] https://github.com/laffra/excel-in-python https://github.com/laffra/excel-in-python [2] https://laffra.github.io/ltk/ https://laffra.github.io/ltk/
- pphysch 3y ago
  For real demos of your Python libs: https://pygments.org/demo/ https://pygments.org/demo/
- Austizzle 3y ago
  The film (especially vfx) industry runs on Python, and so much code is written for Python. Same reason you'd want to run your server in node.js so it matches your front end code ( and you can reuse code on the front end and backend). Pyscript lets us have the same programming language in the front end and the backend, and access to all of our internal packages on the front end. A lot of vfx software is written in QT, so at a previous company I actually made a wrapper around a subset of the QT interface and widgets, and replicated it in Pyscript, so we could write an app once and run it from inside vfx applications, or run it in the browser with no additional coding. It wasn't perfect, but it worked and made things simpler
Extigy 3y ago
Since this is using fetch/XHR under the hood, I guess requests from the browser are restricted only to same-origin URLs or servers responding with permissive CORS headers?
- SethMLarson 3y ago
  Correct! The examples we used were making requests to httpbin.org, but I also was able to query GitHub's API :) `data = urllib3.request("GET", "https://api.github.com").json https://api.github.com").json()`
Austizzle 3y ago
Incredible! Two years ago I was experimenting in porting some code at the company I was at to python in the browser (instead of QT desktop apps) and this was the biggest thing I had to work around! I had to manually monkey patch or proxy a few libraries to make them work, but I'll have to test out this update and see if everything magically works
- SethMLarson 3y ago
  Oh wow, thanks for this story! Would love to hear more if you have time :) Good luck with testing it out. Note that we found an issue w/ emitting an InsecureRequestWarning by default. The request is perfectly secure, it's just we aren't telling the ConnectionPool that information (see: https://github.com/urllib3/urllib3/issues/3331 https://github.com/urllib3/urllib3/issues/3331)
codingglass 3y ago
This a a great example of good work getting 'upstreamed.' The go-to solution for using requests and urllib3 in the browser had been Koen Vossen's 'pyodide-http' package, which you'd install in PyScript/Pyodide and then call 'pyodide_http.patch_all()`. It monkey-patched requests and urllib3 to use the browser API's. [1] Then 'pyodide-http' got adopted into the Pyodide-included packages[2], so you didn't have to specify it for install, just import it and run patch_all(). And now a similar technique has been incorporated directly into urllib3. (Not a direct port - as far as I know Joe Marshall and others did a full reimplementation so that urllib3's API's would be maintained as much as possible). [2] Very cool. Edit: Looks like there's a note to this same effect on Seth Larson's post about urllib3's future. [3] [1] https://github.com/koenvo/pyodide-http https://github.com/koenvo/pyodide-http [2] https://pyodide.org/en/stable/usage/packages-in-pyodide.html https://pyodide.org/en/stable/usage/packages-in-pyodide.html [3] https://sethmlarson.dev/urllib3-is-fundraising-for-http2-support#support-for-webassembly-and-emscripten https://sethmlarson.dev/urllib3-is-fundraising-for-http2-sup...
hoodchatham 3y ago
aiohttp is still work in progress but it works with a monkey patch. https://github.com/aio-libs/aiohttp/pull/7803 https://github.com/aio-libs/aiohttp/pull/7803

108 points by SethMLarson 3y ago

Pyodide examples using urllib3 in browser still requires using fetch, so this still requires javascript. What does this practically mean? Does this ever point to a future where we can use urlib3 instead of fetch?

deleted 3y ago
[deleted]
SethMLarson 3y ago
Lead maintainer of urllib3 here: We're tracking all the options for how we can make the experience better for folks looking to use Python in the browser. Today our biggest blocker is that there's no socket or TLS APIs for Emscripten/WASI. If those were to become available we'd jump on them right away :) I think the biggest benefit of a foundational project like urllib3 getting browser support is that /many/ Python projects are built on top of urllib3 and many of them likely work in browsers now that urllib3 works in the browser. For example, Requests is built on top of urllib3 and now gets browser support for free! And this is recursive, how many projects use Requests (spoiler: many). Pretty exciting IMO :)
- canadiantim 3y ago
  wow that's awesome. That we could even talk about potentially using just python to make requests in browser is pretty wild (of course, as you say still blocks in the way - lack of socket or TLS APIs for Emscripten/WASI). Really cool stuff tho. All the best eh and thank you VERY much for the incredible urllib3 library.
- SushiHippie 3y ago
  Do you know if there are some plans for sockets or TLS APIs or are we far away from this?
  - SethMLarson 3y ago
    Browsers limit the ability for these platforms to use raw sockets, there simply is no API for it. The best that can be done /today/ is to use WebSockets, which are not the same thing and can't be used for HTTP requests without the server expecting a WebSocket connection: https://github.com/emscripten-core/emscripten/issues/5196#issuecomment-299442038 https://github.com/emscripten-core/emscripten/issues/5196#is...
    - SushiHippie 3y ago
      Sure, sorry if it wasn't clear from my question, it wasn't specifically about urllib3, but I meant if there are plans that browsers will make such an API available or if this is something that will never become real.
      - sillysaurusx 3y ago
        I doubt browsers would allow this, though gamedevs have wanted it for decades now. Particularly UDP.
        plopz 3y ago
        isn't the udp problem fixed with http3 unreliable webtransport?
        sillysaurusx 3y ago
        Is it? That would be cool. Under the hood is it actual udp?
        SushiHippie 3y ago
        HTTP3 uses QUIC, it's not "pure" UDP, but it's a protocol based on it https://en.wikipedia.org/wiki/QUIC https://en.wikipedia.org/wiki/QUIC
        3y ago
- at_a_remove 3y ago
  Thank you for your work on urllib3; I will probably never use Requests if I can help it. I'm glad that you're keeping up that "batteries included" tradition.
- mistrial9 3y ago
  except the browser context is being sandboxed with lots of access restriction, individual profiling, geolocation reporting, device identification, and more Big Tech does not like independent protocol operation it seems; therefore your interesting technical advance is likely to be a pawn in other, larger changes to the social contracts on the open net IMHO
msoad 3y ago
Am I misunderstanding this? You need JS to load the WASM module anyways, what does "requiring JS" means here?
- ploxiln 3y ago
  It means that urllib3 (python code) isn't actually making the http request (it's not making the tcp connection, tls auth, writing the request, or serializing the request headers ...) It may be better to describe this as a "shim". Any python code that depended on urllib3 is now transparently shimmed in browser context to use the fetch-api instead. Which is useful.
  - zarzavat 3y ago
    This is definitely out of scope for web browsers. The security implications of allowing arbitrary sockets are prohibitive. To do something like that then you’re going to need a different environment with a different set of security guarantees and expectations, more like a shell where the user is responsible for vouching for the code’s trustworthiness.
- canadiantim 3y ago
  You’re right. It just reflected a momentary misunderstanding of mine. WASM still requires JS.

hpeter 3y ago

why would you want to run python in the browser? Anyone can ELI5?

deleted 3y ago
[deleted]
rmnclmnt 3y ago
Lots of use-cases but to name a few: - Interactive documentation (static website) - Jupyter Notebooks without a server, great for education (Jupyter-Lite) - Python app alternative deployment as static webapp
cle 3y ago
There's a huge volume of existing Python code.
dharmab 3y ago
Python in Excel currently requires a separate cloud server to run Python formulas. Imagine this could run in the Excel web view instead.
- codingglass 3y ago
  Work keeping an eye on Chris Laffra's work in this area - he did a Python in Excel demo [1] a few months back, and while sheets aren't in his ltk framework [2] yet, the work he's done on reactivity-in-python-on-the-web there is really promising. [1] https://github.com/laffra/excel-in-python https://github.com/laffra/excel-in-python [2] https://laffra.github.io/ltk/ https://laffra.github.io/ltk/
pphysch 3y ago
For real demos of your Python libs: https://pygments.org/demo/ https://pygments.org/demo/
Austizzle 3y ago
The film (especially vfx) industry runs on Python, and so much code is written for Python. Same reason you'd want to run your server in node.js so it matches your front end code ( and you can reuse code on the front end and backend). Pyscript lets us have the same programming language in the front end and the backend, and access to all of our internal packages on the front end. A lot of vfx software is written in QT, so at a previous company I actually made a wrapper around a subset of the QT interface and widgets, and replicated it in Pyscript, so we could write an app once and run it from inside vfx applications, or run it in the browser with no additional coding. It wasn't perfect, but it worked and made things simpler

Extigy 3y ago

Since this is using fetch/XHR under the hood, I guess requests from the browser are restricted only to same-origin URLs or servers responding with permissive CORS headers?

SethMLarson 3y ago
Correct! The examples we used were making requests to httpbin.org, but I also was able to query GitHub's API :) `data = urllib3.request("GET", "https://api.github.com").json https://api.github.com").json()`

Austizzle 3y ago

Incredible! Two years ago I was experimenting in porting some code at the company I was at to python in the browser (instead of QT desktop apps) and this was the biggest thing I had to work around! I had to manually monkey patch or proxy a few libraries to make them work, but I'll have to test out this update and see if everything magically works

SethMLarson 3y ago
Oh wow, thanks for this story! Would love to hear more if you have time :) Good luck with testing it out. Note that we found an issue w/ emitting an InsecureRequestWarning by default. The request is perfectly secure, it's just we aren't telling the ConnectionPool that information (see: https://github.com/urllib3/urllib3/issues/3331 https://github.com/urllib3/urllib3/issues/3331)

codingglass 3y ago

This a a great example of good work getting 'upstreamed.' The go-to solution for using requests and urllib3 in the browser had been Koen Vossen's 'pyodide-http' package, which you'd install in PyScript/Pyodide and then call 'pyodide_http.patch_all()`. It monkey-patched requests and urllib3 to use the browser API's. [1] Then 'pyodide-http' got adopted into the Pyodide-included packages[2], so you didn't have to specify it for install, just import it and run patch_all(). And now a similar technique has been incorporated directly into urllib3. (Not a direct port - as far as I know Joe Marshall and others did a full reimplementation so that urllib3's API's would be maintained as much as possible). [2] Very cool. Edit: Looks like there's a note to this same effect on Seth Larson's post about urllib3's future. [3] [1] https://github.com/koenvo/pyodide-http https://github.com/koenvo/pyodide-http [2] https://pyodide.org/en/stable/usage/packages-in-pyodide.html https://pyodide.org/en/stable/usage/packages-in-pyodide.html [3] https://sethmlarson.dev/urllib3-is-fundraising-for-http2-support#support-for-webassembly-and-emscripten https://sethmlarson.dev/urllib3-is-fundraising-for-http2-sup...

hoodchatham 3y ago

aiohttp is still work in progress but it works with a monkey patch. https://github.com/aio-libs/aiohttp/pull/7803 https://github.com/aio-libs/aiohttp/pull/7803