4 ms·

This is awesome, I'm happy that Cloudflare is adding more attention into running Python via WebAssembly at the Edge. I'll try to summarize on how they got it r

This is awesome, I'm happy that Cloudflare is adding more attention into running Python via WebAssembly at the Edge. I'll try to summarize on how they got it running and what are the drawbacks that they have from their current approach (note: I have deep context on running Python with WebAssembly at the Edge as part of my work in Wasmer). Cloudflare Workers are enabling Python at the Edge by using Pyodide [1] (Python compiled to WebAssembly via Emscripten). They bundled Pyodide into Workerd [2], and then use V8 snapshots [3] to try to accelerate startup times. On their best case, cold starts of Python in Cloudflare Workers are about 1 second. While this release is great as it allows them to measure the interest of running Python at the Edge, it has some drawbacks. So, what are those? * Being tied to use only one version of Python/Pyodide (the one that Workerd embeds) * Package resolution is quite hacky and tied to workerd. Only precompiled "native packages" will be allowed to be used at runtime (eg. using a specific version of numpy will turn to be challenging) * Architecturally tied to the JS/v8 world, which may show some challenges as they aim to reduce cold start times (in my opinion, it will be quite hard for them to achieve <100ms startup time with their current architecture). In any case, I welcome this initiative with my open hands and look forward all the cool apps that people will now build with this! [1] https://pyodide.org/ https://pyodide.org/ [2] https://github.com/cloudflare/workerd/blob/main/docs/pyodide.md https://github.com/cloudflare/workerd/blob/main/docs/pyodide... [3] https://github.com/cloudflare/workerd/pull/1875 https://github.com/cloudflare/workerd/pull/1875 Edit: updated wording from "proof of concept" to "release" to reflect the clarification from the Cloudflare team

panqueca 3y ago
Does this architecture supports uvloop?
- syrusakbary 3y ago
  As far as I know uvloop is not supported in Pyodide, mainly because it requires compiling libuv into WebAssembly (which is possible but not trivial). In any case, it shall be possible to run uvloop fully inside of WebAssembly. However, doing so will prove challenging using their current architecture
- hoodchatham 3y ago
  Pyodide uses its own event loop which just subscribes to the JavaScript event loop. My suspicion is that this will be more efficient than using uvloop since v8's event loop is quite well optimized. It also allows us to await JavaScript thenables from Python and Python awaitables from JavaScript, whereas I would be worried about how this behaves with separate event loops. Also, porting uvloop would probably be hard.
deanCommie 3y ago
> (in my opinion, it will be quite hard for them to achieve <100ms startup time with their current architecture). Who's running python workloads with sub-100ms latency requirements?
kflansburg 3y ago
I believe that your summary misunderstands how we will handle versioning. The pyodide /package versions will be controlled by the compatibility date, and we will be able to support multiple in production at once. For packages like langchain (or numpy as you mentioned) the plan is to update quite frequently. Could you expand on why you believe V8 will be a limiting factor? It is quite a powerful Wasm runtime, and most of the optimizations we have planned don’t really depend on the underlying engine. Edit: Also just want to clarify that this is not a POC, it is a Beta that we will continue improving on and eventually GA.
- syrusakbary 3y ago
  > pyodide /package versions will be controlled by the compatibility date That's exactly the issue that I'm mentioning. Ideally you should be able to pin any Python version that you want to use in your app: 2.7, 3.8 or 3.9 regardless of a Workerd compatibility date. Some packages might work in Python 3.11 but not in 3.12, for example. Unfortunately, Python doesn't have the full transpiler architecture that JS ecosystem has, and thus "packaging" Python applications into different "compatibility" bundles will prove much more challenging (webpack factor). > Could you expand on why you believe V8 will be a limiting factor? Sure thing! I think we probably all agree that V8 is a fantastic runtime. However, the tradeoffs that make V8 great for a browser use case, makes the runtime more challenging for Edge environments (where servers can do more specialized workloads on trusted environments). Namely, those are: * Cold starts: V8 Isolates are a bit heavy to initialize. On it's current form it can add up from ~2-5ms in startup just by initializing an Isolate * Snapshots can be quite heavy to save and restore * Not architected with the Edge use case in mind: there are many tricks that you can do if you skip the JS middleware and go all in into a Wasm runtime, that are hard to do with the current V8/Workerd architecture. In any case, I would love to be proven wrong on the long term and I cheer for <100ms cold starts when running Python in Cloudflare Workers. Keep up the good work!
  - kflansburg 3y ago
    We discussed a separate configuration field for Python version. It’s not technically challenging, this was a design choice we made to simplify configuration for users and encourage more efficiencies in terms of shared dependencies. Your concerns about V8 would impact JavaScript Workers as well and do not match what we see in production. It is also definitely possible to invoke C++ host functions directly from Wasm with V8.
deleted 3y ago
[deleted]
syrusakbary 3y ago
As a side note, Wasmer offers an Edge product that has none of the drawbacks commented when running Python in Cloudflare Workers, providing incredibly fast cold-start times: https://wasmer.io/templates?language=python https://wasmer.io/templates?language=python
- kentonv 3y ago
  Wasmer claims to cold start in 50ns. This is obviously impossible: That's 1000x faster than an NVMe read, which is about the fastest cold storage you can get. At least make your claims credible before posting them in competitors' HN threads.
  - syrusakbary 3y ago
    > Wasmer claims to cold start in 50ns I believe this statement was based on Wasmer Edge product page, which was mainly measuring instantiation time if the module is already loaded in the Edge Node (which can be assumed true for most common programs such as WinterJS, static-web-server and more). As it seems the 50ns timing was not well understood, we updated our product page to reflect a more accurate timing thanks to your feedback. We can get into sub-millisecond cold starts (loading a module from disk, instantiating it and serving the first request) for the best case scenario, a timing now properly reflected in the Wasmer Edge product page. In any case, and to be clear, we can get into much faster cold starts than the full second than Cloudflare Workers currently offers for Python. Wasmer Edge should be at least 10x faster than Cloudflare Workers for Python cold starts, with a cold-start time of less than <100ms. You can also expect to see much faster cold start times in next releases :)
    - kentonv 3y ago
      > measuring instantiation time if the module is already loaded in the Edge Node (which can be assumed true for most common programs such as WinterJS, static-web-server and more) It sounds like you're saying that your hosting never incurs cold starts in the first place, because you always preload all customers' applications into memory before serving any traffic. That's a fine optimization, if you can fit everything. Cloudflare Workers also preloads the most popular apps before serving traffic -- but we can't load all applications this way since they wouldn't all fit into memory. We only consider it a "cold start" when a request arrives for an application that wasn't preloaded.

panqueca 3y ago

Does this architecture supports uvloop?

syrusakbary 3y ago
As far as I know uvloop is not supported in Pyodide, mainly because it requires compiling libuv into WebAssembly (which is possible but not trivial). In any case, it shall be possible to run uvloop fully inside of WebAssembly. However, doing so will prove challenging using their current architecture
hoodchatham 3y ago
Pyodide uses its own event loop which just subscribes to the JavaScript event loop. My suspicion is that this will be more efficient than using uvloop since v8's event loop is quite well optimized. It also allows us to await JavaScript thenables from Python and Python awaitables from JavaScript, whereas I would be worried about how this behaves with separate event loops. Also, porting uvloop would probably be hard.

deanCommie 3y ago

> (in my opinion, it will be quite hard for them to achieve <100ms startup time with their current architecture). Who's running python workloads with sub-100ms latency requirements?

kflansburg 3y ago

I believe that your summary misunderstands how we will handle versioning. The pyodide /package versions will be controlled by the compatibility date, and we will be able to support multiple in production at once. For packages like langchain (or numpy as you mentioned) the plan is to update quite frequently. Could you expand on why you believe V8 will be a limiting factor? It is quite a powerful Wasm runtime, and most of the optimizations we have planned don’t really depend on the underlying engine. Edit: Also just want to clarify that this is not a POC, it is a Beta that we will continue improving on and eventually GA.

syrusakbary 3y ago
> pyodide /package versions will be controlled by the compatibility date That's exactly the issue that I'm mentioning. Ideally you should be able to pin any Python version that you want to use in your app: 2.7, 3.8 or 3.9 regardless of a Workerd compatibility date. Some packages might work in Python 3.11 but not in 3.12, for example. Unfortunately, Python doesn't have the full transpiler architecture that JS ecosystem has, and thus "packaging" Python applications into different "compatibility" bundles will prove much more challenging (webpack factor). > Could you expand on why you believe V8 will be a limiting factor? Sure thing! I think we probably all agree that V8 is a fantastic runtime. However, the tradeoffs that make V8 great for a browser use case, makes the runtime more challenging for Edge environments (where servers can do more specialized workloads on trusted environments). Namely, those are: * Cold starts: V8 Isolates are a bit heavy to initialize. On it's current form it can add up from ~2-5ms in startup just by initializing an Isolate * Snapshots can be quite heavy to save and restore * Not architected with the Edge use case in mind: there are many tricks that you can do if you skip the JS middleware and go all in into a Wasm runtime, that are hard to do with the current V8/Workerd architecture. In any case, I would love to be proven wrong on the long term and I cheer for <100ms cold starts when running Python in Cloudflare Workers. Keep up the good work!
- kflansburg 3y ago
  We discussed a separate configuration field for Python version. It’s not technically challenging, this was a design choice we made to simplify configuration for users and encourage more efficiencies in terms of shared dependencies. Your concerns about V8 would impact JavaScript Workers as well and do not match what we see in production. It is also definitely possible to invoke C++ host functions directly from Wasm with V8.

deleted 3y ago

[deleted]

syrusakbary 3y ago

As a side note, Wasmer offers an Edge product that has none of the drawbacks commented when running Python in Cloudflare Workers, providing incredibly fast cold-start times: https://wasmer.io/templates?language=python https://wasmer.io/templates?language=python

kentonv 3y ago
Wasmer claims to cold start in 50ns. This is obviously impossible: That's 1000x faster than an NVMe read, which is about the fastest cold storage you can get. At least make your claims credible before posting them in competitors' HN threads.
- syrusakbary 3y ago
  > Wasmer claims to cold start in 50ns I believe this statement was based on Wasmer Edge product page, which was mainly measuring instantiation time if the module is already loaded in the Edge Node (which can be assumed true for most common programs such as WinterJS, static-web-server and more). As it seems the 50ns timing was not well understood, we updated our product page to reflect a more accurate timing thanks to your feedback. We can get into sub-millisecond cold starts (loading a module from disk, instantiating it and serving the first request) for the best case scenario, a timing now properly reflected in the Wasmer Edge product page. In any case, and to be clear, we can get into much faster cold starts than the full second than Cloudflare Workers currently offers for Python. Wasmer Edge should be at least 10x faster than Cloudflare Workers for Python cold starts, with a cold-start time of less than <100ms. You can also expect to see much faster cold start times in next releases :)
  - kentonv 3y ago
    > measuring instantiation time if the module is already loaded in the Edge Node (which can be assumed true for most common programs such as WinterJS, static-web-server and more) It sounds like you're saying that your hosting never incurs cold starts in the first place, because you always preload all customers' applications into memory before serving any traffic. That's a fine optimization, if you can fit everything. Cloudflare Workers also preloads the most popular apps before serving traffic -- but we can't load all applications this way since they wouldn't all fit into memory. We only consider it a "cold start" when a request arrives for an application that wasn't preloaded.