7 ms·

As a maintainer of the Skulpt Python-to-JS compiler (https://skulpt.org https://skulpt.org), I can reasonably attempt to answer this question. Skulpt compiles

As a maintainer of the Skulpt Python-to-JS compiler (https://skulpt.org https://skulpt.org), I can reasonably attempt to answer this question. Skulpt compiles Python to JS in one shot, so it's not a million miles away from the architecture Brett envisages. (I gave a lightning talk about it here: https://anvil.works/blog/pycon-talk https://anvil.works/blog/pycon-talk). Things I've observed: - "Ordinary code" requires remarkably little fidelity. Python has a very strong "simplicity culture", so most code avoids being clever. If you're creating an environment for users to program against (as we do in Anvil), it's astonishing how little of Python you need to support. - "Be able to use popular Python libraries" is really, really hard. The ordinary user doesn't use all the esoteric stuff Brett mentions, but libraries do, left and right. Even porting PyPy's `datetime` implementation to Skulpt required filling out a lot of odd little corners of the object implementation. We still have a pretty crummy subset of `unittest` because it's eye-achingly dynamic - for example, it really puts metaclasses through their paces. (Metaclasses are a perfect example of a powerful feature that really depends on details that descend directly from CPython: The API exposes the fact that variable scopes are literally `dict` objects, that classes are created dynamically at runtime, etc.) - Finally (and most unfortunately): A lot of the strength of Python is in its ecosystem, and a lot of that ecosystem relies on native code. (Again, much more than you'd think. When porting stdlib modules from PyPy to Skulpt, we have often been disappointed by how many of them use native code - even in PyPy!) The Python-to-C interface is really, really CPython-shaped. Anyone who wants to be compatible with numpy, scipy, etc, needs to literally emulate CPython's data structures, and then translate them to whatever faster/simpler thing they use internally. This emulation overhead is the main reason PyPy is slower than CPython on some workloads. (In Skulpt, we haven't even attempted this. It doesn't bother us much in Anvil, of course, as the server-side CPython is just a function call away, but for Brett's use case I'm guessing it would really hurt. One day we might try, with a JS-to-wasm bridge and doing the emulation in wasm, but it would be a hell of ajob.)

w0utert 6y ago
>> - Finally (and most unfortunately): A lot of the strength of Python is in its ecosystem, and a lot of that ecosystem relies on native code. I was thinking the same thing... One can wonder how much value there is left if you build an alternative Python interpreter that can e.g. run in the browser, but cannot support C extensions. I mean I like Python for what it is, but once you stray from the common/intended use cases, e.g. trying to embed the interpreter in a C++ program, skeletons fall out of closets around every corner. For all it's merits, CPython feels like a house of cards, burdened by a legacy of bad engineering decisions it cannot get rid of anymore. It made me realize it's better to stop fighting it, use CPython as-is (which is great), and use other scripting languages for embedding, live-coding, web, etc. I very much prefer Python over JavaScript for example, but even then it wouldn't cross my mind to use some kind of Python-to-JS implementation for web frontend code, honestly.
- meredydd 6y ago
  > CPython feels like a house of cards, burdened by a legacy of bad engineering decisions it cannot get rid of anymore At PyBay last year, I ended up with a knot of Python core devs talking about this problem, and talking about building a better, more portable API for native modules. The transition would be nasty (the 2->3 transition was..scarring), but it would solve this problem at a stroke. > I very much prefer Python over JavaScript for example, but even then it wouldn't cross my mind I would encourage you to at least check out Anvil (https://anvil.works https://anvil.works). It's more than a Python->JS compiler; it's a reimagining of what the web as a platform could look like. FWIW, I agree with you that it's a bad idea to swap out JS for Python at one layer, while keeping the rest of the stack intact. It's extra complexity, and it doesn't solve the actual problem with the Web, which is the sprawling complexity of the stack and all the weird frameworks that work around it (while themselves making the problem worse). The Big Idea of Anvil is to replace all those different layers of abstraction with one big, coherent abstraction. That's one reason why we deliberately chose Python rather than JS, so that users wouldn't reflexively reach out and break that abstraction every time they Googled how to do something.
  - throwaway894345 6y ago
    > a better, more portable API for native modules Pretty sure this exists already: https://github.com/pyhandle/hpy https://github.com/pyhandle/hpy
  - pletsch 6y ago
    I really liked Anvil, the thing that made me go back was the requirement for an enterprise account to not host on AWS.
    - meredydd 6y ago
      Good news! We open-sourced the runtime, and the standalone App Server, so you can now deploy anywhere you like: https://anvil.works/open-source https://anvil.works/open-source (And yes, it even works with the Free plan)
foota 6y ago
Our of curiosity, why not just compile cpython to webasm and call it a day by shimming in module loading? I'm sure it's for performance (or extensibility?), but I'm curious where it really falls down.
- Doxin 6y ago
  That totally does work, but in practice it makes the initial bundle download too big to really be workable for the average website.
  - 6gvONxR4sf7o 6y ago
    Would the same be true of javascript? If chrome didn't come with V8 in it, would it be a lot for the average website?
    - pansa2 6y ago
      One way to find out would be to compile QuickJS [0] to WebAssembly. [0] https://bellard.org/quickjs/ https://bellard.org/quickjs/
      - asaddhamani 6y ago
        Seems to be 945KB in WASM. http://numcalc.com/ http://numcalc.com/
- meredydd 6y ago
  This is exactly what Pyodide (mentioned in the article) does, and it works great for some use cases. The problem is that downloading and wasm-compiling the entirety of CPython and all its native modules is big and slow. A colleague of mine collected and compared a few Python-in-the-browser implementations - including Skulpt and Pyodide - with code samples and a description of the trade-offs. You might find the write-up interesting: https://anvil.works/blog/python-in-the-browser-talk https://anvil.works/blog/python-in-the-browser-talk
pansa2 6y ago
> we have often been disappointed by how many [stdlib modules] use native code - even in PyPy! The CPython interpreter is so slow that it makes sense to implement as much as possible in C. In the past, I've used this as an argument against slow interpreted languages - a faster implementation would not only run code faster, it would allow code to be written faster because more of it could be in a higher-level language. However, I'm surprised that PyPy uses a lot of native code - not only does it have a fast JIT, but its whole raison d'etre is to be Python-written-in-Python - that's even the project's name!
throwaway894345 6y ago
It’s interesting to me that Python depends so much on native code to have even reasonable performance, but because so much of the ecosystem is native code and because the native code interface is “CPython shaped”, it becomes incredibly difficult to improve CPython’s performance because it would very likely change the shape of CPython. Of course we could have the best of both worlds—a fast Python that allowed you to use native extensions—but the native extension interface would have to be minimized to something that would give Python implementations some ability to look different than CPython today. And when you have a performant Python implementation, much less needs to be implemented natively, which makes things like package management and cross platform development much easier (specifically you don’t have to worry about cross-compiling and testing on your target becomes considerably less important because Python is portable while native extensions likely are not). The latter becomes a lot more pertinent if you’re on x86 targeting ARM or if you’re on a new ARM MacBook targeting x86 or if you’re targeting WASM from either. These cases aren’t common today, but ARM and WASM seem to becoming increasingly likely in the future.
- mumblemumble 6y ago
  I think that what's even more interesting to me is that people perceive this as such a preoccupying problem. When I was a young little programmer, the dream was to have a base layer of highly performant native code for handling the heavy lifting, tightly coupled to a very high level scripting language that wasn't really even trying to be performant. The thought being that the scripting language should be more focused on flexibility than performance, because, for the bits of the software that it handled, developer productivity was more important than raw performance. Fast forward a year or two, and we are living the dream. It's Python, and, while I could nitpick implementation details and language features all day, the big picture is that it's pretty awesome. So awesome that, for the machine learning and data engineering work that I do, the most performant and most productive language I have to work in is Python. (Also R, which is a pretty similar technical story, but I realize that R lost the popularity contest, and I understand why.) I guess it starts to feel to me like we're making the perfect the enemy of the good, here. We've got a well-established, productive, flexible, and easy-to-learn language where top-tier performance can generally be implemented as a library solution. Despite its many imperfections, that's pretty darned good. In my career to date, I haven't seen much cause to believe that I could realistically expect much better than that. I think that, when I do need better performance, I'd rather write a library than rewrite the language.
acbart 6y ago
I love Skulpt! I'm so glad that it's being actively maintained. Thanks for all your hard work!
mamcx 6y ago
Lurking here is the main problem, not only of python but of so many langs: Dependency on the C-ABI. EVERY interfacing with C sucks, even made on C! Hopefully web assembly become good enough to change that...
sanxiyn 6y ago
Starlark is a Python dialect implemented by Google. It is rather liberal with compatibility, but as you observed, none of that matters for "ordinary code". It doesn't even pretend to support existing Python library ecosystem. In exchange, Starlark is GIL-free. This was the primary motivation to reimplement. It really is that easy if you don't need to be compatible. I think Starlark is a good approximation of the core of Python. It is easy to implement. It DOES 100% look and feel like Python, and if you use it as Python-as-pseudo-code mode, you will never realize it is not Python. On the other hand, almost no Python library will run. This feels paradoxical, but that's what it is. https://github.com/bazelbuild/starlark https://github.com/bazelbuild/starlark
- Tyr42 6y ago
  Wait I didn't realize it wasn't python. Google engineer for 3 years now.
  - sanxiyn 6y ago
    Haha. It is 100% obvious if you are a Python implementer that Starlark IS NOT Python, but it is ALSO 100% obvious if you are a Python user that Starlark IS Python. Strange, isn't it? Such is life.
    - laurentlb 6y ago
      A lot of Starlark code at Google is still tested using a Python framework. So the code is compatible with Python to a fairly large extent. A Starlark interpreter can easily be compiled to webassembly and run in a browser. I did it here: http://laurent.le-brun.eu/starlark/ http://laurent.le-brun.eu/starlark/
- joshuamorton 6y ago
  Just to emphasize this: https://github.com/bazelbuild/bazel-skylib/commit/9948d5538b8c400aec0dc23c545ec5094f8e079b https://github.com/bazelbuild/bazel-skylib/commit/9948d5538b... It's possible to `exec` bazel and have it work, with some trickery.
- 6y ago
ris 6y ago
I think the answer to the "how to support enough of the ecosystem to be useful" question really lies in making it super straightforward for package authors to integrate it into their packaging/testing/ci pipelines. If people test for your implementation, they will very quickly pick up when they're starting to use a feature that may unnecessarily exclude users and hopefully avoid it before it becomes too deeply ingrained in their design.

w0utert 6y ago

>> - Finally (and most unfortunately): A lot of the strength of Python is in its ecosystem, and a lot of that ecosystem relies on native code. I was thinking the same thing... One can wonder how much value there is left if you build an alternative Python interpreter that can e.g. run in the browser, but cannot support C extensions. I mean I like Python for what it is, but once you stray from the common/intended use cases, e.g. trying to embed the interpreter in a C++ program, skeletons fall out of closets around every corner. For all it's merits, CPython feels like a house of cards, burdened by a legacy of bad engineering decisions it cannot get rid of anymore. It made me realize it's better to stop fighting it, use CPython as-is (which is great), and use other scripting languages for embedding, live-coding, web, etc. I very much prefer Python over JavaScript for example, but even then it wouldn't cross my mind to use some kind of Python-to-JS implementation for web frontend code, honestly.

meredydd 6y ago
> CPython feels like a house of cards, burdened by a legacy of bad engineering decisions it cannot get rid of anymore At PyBay last year, I ended up with a knot of Python core devs talking about this problem, and talking about building a better, more portable API for native modules. The transition would be nasty (the 2->3 transition was..scarring), but it would solve this problem at a stroke. > I very much prefer Python over JavaScript for example, but even then it wouldn't cross my mind I would encourage you to at least check out Anvil (https://anvil.works https://anvil.works). It's more than a Python->JS compiler; it's a reimagining of what the web as a platform could look like. FWIW, I agree with you that it's a bad idea to swap out JS for Python at one layer, while keeping the rest of the stack intact. It's extra complexity, and it doesn't solve the actual problem with the Web, which is the sprawling complexity of the stack and all the weird frameworks that work around it (while themselves making the problem worse). The Big Idea of Anvil is to replace all those different layers of abstraction with one big, coherent abstraction. That's one reason why we deliberately chose Python rather than JS, so that users wouldn't reflexively reach out and break that abstraction every time they Googled how to do something.
- throwaway894345 6y ago
  > a better, more portable API for native modules Pretty sure this exists already: https://github.com/pyhandle/hpy https://github.com/pyhandle/hpy
- pletsch 6y ago
  I really liked Anvil, the thing that made me go back was the requirement for an enterprise account to not host on AWS.
  - meredydd 6y ago
    Good news! We open-sourced the runtime, and the standalone App Server, so you can now deploy anywhere you like: https://anvil.works/open-source https://anvil.works/open-source (And yes, it even works with the Free plan)

foota 6y ago

Our of curiosity, why not just compile cpython to webasm and call it a day by shimming in module loading? I'm sure it's for performance (or extensibility?), but I'm curious where it really falls down.

Doxin 6y ago
That totally does work, but in practice it makes the initial bundle download too big to really be workable for the average website.
- 6gvONxR4sf7o 6y ago
  Would the same be true of javascript? If chrome didn't come with V8 in it, would it be a lot for the average website?
  - pansa2 6y ago
    One way to find out would be to compile QuickJS [0] to WebAssembly. [0] https://bellard.org/quickjs/ https://bellard.org/quickjs/
    - asaddhamani 6y ago
      Seems to be 945KB in WASM. http://numcalc.com/ http://numcalc.com/
meredydd 6y ago
This is exactly what Pyodide (mentioned in the article) does, and it works great for some use cases. The problem is that downloading and wasm-compiling the entirety of CPython and all its native modules is big and slow. A colleague of mine collected and compared a few Python-in-the-browser implementations - including Skulpt and Pyodide - with code samples and a description of the trade-offs. You might find the write-up interesting: https://anvil.works/blog/python-in-the-browser-talk https://anvil.works/blog/python-in-the-browser-talk

pansa2 6y ago

> we have often been disappointed by how many [stdlib modules] use native code - even in PyPy! The CPython interpreter is so slow that it makes sense to implement as much as possible in C. In the past, I've used this as an argument against slow interpreted languages - a faster implementation would not only run code faster, it would allow code to be written faster because more of it could be in a higher-level language. However, I'm surprised that PyPy uses a lot of native code - not only does it have a fast JIT, but its whole raison d'etre is to be Python-written-in-Python - that's even the project's name!

throwaway894345 6y ago

It’s interesting to me that Python depends so much on native code to have even reasonable performance, but because so much of the ecosystem is native code and because the native code interface is “CPython shaped”, it becomes incredibly difficult to improve CPython’s performance because it would very likely change the shape of CPython. Of course we could have the best of both worlds—a fast Python that allowed you to use native extensions—but the native extension interface would have to be minimized to something that would give Python implementations some ability to look different than CPython today. And when you have a performant Python implementation, much less needs to be implemented natively, which makes things like package management and cross platform development much easier (specifically you don’t have to worry about cross-compiling and testing on your target becomes considerably less important because Python is portable while native extensions likely are not). The latter becomes a lot more pertinent if you’re on x86 targeting ARM or if you’re on a new ARM MacBook targeting x86 or if you’re targeting WASM from either. These cases aren’t common today, but ARM and WASM seem to becoming increasingly likely in the future.

mumblemumble 6y ago
I think that what's even more interesting to me is that people perceive this as such a preoccupying problem. When I was a young little programmer, the dream was to have a base layer of highly performant native code for handling the heavy lifting, tightly coupled to a very high level scripting language that wasn't really even trying to be performant. The thought being that the scripting language should be more focused on flexibility than performance, because, for the bits of the software that it handled, developer productivity was more important than raw performance. Fast forward a year or two, and we are living the dream. It's Python, and, while I could nitpick implementation details and language features all day, the big picture is that it's pretty awesome. So awesome that, for the machine learning and data engineering work that I do, the most performant and most productive language I have to work in is Python. (Also R, which is a pretty similar technical story, but I realize that R lost the popularity contest, and I understand why.) I guess it starts to feel to me like we're making the perfect the enemy of the good, here. We've got a well-established, productive, flexible, and easy-to-learn language where top-tier performance can generally be implemented as a library solution. Despite its many imperfections, that's pretty darned good. In my career to date, I haven't seen much cause to believe that I could realistically expect much better than that. I think that, when I do need better performance, I'd rather write a library than rewrite the language.

acbart 6y ago

I love Skulpt! I'm so glad that it's being actively maintained. Thanks for all your hard work!

mamcx 6y ago

Lurking here is the main problem, not only of python but of so many langs: Dependency on the C-ABI. EVERY interfacing with C sucks, even made on C! Hopefully web assembly become good enough to change that...

sanxiyn 6y ago

Starlark is a Python dialect implemented by Google. It is rather liberal with compatibility, but as you observed, none of that matters for "ordinary code". It doesn't even pretend to support existing Python library ecosystem. In exchange, Starlark is GIL-free. This was the primary motivation to reimplement. It really is that easy if you don't need to be compatible. I think Starlark is a good approximation of the core of Python. It is easy to implement. It DOES 100% look and feel like Python, and if you use it as Python-as-pseudo-code mode, you will never realize it is not Python. On the other hand, almost no Python library will run. This feels paradoxical, but that's what it is. https://github.com/bazelbuild/starlark https://github.com/bazelbuild/starlark

Tyr42 6y ago
Wait I didn't realize it wasn't python. Google engineer for 3 years now.
- sanxiyn 6y ago
  Haha. It is 100% obvious if you are a Python implementer that Starlark IS NOT Python, but it is ALSO 100% obvious if you are a Python user that Starlark IS Python. Strange, isn't it? Such is life.
  - laurentlb 6y ago
    A lot of Starlark code at Google is still tested using a Python framework. So the code is compatible with Python to a fairly large extent. A Starlark interpreter can easily be compiled to webassembly and run in a browser. I did it here: http://laurent.le-brun.eu/starlark/ http://laurent.le-brun.eu/starlark/
joshuamorton 6y ago
Just to emphasize this: https://github.com/bazelbuild/bazel-skylib/commit/9948d5538b8c400aec0dc23c545ec5094f8e079b https://github.com/bazelbuild/bazel-skylib/commit/9948d5538b... It's possible to `exec` bazel and have it work, with some trickery.
6y ago

ris 6y ago

I think the answer to the "how to support enough of the ecosystem to be useful" question really lies in making it super straightforward for package authors to integrate it into their packaging/testing/ci pipelines. If people test for your implementation, they will very quickly pick up when they're starting to use a feature that may unnecessarily exclude users and hopefully avoid it before it becomes too deeply ingrained in their design.