15 ms·
Pyodide: Bringing the scientific Python stack to the browser
- jahewson 7y agoIs anyone using this yet? What’s your experience been?
- eapartridge 7y agoI've used it for a couple small projects in the context of trying Iodide. It's pretty good for data science—I can't speak to DOM manipulation that much, I've only done very minimal work with it for that. My two cents: it's bit rough around the edges—the performance is noticeably worse, and for data science, with medium-sized datasets it can freeze a tab for 10-15 seconds. Sometimes you get an odd edge case though where standard Python cannot run perfectly due to some combination of operations. That said, overall, I'm really excited by where it's going, and I've always been able to work around the limitations.
- VectorLock 7y agoI want to believe so bad. Being able to do web development with Python instead of JavaScript has been something I've dreamed about forever.
- aldoushuxley001 7y agoWhy not just use Django or Flask? Python web dev is great because you do have access to so many of these incredible scientific libraries like Pandas, etc.
- dekhn 7y agothe problem is now you have to rewrite your entire application around the web stack. That's a huge impedence mismatch. Taken to extreme, you end up with Pandas.
- VectorLock 7y agoThose work great for the "back end" but don't cover the aspects of the in-browser "front end" unless you do everything on the server side.
- _raoulcousins 7y agoI don't know anything about web dev, so maybe it's a stupid question, but you still need javascript for the front end, right? You can't build the front and back end development completely in python?
- dfinninger 7y agoI think they're implying that you use a Python web framework and server-side rendering.
- VectorLock 7y agoRight now, reasonably you can't. Maybe with Pyodide you may soon be able to.
- mythrwy 7y agoThere's Brython. https://brython.info/ https://brython.info/ I worked with a researcher a few years ago who used it to do somewhat complex frontend presentations. But why? Just learn a little javascript. Its fun.
- VectorLock 7y agoPython has spoiled every other programming language for me. I look at their syntax and it makes me sad.
- deleted 7y ago[deleted]
- mahmoudimus 7y agoIt’s never learn a little JavaScript. It’s learn webpack, learn parcel, learn react, learn vue, learn typescript, learn redux, learn npm, learn yarn etc etc. If it was python, at least it wouldn’t change every three months.
- speedplane 7y agoI find it ridiculous that javascript is the only language available for the web. Almost every other platform, from bare metal to Excel, support multiple languages. It's well past due to untie the browser from a mandated language.
- dodobirdlord 7y agoWell, that's largely the problem that WebAssembly is aiming to solve. Progress has been significant.
- fulafel 7y agoYou've been living under a rock :) People are using a variety languages that compile to JS on the frontend for many years. Elm, TypeScript, ClojureScript, Reason, Scala.js, Fable, etc. In fact even modern JS is often compiled into backward compatible legacy-JS these days. WebAssembly is porting the C machine model to the web, so for the foreseeable future JS is still going to be a better runtime for most GCd languages.
- didibus 7y agoPython no, but there's a lot of languages that have a JS transpiler, like ClojureScript, BuckleScript, Fantom, Nim, Haxe, etc. And of course, C and C++ such as what this project did. Actually, it seems there are some Python transpilers as well, though they don't seem as supported as what I listed. Anyways, my point is there's quite a lot of options on top of JavaScript. And for web dev, transpilers are the way to go, because bundle size matters for page loads, so options like this, where you compile the whole interpreter into a bundle aren't valid for that. For example, I use ClojureScript for web dev, and it works great. There are options to escape the shackles of having to write JS.
- pjmlp 7y agoIt was available as browser plugin since the early days, it just went nowhere. If I remember correctly, ActiveState was the one offering it.
- jononor 7y agoIncredible undertaking. Looking forward to play with it for simple machine learning tasks. I think that loading packages directly from PyPI will be a huge milestone.
- ngcc_hk 7y agoCan it run cuda? Otherwise not that practical. Minor interest. Can this approach run j/k or more important lisp?
- geofft 7y agoYou can do GPGPU stuff via WebGL, right?
- armitron 7y agoWebGL doesn't have compute shaders. There are (gross) hacks that "emulate" gpu compute on top of stock shaders but they're limited and can't possibly compete with CUDA.
- iandanforth 7y agoWhat now? https://www.tensorflow.org/js/guide/platform_environment#webgl_backend https://www.tensorflow.org/js/guide/platform_environment#web...
- deleted 7y ago[deleted]
- mschuetz 7y agoI don't know the details but the WebGL backend will have severe limitations because it doesn't use compute shaders or CUDA. This means that certain functionality like random writes to arbitrary buffers can only be emulated through workarounds that are like an order of magnitude slower than using compute shaders, and some things are going to be impossible entirely. There is something about CUDA in the link you provided but since that isn't natively supported by browser either, it will require the user to install something or use a server backend where communication between server and client is going to be an excruciatingly slow bottleneck. So no, this isn't a web thing and not usefull for web apps/web pages, except maybe for a very limited and small set of use cases.
- amasad 7y agoIt's great to see how far WebAssembly and Emscripten has come and this is a really cool app! > If you haven’t already tried Pyodide in action, go try it now! (50MB download) I wonder though how ready is it for production usage. At Repl.it, years ago[1], we moved away from browser-based execution to the cloud because it excluded many users who don't have the client-side firepower to download/parse/execute this much JS/WASM. Repl.it can already run a lot of the examples here [2] but I gotta say the DOM-integration is pretty neat. We could do interactive stuff on Repl.it using our new Graphics infrastructure (GFX)[3] but there is always the roundtrip delay. Here is the same matplotlib example running and streamed down on X11: https://repl.it/@amasad/matplotlib https://repl.it/@amasad/matplotlib [1]: https://news.ycombinator.com/item?id=16578943 https://news.ycombinator.com/item?id=16578943 [2]: https://repl.it/@amasad/pyodide-example https://repl.it/@amasad/pyodide-example [3]: https://repl.it/blog/gfx https://repl.it/blog/gfx
- ngcc_hk 7y agoThat is no doubt great tool. But imagine even one guy like me has manny gpu card for trying some large memory ai model (not even production). It just can’t work on the cloud during development. What the appeal of this is Standardise on certain way to do things and we can share with others. So far all python and r notebook. Is browser an option?
- zamadatix 7y agoBrowsers don't expose access to GPU compute APIs or multiple GPUs. Any web GPU usage you see right now is limited to shader programs on a single GPU.
- jononor 7y agoTensorflow.js can actually use WebGL to accelerate machine learning. Have not tried it though.
- zamadatix 7y agoTensorflow.js is a example of using the fragment shaders I was talking about: "Tensors are stored as WebGL textures and mathematical operations are implemented in WebGL shaders." https://www.tensorflow.org/js/guide/platform_environment https://www.tensorflow.org/js/guide/platform_environment
- jdpigeon 7y agoI'm looking forward to try out Pyodide in Electron so that I can make use of all the great Python scientific and signal processing libraries for data processing.
- chowes 7y agoWhy wouldn't you just use regular Python?
- jdpigeon 7y agoMy goal would primarily be to avoid having to make sure there's an active python installation with all the right dependencies set up on a client's computer. That's been a huge problem for me in the past during deployment (esp. older windows OS's). A nice side benefit I foresee would be easier coordination of state between a JS front end and a Python process running specialized computations. The current best approach I'm aware of now involves sending data back and forth as JSON and trying to mirror what's going in Python with JS state management. The system described in this article where JS and Python can reference the same objects (and somewhat the same typing?) sounds like it could be an improvement.
- arduinomancer 7y agoI wonder if that could be accomplished with something like PyInstaller? I guess if you're using JS features too then there might be other benefits.
- ptx 7y agoYou can bundle the whole Python distribution with your application and it will still be smaller than Electron. Recent releases of Python have an embeddable distribution for this purpose: https://docs.python.org/3.7/using/windows.html#windows-embeddable https://docs.python.org/3.7/using/windows.html#windows-embed... This is also the way Java is supposed to be deployed on client machines nowadays, apparently - using the jlink tool to create a bundle of the Java runtime for your application.
- 7y ago
- saboot 7y agoWould be curious to see if a version of Cython could be made to work with this. Much of my data analysis is me passing numpy arrays into C functions which Cython helps a lot with. So I would be looking for a version of Cython that would convert that to webassembly.
- mdboom 7y agoCython works for ahead-of-time compilation. (Pandas requires it, for example). Making it work in the browser would also mean putting a C compiler there, which people have done. I have no idea how well that would all hold together, though.
- bobbylarrybobby 7y agoWeb assembly?
- jononor 7y agoIdeally Cython would allow to compile directly to Webassembly (and maybe using Pyodine converters), without going through C and requiring a C compiler also. Of course that is a huge job and might very well never happen.
- make3 7y agoisn't it already doing that, converting the C generated by Cython to JS with emscripten? otherwise a huge amount of numerical libs wouldn't work. or did you mean directly without passing by C
- TeMPOraL 7y agoWow, it seems to work really well. After giving it a small test drive, I'm impressed. Solid work from Mozilla, and a pretty informative article too! That said, between this, iodide and ObservableHQ, I guess I give up. The browser is the new OS. My question now is - how can we make browser environment to be more like Emacs (bear with me)? My main complaints are: - Browser ergonomy absolutely sucks, and there's nothing you can do about it. You're at the mercy of each service's UI, which usually means you'll be clicking a lot and maybe sometimes get a keyboard shortcut here and there. Forget about advanced features, or consistency between sites. - You have near-zero control over your computing environment. Between sandboxing and half the code being obfuscated blob of transpiled JS (the other half of the code is on the server), you can maybe automate the UI a little with userscripts and fix it up with userstyles. - There's near-zero interoperability. Unless service authors agree on and implement some APIs, things can't talk to each other. Forget about making them talk. Whatever little wiring you can sometimes do (thanks to services like Zapier and IFTTT), you never have control over the process, and it always involves communication through third-party servers, even if all you'd like is to transfer some local data between browser tabs. If the browser is the next OS, can we make it suck less than desktop OSes in terms of ergonomy / productivity? Desktop OSes already suck in this regard compared to the promise of old Smalltalk and Lisp systems (that's why I live in Emacs most of the time), so all I see is a downwards trend. Current state of things is fine for casual use and casual users, but if I'm to spend 8+ hours a day doing serious work in a web browser, the browser needs to be better.
- Pete_D 7y agoI think it's impossible. We lost. The incentives aren't there for interoperability and support for client-side customization to meaningfully improve - big names will resist, either because it makes it easier for users to sidestep ads and tracking, or because the kinds of features it would require aren't within reach of the minimum viable user (https://www.reddit.com/r/dredmorbius/comments/69wk8y/the_tyranny_of_the_minimum_viable_user/ https://www.reddit.com/r/dredmorbius/comments/69wk8y/the_tyr...). Legally-enforced open APIs and client-agnosticity like you mention downthread sounds wonderful (maybe we could start by deprecating the User-Agent header) but for similar reasons, I can't see it ever taking off. Pessimism notwithstanding, here's an idea: imagine a browser extension where you could implement privileged pages/tabs that had access to other tabs' script environments/DOM, and could inject elements from other pages into itself. So you could write a workflow like: 1. open a Wikipedia page in a background tab (`const wiki = new Tab(etc...);`) 2. run some JS to pick out values from a <table> in it 3. open a Google Docs spreadsheet in a background tab 4. fill out the spreadsheet using the previously obtained data 5. as a bonus, inject the table + spreadsheet elements into the extension page for side-by-side reference Not sure what the emacs analogy would be - maybe something like org-mode and Babel being able to generate things using the contents of other buffers.
- deleted 7y ago[deleted]
- outlace 7y agoCan this be used to create a desktop application using Electron?
- arduinomancer 7y agoThat would be an interesting stack if you could write desktop apps using Python but use the DOM/Browser stuff for the UI layer.
- frmdstryr 7y agohttps://github.com/codelv/enaml-web/ https://github.com/codelv/enaml-web/ does exactly that
- outlace 7y agoYeah I often run into the need to develop basic UIs for data analysis for people with limited programming experience. Currently I have to settle for using Jupyter notebooks with embedded ipywidgets, but it's not still not very user-friendly. And the desktop app libraries like PyQt5 are overkill for what I need, and most of the visualization libraries I use are designed for jupyter notebooks/web e.g. Bokeh. If I could just port my jupyter notebook+widgets+plots into an Electron window and ship it to people that'd be awesome.
- geophph 7y agoMaybe check out Dash by Plotly. It's helped me address that exact situation as well.
- smoqadam 7y agoA couple of years ago I started to learn Python and made something like this. Desktop applications with Flask and PyQtWebkit. https://github.com/smoqadam/PyFladesk https://github.com/smoqadam/PyFladesk and this is the sample app: https://github.com/smoqadam/PyFladesk-rss-reader https://github.com/smoqadam/PyFladesk-rss-reader
- pjmlp 7y ago
- snthpy 7y agoLooks really nice! I prefer the look of the JSMD format to the Jupyter notebook format and hope that is something that might get integrated into JupyterHub as well. Question: is there some documentation for deploying your own Iodide server behind a firewall? The original Iodide post mentions this is possible.
- sandeepind 7y agowow now Mozilla browser can interpreter data science directly from client node. as it will be far better than java script. thx https://www.nineteenlives.com https://www.nineteenlives.com
- deleted 7y ago[deleted]
- alexgmcm 7y agoIt's a cool project technically, but I don't understand the use case? I mean surely everyone can just have real python/jupyterlab - if you run it in Docker it's easy to handle the dependencies etc. Am I missing something?
- arduinomancer 7y agoSounds like you could use it for interactive visualizations in the browser, similar to D3
- callahad 7y agoI gave a keynote at PyCon last year that touched on this: https://www.youtube.com/watch?v=ITksU31c1WY#t=32 https://www.youtube.com/watch?v=ITksU31c1WY#t=32 For me, the most important aspect of Pyodide is that it allows the scientific Python stack to go everywhere the Web goes. And the Web goes everywhere. That means I can share a link to a Pyodide notebook, and the recipient gets a fully interactive experience, no installation required. And that's really important for the generation that's getting their start in computing through classroom iPads, Chromebooks, etc. which do not easily allow for "real" Python.
- Vinnl 7y agoAs someone not familiar with any of this: is that not exactly what Jupyter Lab does? I understood that the main difference between Pyodide and that is that Pyodide allows you to also use e.g. D3 for rendering?
- drej 7y agoThere is one area where "computing in the browser" misses the mark a little - browser interfaces (like Jupyter) are often used not just for the convenience they offer, but also because they serve as thin clients, which provision vastly better computing resources than you have locally. So while I do some of my coding in the browser, the code executed is an EC2 far far away, close to my data, with excellent networking etc. There is very little that a Python stack directly in my browser would offer, to me (!) at least (your mileage may vary).
- _visgean 7y agoNot just that, but a lot of organization use the setup to give good starting interface to data scientist who might know nothing about how to set up all the python packages...
- maxnoe 7y agoA data scientist wo knows nothing about how to setup the tools he or she uses daily? Tell me more.
- drej 7y agoYes, it does happen. More often than I think is healthy.
- _visgean 7y agoWell I have some friends doing genetics and their infrastructure is maintained by IT team. For them the whole programming experience is connecting to jupyter server running on pre-configured server. I think there are even some online services offering this kind of configuration for teams.
- Hendrikto 7y ago> It’s also been argued more generally that Python not running in the browser represents an existential threat to the language I do not believe that.