7 ms·

Iodide: An experimental Mozilla tool for data exploration on the web

300 points by pablobaz 8y ago

pablobaz 8y ago
"Over the next couple months, we added Numpy, Pandas, and Matplotlib" The python data stack running in web assembly!
- paulgb 8y ago
  And it's also available standalone for other applications! https://github.com/iodide-project/pyodide https://github.com/iodide-project/pyodide Hats off to the team, this is some really neat stuff and I'm happy to see it coming from Mozilla.
bayesian_horse 8y ago
I was trying to make coroutines work through the javascript event loop, such that you could basically "await" any javascript Promise (or wrap a Promise in an awaitable). So you'd have things like 'await sleep(1)' which would be implemented with a Promise and setTimeout. Or 'await fetch("/api/v1/list")'. Unfortunately I haven't gotten very far yet.
- bijection 8y ago
  I'm pretty sure you can await any javascript Promise! For example, running `await Promise.resolve(3)` in Chrome's js console prints `3`.
  - alexkavon 8y ago
    Also MDN has a fantastic Promise article with examples about wrapping a function with a promise.[0] [0] https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...
  - JonathonW 8y ago
    It's a little unclear, but I think he wants to integrate Python (Pyodide) coroutines into the JS event loop-- so async/await of Python code works seamlessly in the browser, and you can await Python code from within JS (and vice versa). Any JS Promise is awaitable from JS; JS async/await is (more or less) syntactic sugar on top of promises.
  - bayesian_horse 8y ago
    Sorry, I meant to use "await" in Python to await Javascript Promises.
- KenoFischer 8y ago
  My preferred way to do this would be a compiler pass that rewrites everything to use the explicitly managed stack. I had started on writing that transform, but ran out of time. That's currently the major blocker for making the Julia/Iodide integration work well.
kasperset 8y ago
Interesting to see this tool along with Observablehq.
ZeroCool2u 8y ago
The SWE in me is very enthusiastic about this. I love the push that Mozilla is making with WASM in really practical and tangible ways. That being said, I have a hard time seeing myself moving from Python to JS for any 'real' data science work when I get a lot of the more advanced viz features here for free from libraries like plotly without having to touch JS and with similar or better performance. Not to mention, I don't have any desire to move away from PyCharm after getting comfortable with and really taking advantage of of its full feature set. I think for small projects this could be very useful, especially with GDocs style real time collaboration, but for now I can't see myself or my teammates taking advantage of this. That being said, I vote full steam ahead and lets see where this takes us!
- hckr1292 8y ago
  If I'm reading the near the end of the post correctly, they actually compiled Numpy, Pandas, SciPy, and scikit-learn to WebAssembly and have one video at the end of the post showing a python-only visualization running in the browser.
  - callahad 8y ago
    You are reading that correctly.
- FridgeSeal 8y ago
  I agree: you’d have to quite literally force me to use JS to do data science stuff. Surely there’s a better solution here? Maybe something like pandoc but for presenting science stuff in the browser? Do your work in whatever language, write up results and graphs using preferred tools and packages, output some kind of format that can be parsed to produce this stuff?
  - hamilton 8y ago
    Iodide does actually support that workflow, both in JS and our wasm python stack (Pyodide). We have some data scientists internally who have made Iodide reports using only pandas & matplotlib. It's still early days for this tech, but it's quite doable.
dlukes 8y ago
It's impressive how streamlined matplotlib plotting is and how well it works -- you just import matplotlib and start plotting, you can have multiple interactive plots alongside each other. Especially compared to Jupyter, where it's unfortunately sort of clunky (interactive plots first have to be enabled with %matplotlib notebook, and then you can only have one at a time, you have to manually "freeze" the previous one before you start a new one, or weird stuff -- overplotting -- happens). To be clear, I love Jupyter, and I think it's especially great for teaching beginners (especially because it has "real" cells, as in GUI widgets, though I do get the appeal of a flat plain text format for advanced users). But plotting has been a bit of a stumbling block, whereas the way it works in Iodide blew me away.
mrkstu 8y ago
Instead of being targeted directly at scientific markets, I'd love to see a more generalized 'smart document' version of this. The promise of HyperCard, Glue, OpenDoc and even OLE of compound, programmable documents is something I've been looking for for decades. I'd love to see someone come up with a path taking something like this or Jupyter and have it target a wide market and extend the capabilities in an accessible way. I think you'd blow half steps like Airtable out of the water.
codeisawesome 8y ago
I like that when scientists don’t like a tool they write a white paper complaining about it (the HAL paper linked in this article) =D
simula67 8y ago
It is funny how the world wide web was invented for the purpose of improving scientific publishing. It revolutionized everything but scientific publishing. Scientists still seem to be publishing their research as PDFs with citations written at the end of the page
- kybernetikos 8y ago
  Same for Java - it was invented for applets and ended up being used for everything except applets.
  - arethuza 8y ago
    I think it was actually first created for devices (specifically set top boxes) but never really found a home there and then someone wrote a browser in it (HotJava) that supported executable content and it took off from there. https://en.wikipedia.org/wiki/Java_(programming_language)#History https://en.wikipedia.org/wiki/Java_(programming_language)#Hi...
- Cyphase 8y ago
  As an aside, you may be interested in this article from The Atlantic: https://www.theatlantic.com/science/archive/2018/04/the-scientific-paper-is-obsolete/556676/ https://www.theatlantic.com/science/archive/2018/04/the-scie... It basically makes the case that things like Jupyter notebooks are the future of scientific publishing. And here's the accompanying HN discussion for convenience: https://news.ycombinator.com/item?id=16764321 https://news.ycombinator.com/item?id=16764321
- deleted 8y ago
  [deleted]
- SiempreViernes 8y ago
  I'd say open access journals are a pretty big change to scientific publishing, and that started with the preprint archives at around 1991 with arxiv. Beyond that, the web wasn't created to improve publishing, it just did as a side effect. It was started for information exchange within collaboration, and it certainly has done that. Experimental collaborations at CERN in the 80's were not too large at most around 50 authors (NA31 < 40 , NA32 < 50 ), by the next generation it was up to 350 distributed over 32 locations, and the current collaborations are at around 3000 over 180 locations. The web is a crucial component of the information systems that allow these collaborations to function at all.
j0e1 8y ago
> Pyodide: The Python science stack in the browser You have convinced me to try this now!
kbumsik 8y ago
I just read the documentation of JSMD, which is Iodide's flavor for Markdown [1]. So they made a Markdown syntax which is capable of running JS and Python, and they name it JSMD. Just nitpicking but they probably want to come up with a more suitable name other than JSMD. I'm genuinely amazed by this project though. [1]: https://iodide-project.github.io/docs/jsmd/ https://iodide-project.github.io/docs/jsmd/
- hamilton 8y ago
  (iodide dev here) Yeah, that's a good point. We called it JSMD before we even thought of starting Pyodide, and in hindsight seems a bit constricting.
  - kbumsik 8y ago
    Thanks for your comment. It is probably not too late to change the name because it's alpha yet. It may be misleading for newcomers, when JSMD tries to support even more languages through plugins. Other than that I love the design of the syntax, especially raw cell seem so great to me. I'm often supersized by the fact that both Markdown and Jupyter don't have the official ways to insert comments. Fetch cell is also a smart idea.
kbumsik 8y ago
After having a quick look, I found that they use Emscripten and it is capable of compiling it to JS (asm.js) as well as WebAssembly. So the whole Pyodide stack can be compiled to JS. I'm wondering how the performance looks like when running Pyodide stack on JS.
- mesaframe 8y ago
  ASM.js is not the JS you use. It's follows a different spec, and it's way faster than JS but slower than WASM.
  - kbumsik 8y ago
    I'm not sure how that matters. It is still JS, it can run on any JS engine even without transpiling. I just wanted to know the exact performance difference between the two different compiled outputs.
smackay 8y ago
This is the future I want: tools so anybody who has the inclination can publish interesting things for anyone to see. Imagine if web servers were just standard containers and services which anyone could upload applications to. That would take us to a decentralised internet once again. No more centralized, fossilized, web application with stacks that are impossibly complex that only the original developers could change. All the power and innovation would be put in the hands of the content creators once more and the internet would just be infrastructure. Sound familiar? I hope so. Time to program like it's 1999.
- SiempreViernes 8y ago
  <blink> here we go! </blink>
xvilka 8y ago
What is the difference with Jupyter/JupyterLab?
- eggy 8y ago
  Jupyter follows Mathematica's input and output cell format. JupyterLab uses split windows, and is closer to Iodide from what I can tell, but Iodide is running in WebAssembly, they ported Python and associated libraries to WebAssembly. I am still looking though.
eggy 8y ago
I want to see APL ported to this, specifically Dyalog APL. APL is such a great math/data language, and with Co-dfns for GPU computing it would be great. Shen [1] would also be cool on this. [1] www.shenlanguage.org
- hamilton 8y ago
  Feel free to reach out to us on Gitter https://gitter.im/iodide-project/iodide https://gitter.im/iodide-project/iodide if you're interested in doing this. If you know anything about compilation, it's pretty straightforward, depending on the language & what's happening under the hood. Here are our docs on language plugins: https://iodide-project.github.io/docs/language_plugins/ https://iodide-project.github.io/docs/language_plugins/
  - eggy 8y ago
    I know very little about compilation, but I'm going to look at the link later, and get back to you. Thanks for the links!
ktpsns 8y ago
I like how the first comment at the end of the page asks: > It’s like JupyterHub, but for people who only know javascript? That also was my feeling. And since Jupyter is language agnostic, why not just run a JavaScript kernel, such as https://github.com/n-riesco/ijavascript https://github.com/n-riesco/ijavascript . It looks somehow as if Iodide is exactly that.
- ropeladder 8y ago
  Isn't a big difference that all of this is running your code on the front end? (Or am I misunderstanding how these work?)
mindcrime 8y ago
There is a sense in which this is pretty cool... even very cool. And I'm all for anything that makes scientific / data-oriented analysis and exploration more accessible. But on the other hand, I remain unsure that it makes sense to continually try to push everything into the browser. Take: Iodide documents live in the browser, which means the computation engine is always available. Whenever you share your work, you share a live interactive report with running code. Moreover, since the computation happens in the browser alongside the presentation, there is no need to call a language backend in another process. This means that interactive documents update in real-time, opening up the possibility of seamless 3D visualizations, even with the low-latency and high frame-rate required for VR. I mean, yeah, OK, there are aspects of this that make sense. But there are always tradeoffs. Take, for example this point: "there is no need to call a language backend in another process". This also means that you're limited to the processing power available on your local machine, which is - BTW, being shared among everything running on your computer. I'm not saying that Iodide is bad, mind you. But it - like any other tool - may not be appropriate for everything. As a corollary to that, I think it might be a fun experiment to see what it would take to provide an ability to move the computationally expensive parts between the local computer and a remote $BEEFY_HOST in a seamless way.

300 points by pablobaz 8y ago

pablobaz 8y ago

"Over the next couple months, we added Numpy, Pandas, and Matplotlib" The python data stack running in web assembly!

paulgb 8y ago
And it's also available standalone for other applications! https://github.com/iodide-project/pyodide https://github.com/iodide-project/pyodide Hats off to the team, this is some really neat stuff and I'm happy to see it coming from Mozilla.

bayesian_horse 8y ago

I was trying to make coroutines work through the javascript event loop, such that you could basically "await" any javascript Promise (or wrap a Promise in an awaitable). So you'd have things like 'await sleep(1)' which would be implemented with a Promise and setTimeout. Or 'await fetch("/api/v1/list")'. Unfortunately I haven't gotten very far yet.

bijection 8y ago
I'm pretty sure you can await any javascript Promise! For example, running `await Promise.resolve(3)` in Chrome's js console prints `3`.
- alexkavon 8y ago
  Also MDN has a fantastic Promise article with examples about wrapping a function with a promise.[0] [0] https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...
- JonathonW 8y ago
  It's a little unclear, but I think he wants to integrate Python (Pyodide) coroutines into the JS event loop-- so async/await of Python code works seamlessly in the browser, and you can await Python code from within JS (and vice versa). Any JS Promise is awaitable from JS; JS async/await is (more or less) syntactic sugar on top of promises.
- bayesian_horse 8y ago
  Sorry, I meant to use "await" in Python to await Javascript Promises.
KenoFischer 8y ago
My preferred way to do this would be a compiler pass that rewrites everything to use the explicitly managed stack. I had started on writing that transform, but ran out of time. That's currently the major blocker for making the Julia/Iodide integration work well.

kasperset 8y ago

Interesting to see this tool along with Observablehq.

ZeroCool2u 8y ago

The SWE in me is very enthusiastic about this. I love the push that Mozilla is making with WASM in really practical and tangible ways. That being said, I have a hard time seeing myself moving from Python to JS for any 'real' data science work when I get a lot of the more advanced viz features here for free from libraries like plotly without having to touch JS and with similar or better performance. Not to mention, I don't have any desire to move away from PyCharm after getting comfortable with and really taking advantage of of its full feature set. I think for small projects this could be very useful, especially with GDocs style real time collaboration, but for now I can't see myself or my teammates taking advantage of this. That being said, I vote full steam ahead and lets see where this takes us!

hckr1292 8y ago
If I'm reading the near the end of the post correctly, they actually compiled Numpy, Pandas, SciPy, and scikit-learn to WebAssembly and have one video at the end of the post showing a python-only visualization running in the browser.
- callahad 8y ago
  You are reading that correctly.
FridgeSeal 8y ago
I agree: you’d have to quite literally force me to use JS to do data science stuff. Surely there’s a better solution here? Maybe something like pandoc but for presenting science stuff in the browser? Do your work in whatever language, write up results and graphs using preferred tools and packages, output some kind of format that can be parsed to produce this stuff?
- hamilton 8y ago
  Iodide does actually support that workflow, both in JS and our wasm python stack (Pyodide). We have some data scientists internally who have made Iodide reports using only pandas & matplotlib. It's still early days for this tech, but it's quite doable.

dlukes 8y ago

It's impressive how streamlined matplotlib plotting is and how well it works -- you just import matplotlib and start plotting, you can have multiple interactive plots alongside each other. Especially compared to Jupyter, where it's unfortunately sort of clunky (interactive plots first have to be enabled with %matplotlib notebook, and then you can only have one at a time, you have to manually "freeze" the previous one before you start a new one, or weird stuff -- overplotting -- happens). To be clear, I love Jupyter, and I think it's especially great for teaching beginners (especially because it has "real" cells, as in GUI widgets, though I do get the appeal of a flat plain text format for advanced users). But plotting has been a bit of a stumbling block, whereas the way it works in Iodide blew me away.

mrkstu 8y ago

Instead of being targeted directly at scientific markets, I'd love to see a more generalized 'smart document' version of this. The promise of HyperCard, Glue, OpenDoc and even OLE of compound, programmable documents is something I've been looking for for decades. I'd love to see someone come up with a path taking something like this or Jupyter and have it target a wide market and extend the capabilities in an accessible way. I think you'd blow half steps like Airtable out of the water.

codeisawesome 8y ago

I like that when scientists don’t like a tool they write a white paper complaining about it (the HAL paper linked in this article) =D

simula67 8y ago

It is funny how the world wide web was invented for the purpose of improving scientific publishing. It revolutionized everything but scientific publishing. Scientists still seem to be publishing their research as PDFs with citations written at the end of the page

kybernetikos 8y ago
Same for Java - it was invented for applets and ended up being used for everything except applets.
- arethuza 8y ago
  I think it was actually first created for devices (specifically set top boxes) but never really found a home there and then someone wrote a browser in it (HotJava) that supported executable content and it took off from there. https://en.wikipedia.org/wiki/Java_(programming_language)#History https://en.wikipedia.org/wiki/Java_(programming_language)#Hi...
Cyphase 8y ago
As an aside, you may be interested in this article from The Atlantic: https://www.theatlantic.com/science/archive/2018/04/the-scientific-paper-is-obsolete/556676/ https://www.theatlantic.com/science/archive/2018/04/the-scie... It basically makes the case that things like Jupyter notebooks are the future of scientific publishing. And here's the accompanying HN discussion for convenience: https://news.ycombinator.com/item?id=16764321 https://news.ycombinator.com/item?id=16764321
deleted 8y ago
[deleted]
SiempreViernes 8y ago
I'd say open access journals are a pretty big change to scientific publishing, and that started with the preprint archives at around 1991 with arxiv. Beyond that, the web wasn't created to improve publishing, it just did as a side effect. It was started for information exchange within collaboration, and it certainly has done that. Experimental collaborations at CERN in the 80's were not too large at most around 50 authors (NA31 < 40 , NA32 < 50 ), by the next generation it was up to 350 distributed over 32 locations, and the current collaborations are at around 3000 over 180 locations. The web is a crucial component of the information systems that allow these collaborations to function at all.

j0e1 8y ago

> Pyodide: The Python science stack in the browser You have convinced me to try this now!

kbumsik 8y ago

I just read the documentation of JSMD, which is Iodide's flavor for Markdown [1]. So they made a Markdown syntax which is capable of running JS and Python, and they name it JSMD. Just nitpicking but they probably want to come up with a more suitable name other than JSMD. I'm genuinely amazed by this project though. [1]: https://iodide-project.github.io/docs/jsmd/ https://iodide-project.github.io/docs/jsmd/

hamilton 8y ago
(iodide dev here) Yeah, that's a good point. We called it JSMD before we even thought of starting Pyodide, and in hindsight seems a bit constricting.
- kbumsik 8y ago
  Thanks for your comment. It is probably not too late to change the name because it's alpha yet. It may be misleading for newcomers, when JSMD tries to support even more languages through plugins. Other than that I love the design of the syntax, especially raw cell seem so great to me. I'm often supersized by the fact that both Markdown and Jupyter don't have the official ways to insert comments. Fetch cell is also a smart idea.

kbumsik 8y ago

After having a quick look, I found that they use Emscripten and it is capable of compiling it to JS (asm.js) as well as WebAssembly. So the whole Pyodide stack can be compiled to JS. I'm wondering how the performance looks like when running Pyodide stack on JS.

mesaframe 8y ago
ASM.js is not the JS you use. It's follows a different spec, and it's way faster than JS but slower than WASM.
- kbumsik 8y ago
  I'm not sure how that matters. It is still JS, it can run on any JS engine even without transpiling. I just wanted to know the exact performance difference between the two different compiled outputs.

smackay 8y ago

This is the future I want: tools so anybody who has the inclination can publish interesting things for anyone to see. Imagine if web servers were just standard containers and services which anyone could upload applications to. That would take us to a decentralised internet once again. No more centralized, fossilized, web application with stacks that are impossibly complex that only the original developers could change. All the power and innovation would be put in the hands of the content creators once more and the internet would just be infrastructure. Sound familiar? I hope so. Time to program like it's 1999.

SiempreViernes 8y ago
<blink> here we go! </blink>

xvilka 8y ago

What is the difference with Jupyter/JupyterLab?

eggy 8y ago
Jupyter follows Mathematica's input and output cell format. JupyterLab uses split windows, and is closer to Iodide from what I can tell, but Iodide is running in WebAssembly, they ported Python and associated libraries to WebAssembly. I am still looking though.

eggy 8y ago

I want to see APL ported to this, specifically Dyalog APL. APL is such a great math/data language, and with Co-dfns for GPU computing it would be great. Shen [1] would also be cool on this. [1] www.shenlanguage.org

hamilton 8y ago
Feel free to reach out to us on Gitter https://gitter.im/iodide-project/iodide https://gitter.im/iodide-project/iodide if you're interested in doing this. If you know anything about compilation, it's pretty straightforward, depending on the language & what's happening under the hood. Here are our docs on language plugins: https://iodide-project.github.io/docs/language_plugins/ https://iodide-project.github.io/docs/language_plugins/
- eggy 8y ago
  I know very little about compilation, but I'm going to look at the link later, and get back to you. Thanks for the links!

ktpsns 8y ago

I like how the first comment at the end of the page asks: > It’s like JupyterHub, but for people who only know javascript? That also was my feeling. And since Jupyter is language agnostic, why not just run a JavaScript kernel, such as https://github.com/n-riesco/ijavascript https://github.com/n-riesco/ijavascript . It looks somehow as if Iodide is exactly that.

ropeladder 8y ago
Isn't a big difference that all of this is running your code on the front end? (Or am I misunderstanding how these work?)

mindcrime 8y ago

There is a sense in which this is pretty cool... even very cool. And I'm all for anything that makes scientific / data-oriented analysis and exploration more accessible. But on the other hand, I remain unsure that it makes sense to continually try to push everything into the browser. Take: Iodide documents live in the browser, which means the computation engine is always available. Whenever you share your work, you share a live interactive report with running code. Moreover, since the computation happens in the browser alongside the presentation, there is no need to call a language backend in another process. This means that interactive documents update in real-time, opening up the possibility of seamless 3D visualizations, even with the low-latency and high frame-rate required for VR. I mean, yeah, OK, there are aspects of this that make sense. But there are always tradeoffs. Take, for example this point: "there is no need to call a language backend in another process". This also means that you're limited to the processing power available on your local machine, which is - BTW, being shared among everything running on your computer. I'm not saying that Iodide is bad, mind you. But it - like any other tool - may not be appropriate for everything. As a corollary to that, I think it might be a fun experiment to see what it would take to provide an ability to move the computationally expensive parts between the local computer and a remote $BEEFY_HOST in a seamless way.