5 ms·

Hands down the best and most used tool in my portfolio. Well done, guys! Does anyone know if there's a plan to introduce multi-kernel support in single noteboo

Hands down the best and most used tool in my portfolio. Well done, guys! Does anyone know if there's a plan to introduce multi-kernel support in single notebooks like what Zeppelin does? Not that I have a strong preference for it but it appears to hold a lot of appeal in Spark-like environments where not all packages are available in Pyspark and you need to move between native Scala/spark and Pyspark.

nl 10y ago
Spark-like environments where not all packages are available in Pyspark and you need to move between native Scala/spark and Pyspark. What exactly do you mean here? Are you referring to the parts of Spark which don't currently have a Python API? Because those are becoming smaller and smaller. There is also Jupyter magics to let you change languages within a notebook. See %Rpush and %RPull from [1]. Not sure if there is a way to have a Scala kernel running and sharing the same Spark context though. I think IBM is working on something in this area. [1] https://blog.dominodatalab.com/lesser-known-ways-of-using-notebooks/ https://blog.dominodatalab.com/lesser-known-ways-of-using-no...
- roystonvassey 10y ago
  Yes, that's what I exactly meant. And you're right - that's a pretty small subset and that's what I tell others too. But, the ease with which you can load interpreters on Zeppelin (apart from the pre-loaded ones) is impressive. I imagine it comes at a cost of some instability because it hangs more often than Jupyter.
carreau 10y ago
See project like metakernel [1], in the end a kernel can be "just" controlling process that manage multiple language. So why you won't get "multiple kernel" you can get "one kernel with multiple language". Its just asking for "multiple kernel" which not the best way to describe what you want to do. In the end it the same excpet you do the language dispatch in the backend instead of the frontend. It's also more efficient for data sharing. [2] show you haw Hydrogen (based on Jupyter as well), does it. And [3] (mine), show you how in the the same notebook to use Python, R, C, Rust, Fortran, Cython and Julia with data sharing and sending functions back and forth between languages. I was definitively lazy and did not include things like SQL, javascript and a few others. I haven't used spark in a while, and definitively never from scala directly, but I doubt it would be much harder to do as the C/Rust/Fortran/Cython took me an afternoon to write. [1]: https://github.com/Calysto/metakernel https://github.com/Calysto/metakernel [2]: https://www.google.com/url?hl=en&q=https://medium.com/nteract/hydrogen-introducing-rich-multi-language-documents-b5057ff34efc&source=gmail&ust=1491411006830000&usg=AFQjCNF-dWWzLXr2H6zJRuJyMpc2RqiWpQ https://www.google.com/url?hl=en&q=https://medium.com/nterac... [3]: http://carreau.github.io/posts/23-Cross-Language-Integration.html http://carreau.github.io/posts/23-Cross-Language-Integration...
dmix 10y ago
I'm curious, what do you use Notebook for on a frequent basis? Data analysis?
- roystonvassey 10y ago
  Primarily, yes. And also for building and trying out ML models.
spot 10y ago
We are actually working on improving the polyglot support in Jupyter by porting/merging work from Beaker Notebook (http://beakernotebook.com http://beakernotebook.com) into Jupyter and especially into Jupyter Lab. This new effort is called BeakerX: https://github.com/twosigma/beakerx https://github.com/twosigma/beakerx. We have autotranslation working in one direction now, should have multi-way in a couple of weeks.
cercatrova 10y ago
Like Beaker Notebooks?

nl 10y ago

Spark-like environments where not all packages are available in Pyspark and you need to move between native Scala/spark and Pyspark. What exactly do you mean here? Are you referring to the parts of Spark which don't currently have a Python API? Because those are becoming smaller and smaller. There is also Jupyter magics to let you change languages within a notebook. See %Rpush and %RPull from [1]. Not sure if there is a way to have a Scala kernel running and sharing the same Spark context though. I think IBM is working on something in this area. [1] https://blog.dominodatalab.com/lesser-known-ways-of-using-notebooks/ https://blog.dominodatalab.com/lesser-known-ways-of-using-no...

roystonvassey 10y ago
Yes, that's what I exactly meant. And you're right - that's a pretty small subset and that's what I tell others too. But, the ease with which you can load interpreters on Zeppelin (apart from the pre-loaded ones) is impressive. I imagine it comes at a cost of some instability because it hangs more often than Jupyter.

carreau 10y ago

See project like metakernel [1], in the end a kernel can be "just" controlling process that manage multiple language. So why you won't get "multiple kernel" you can get "one kernel with multiple language". Its just asking for "multiple kernel" which not the best way to describe what you want to do. In the end it the same excpet you do the language dispatch in the backend instead of the frontend. It's also more efficient for data sharing. [2] show you haw Hydrogen (based on Jupyter as well), does it. And [3] (mine), show you how in the the same notebook to use Python, R, C, Rust, Fortran, Cython and Julia with data sharing and sending functions back and forth between languages. I was definitively lazy and did not include things like SQL, javascript and a few others. I haven't used spark in a while, and definitively never from scala directly, but I doubt it would be much harder to do as the C/Rust/Fortran/Cython took me an afternoon to write. [1]: https://github.com/Calysto/metakernel https://github.com/Calysto/metakernel [2]: https://www.google.com/url?hl=en&q=https://medium.com/nteract/hydrogen-introducing-rich-multi-language-documents-b5057ff34efc&source=gmail&ust=1491411006830000&usg=AFQjCNF-dWWzLXr2H6zJRuJyMpc2RqiWpQ https://www.google.com/url?hl=en&q=https://medium.com/nterac... [3]: http://carreau.github.io/posts/23-Cross-Language-Integration.html http://carreau.github.io/posts/23-Cross-Language-Integration...

dmix 10y ago

I'm curious, what do you use Notebook for on a frequent basis? Data analysis?

roystonvassey 10y ago
Primarily, yes. And also for building and trying out ML models.

spot 10y ago

We are actually working on improving the polyglot support in Jupyter by porting/merging work from Beaker Notebook (http://beakernotebook.com http://beakernotebook.com) into Jupyter and especially into Jupyter Lab. This new effort is called BeakerX: https://github.com/twosigma/beakerx https://github.com/twosigma/beakerx. We have autotranslation working in one direction now, should have multi-way in a couple of weeks.

cercatrova 10y ago

Like Beaker Notebooks?