5 ms·

Interesting. I wonder how this compares to an IPython notebook? While I haven't tried it, I imagine that you can use pyspark from an IPython notebook and have

by rkrzr 11y ago

Interesting. I wonder how this compares to an IPython notebook? While I haven't tried it, I imagine that you can use pyspark from an IPython notebook and have much of the same functionality as with Apache Zeppelin. Makes me wonder if it wouldn't have been better to just extend IPython notebooks instead of starting a new project. Edit: How to run PySpark from an IPython notebook[1] [1] http://ramhiser.com/2015/02/01/configuring-ipython-notebook-support-for-pyspark/ http://ramhiser.com/2015/02/01/configuring-ipython-notebook-...

sshb 11y ago
Also IPython is moving towards splitting the notebook part(Jupyter) and python part(IPython), so you can run any language in notebook (like IHaskell). Details are in IPython changelog: > 3.x will be the last monolithic release of IPython, as the next release cycle will see the growing project split into its Python-specific and language-agnostic components. Language-agnostic projects (notebook, qtconsole, etc.) will move under the umbrella of the new Project Jupyter name https://jupyter.org/ https://jupyter.org/ http://ipython.org/ipython-doc/3/whatsnew/version3.html http://ipython.org/ipython-doc/3/whatsnew/version3.html
- codemac 11y ago
  I'm so excited for this. The power of org-babel is the common 'org table' format that can be used to interchange data between different languages. Has IPython thought about how to send data / state between steps without python?
threeseed 11y ago
Hilarious that i've tried this one but not the IPython one. It's actually quite nice and very promising but still a little raw and rough around the edges. It's just great to see an open source project to compete with the commercial ones that seem to be popping up all over the place. And it's very modular so it could be used for more than just Spark based jobs.
samkone 11y ago
Yes you can use IPython with Spark for Pyspark. Spark-notebook and Zeppelin, gives let's you use Scala with Spark,and support all of the other spark libraries. There are other projects like spark-kernel https://github.com/ibm-et/spark-kernel https://github.com/ibm-et/spark-kernel that let's you use IPython (Jupyter ) I guess, supporting spark's scala API.
leemoonsoo 11y ago
I'm one of committers of Apache Zeppelin (incubating). Zeppelin is inspired by iPython notebook and many other amazing softwares that has notebook interface. I know iPython notebook has long history and large community, i really like it. Zeppelin is young, new project compare to iPython notebook. Zeppelin and iPython notebook, they are opensource. iPython notebook is lead by IPython Development Team. Zeppelin is under Apache Software Foundation and it is being developed in Apache way, from copyrights, development process, decision making process to community development. Zeppelin is focusing on providing analytical environment on top of Hadoop eco-system. I'm not sure about iPython's direction, but i don't think it's the same to Zeppelin. I see many projects that has notebook interface. Not only iPython and Zeppelin, but also Databricks Cloud, Spark Notebook, Beaker and many others. I'm sure they all have their own advantages. Hope see all softwares are beloved by users. Cheers
- jjtheblunt 11y ago
  No one seems to mention that they're all imitating Mathematica, which is still going strong since 1988 with an amazing notebook facility.
- sbenario 11y ago
  I find it concerning when project developers don't seem to know exactly WHY the product exists or how it compares to competition on the market. If I'm spending a lot of time working on a product that has a clear competitor(s), I'd expect to know exactly why my product is going to be better than those competitors (ie. in which ways), and what their weaknesses are.
  - wmf 11y ago
    A lot of open source developers go to great lengths to avoid admitting that anything competes with anything else. I don't know if it's just conflict avoidance, rationalizing failure, or what.
    - leemoonsoo 11y ago
      I can spend hours and hours to explain why Zeppelin is good and what is different feature compare to the other. for example, Zeppelin has pluggable architecture so it does support not only scala but also python with built in Spark integration. Zeppelin not only support data exchange between scala and python environment but also SparkContext sharing for spark cluster resource utilization. It's got ability create rich interactive analytics GUI inside of notebook. It's got customizable layout system, and so on.. But I wanted to address more fundamental difference. Most competitors are opensource project. And as a opensource project, how community works and project direction are the most important things. That'll make huge difference in the end. And i think i explained and compared them.

by rkrzr 11y ago

sshb 11y ago

Also IPython is moving towards splitting the notebook part(Jupyter) and python part(IPython), so you can run any language in notebook (like IHaskell). Details are in IPython changelog: > 3.x will be the last monolithic release of IPython, as the next release cycle will see the growing project split into its Python-specific and language-agnostic components. Language-agnostic projects (notebook, qtconsole, etc.) will move under the umbrella of the new Project Jupyter name https://jupyter.org/ https://jupyter.org/ http://ipython.org/ipython-doc/3/whatsnew/version3.html http://ipython.org/ipython-doc/3/whatsnew/version3.html

codemac 11y ago
I'm so excited for this. The power of org-babel is the common 'org table' format that can be used to interchange data between different languages. Has IPython thought about how to send data / state between steps without python?

threeseed 11y ago

Hilarious that i've tried this one but not the IPython one. It's actually quite nice and very promising but still a little raw and rough around the edges. It's just great to see an open source project to compete with the commercial ones that seem to be popping up all over the place. And it's very modular so it could be used for more than just Spark based jobs.

samkone 11y ago

Yes you can use IPython with Spark for Pyspark. Spark-notebook and Zeppelin, gives let's you use Scala with Spark,and support all of the other spark libraries. There are other projects like spark-kernel https://github.com/ibm-et/spark-kernel https://github.com/ibm-et/spark-kernel that let's you use IPython (Jupyter ) I guess, supporting spark's scala API.

leemoonsoo 11y ago

I'm one of committers of Apache Zeppelin (incubating). Zeppelin is inspired by iPython notebook and many other amazing softwares that has notebook interface. I know iPython notebook has long history and large community, i really like it. Zeppelin is young, new project compare to iPython notebook. Zeppelin and iPython notebook, they are opensource. iPython notebook is lead by IPython Development Team. Zeppelin is under Apache Software Foundation and it is being developed in Apache way, from copyrights, development process, decision making process to community development. Zeppelin is focusing on providing analytical environment on top of Hadoop eco-system. I'm not sure about iPython's direction, but i don't think it's the same to Zeppelin. I see many projects that has notebook interface. Not only iPython and Zeppelin, but also Databricks Cloud, Spark Notebook, Beaker and many others. I'm sure they all have their own advantages. Hope see all softwares are beloved by users. Cheers

jjtheblunt 11y ago
No one seems to mention that they're all imitating Mathematica, which is still going strong since 1988 with an amazing notebook facility.
sbenario 11y ago
I find it concerning when project developers don't seem to know exactly WHY the product exists or how it compares to competition on the market. If I'm spending a lot of time working on a product that has a clear competitor(s), I'd expect to know exactly why my product is going to be better than those competitors (ie. in which ways), and what their weaknesses are.
- wmf 11y ago
  A lot of open source developers go to great lengths to avoid admitting that anything competes with anything else. I don't know if it's just conflict avoidance, rationalizing failure, or what.
  - leemoonsoo 11y ago
    I can spend hours and hours to explain why Zeppelin is good and what is different feature compare to the other. for example, Zeppelin has pluggable architecture so it does support not only scala but also python with built in Spark integration. Zeppelin not only support data exchange between scala and python environment but also SparkContext sharing for spark cluster resource utilization. It's got ability create rich interactive analytics GUI inside of notebook. It's got customizable layout system, and so on.. But I wanted to address more fundamental difference. Most competitors are opensource project. And as a opensource project, how community works and project direction are the most important things. That'll make huge difference in the end. And i think i explained and compared them.