3 ms·
Spark-like environments where not all packages are available in Pyspark and you need to move between native Scala/spark and Pyspark. What exactly do you mean h
by nl 10y ago
Spark-like environments where not all packages are available in Pyspark and you need to move between native Scala/spark and Pyspark.
What exactly do you mean here? Are you referring to the parts of Spark which don't currently have a Python API? Because those are becoming smaller and smaller.
There is also Jupyter magics to let you change languages within a notebook. See %Rpush and %RPull from [1]. Not sure if there is a way to have a Scala kernel running and sharing the same Spark context though.
I think IBM is working on something in this area.
[1] https://blog.dominodatalab.com/lesser-known-ways-of-using-notebooks/ https://blog.dominodatalab.com/lesser-known-ways-of-using-no...
- roystonvassey 10y agoYes, that's what I exactly meant. And you're right - that's a pretty small subset and that's what I tell others too. But, the ease with which you can load interpreters on Zeppelin (apart from the pre-loaded ones) is impressive. I imagine it comes at a cost of some instability because it hangs more often than Jupyter.