3 ms·
I am a Spark data engineer and spend a lot of time in Scala / Python IDEs & browser notebooks. Databricks lets you package code as JAR / wheel files & attach t
by MrPowers 8y ago
I am a Spark data engineer and spend a lot of time in Scala / Python IDEs & browser notebooks. Databricks lets you package code as JAR / wheel files & attach the binaries to the cluster. I write all the complicated code in tested projects that are checked into GitHub & use the notebooks to invoke the functions and visualize results.
Folks that try to do all programming in notebooks typically drown in complexity and suffer.
- agibsonccc 8y agoYeah I agree. We do something similar if we're using zeppelin or beaker. I organize it, put an uber jar in there and then run everything from there. That's a ton easier.