3 ms·
That's one of the things I really like about Spark (though it's probably a lot less general than what you're after - running a pipeline of steps that are Scala
by lmm 4y ago
That's one of the things I really like about Spark (though it's probably a lot less general than what you're after - running a pipeline of steps that are Scala code rather than arbitrary commands). I can run, test, and debug my pipeline locally in my IDE where it's just plain old Scala. Then I can change one line and run my same pipeline on the company Hadoop cluster.