I'm slightly concerned about the Hadoop/Spark + machine learning ecosystem right now. There seem to be a lot of technologies and projects being built up on the

I'm slightly concerned about the Hadoop/Spark + machine learning ecosystem right now. There seem to be a lot of technologies and projects being built up on the core foundation of in memory distributed computation. MLlib in Spark, H2O, Apache Mahout/Samsara, probably many others. My impression is Samsara and SystemML's DML are being designed to supply the primitives you need to build a machine learning model with less awareness of the underlying system model. So this is ostensibly to save the developer of a new algorithm the headache of thinking how their algorithm needs to relate to the distributed computation model like one should consider for developing with H2O or Spark. However, It seems like many common off the shelf algos are already available and implemented indicating that perhaps working with Spark or H2O backends directly is not that hard to produce performant ML codes. As a computational scientist learning a DSL to accelerate development of an algorithm of mine into production via Spark seems like a potential distraction from just getting better with Spark.