MLlib [1] is definitely worth checking out as a Spark library for common ML algorithms - it's crazy how little code is needed to implement efficient, distribute

MLlib [1] is definitely worth checking out as a Spark library for common ML algorithms - it's crazy how little code is needed to implement efficient, distributed inference algorithms. Spark's primitives (especially the RDD abstraction) make it feel like a DSL for distributed ML algorithms - for example, check out the implementation of distributed alternating least squares matrix factorization at [2] - ~100 lines of Scala code once stripped of the Java interoperability boilerplate. On the weekend, I took a crack at implementing some distributed ML algorithms using Alternating Direction of Multipliers (ADMM) in Spark, following [3]. In a few hours and ~400 lines of Scala code, it was possible to implement distributed versions of L^1 regularized logistic regression, ridge regression, and SVMs [4]. It's a very impressive framework, and very much empowering. [1]: http://spark.incubator.apache.org/docs/latest/mllib-guide.html http://spark.incubator.apache.org/docs/latest/mllib-guide.ht... [2]: https://github.com/apache/incubator-spark/blob/fdaabdc67387524ffb84354f87985f48bd31cf60/mllib/src/main/scala/org/apache/spark/mllib/recommendation/ALS.scala https://github.com/apache/incubator-spark/blob/fdaabdc673875... [3]: http://www.stanford.edu/~boyd/papers/pdf/admm_distr_stats.pdf http://www.stanford.edu/~boyd/papers/pdf/admm_distr_stats.pd... [4]: https://github.com/ajtulloch/admmlrspark/tree/master/src/main/scala https://github.com/ajtulloch/admmlrspark/tree/master/src/mai...