5 ms·

So what's the benefit to using SystemML on top of Spark (or another scheduler) vs. using Spark's own MLlib? From the links below, it seems MLlib supports a supe

by btown 11y ago

So what's the benefit to using SystemML on top of Spark (or another scheduler) vs. using Spark's own MLlib? From the links below, it seems MLlib supports a superset of SystemML's algorithms (with the small exception of survival analysis). Are there any plans for the projects to use insights from each other, or to merge in some way? Because it seems quite silly for Apache to sponsor two competing machine learning libraries with exactly the same goals. https://sparktc.github.io/systemml/algorithms-reference.html https://sparktc.github.io/systemml/algorithms-reference.html https://spark.apache.org/docs/latest/mllib-guide.html https://spark.apache.org/docs/latest/mllib-guide.html

rectang 11y ago
> Because it seems quite silly for Apache to sponsor two competing machine learning libraries with exactly the same goals. Apache is willing to provide a home for any community who abides by the Foundation's governance rules and traditions. It is acceptable to have competing projects, and there are many examples in the Foundation's history. Additionally, Apache doesn't "sponsor" projects per se -- no developers get paid through Apache. The Foundation spends what it takes to support projects, but the annual budget is tiny -- a little over a million each year, when there are ~200 projects and ~5000 committers.
dusenberrymw 11y ago
Great question. I'm part of the committer team for the project at IBM, so I'll leave a few comments representing our thoughts. As a quick overview, SystemML provides an R-like DSL, called DML, consisting of linear algebra primitives (vectors, matrices), built-in functions for common functions (such as sums, means, matrix construction, etc.), UDFs, etc., as well as a compiler/optimizer engine that can generate optimized runtime plans from the same DML script for a single node (laptop), Spark, or Hadoop MapReduce. We definitely have algorithms already available as production-ready examples, but the goal of the project is to allow for declarative ML using customizable scripts written at the mathematical DSL level, rather than to provide a fixed library of algorithms at the base language level (Scala, Python, etc.). MLlib (including the newer ML API) is awesome, and provides a great set of algorithms that fit in quite well with Scala, Python (& Java). SystemML is great in that it provides the ability to run customizable, linear algebra-based ML scripts (that can be automatically optimized within the engine) on Spark. Together, it's a great combo. We also have an API for Scala that lets one embed DML into a Scala program similar in manner to how an SQL script can be embedded [http://sparktc.github.io/systemml/mlcontext-programming-guide.html http://sparktc.github.io/systemml/mlcontext-programming-guid...]. Here are our new Apache links: https://systemml.apache.org https://systemml.apache.org https://github.com/apache/incubator-systemml https://github.com/apache/incubator-systemml
- btown 11y ago
  Very cool - so the algorithms shared with MLlib are just an example of what can be achieved with such a DSL. This could actually be very useful for inference on domain specific generative models that need custom-derived update steps. And probabilistic programming could be built on top of this. I stand very much corrected - this is definitely a project worth attention from the community separate from Spark itself!

by btown 11y ago

rectang 11y ago

> Because it seems quite silly for Apache to sponsor two competing machine learning libraries with exactly the same goals. Apache is willing to provide a home for any community who abides by the Foundation's governance rules and traditions. It is acceptable to have competing projects, and there are many examples in the Foundation's history. Additionally, Apache doesn't "sponsor" projects per se -- no developers get paid through Apache. The Foundation spends what it takes to support projects, but the annual budget is tiny -- a little over a million each year, when there are ~200 projects and ~5000 committers.

dusenberrymw 11y ago

Great question. I'm part of the committer team for the project at IBM, so I'll leave a few comments representing our thoughts. As a quick overview, SystemML provides an R-like DSL, called DML, consisting of linear algebra primitives (vectors, matrices), built-in functions for common functions (such as sums, means, matrix construction, etc.), UDFs, etc., as well as a compiler/optimizer engine that can generate optimized runtime plans from the same DML script for a single node (laptop), Spark, or Hadoop MapReduce. We definitely have algorithms already available as production-ready examples, but the goal of the project is to allow for declarative ML using customizable scripts written at the mathematical DSL level, rather than to provide a fixed library of algorithms at the base language level (Scala, Python, etc.). MLlib (including the newer ML API) is awesome, and provides a great set of algorithms that fit in quite well with Scala, Python (& Java). SystemML is great in that it provides the ability to run customizable, linear algebra-based ML scripts (that can be automatically optimized within the engine) on Spark. Together, it's a great combo. We also have an API for Scala that lets one embed DML into a Scala program similar in manner to how an SQL script can be embedded [http://sparktc.github.io/systemml/mlcontext-programming-guide.html http://sparktc.github.io/systemml/mlcontext-programming-guid...]. Here are our new Apache links: https://systemml.apache.org https://systemml.apache.org https://github.com/apache/incubator-systemml https://github.com/apache/incubator-systemml

btown 11y ago
Very cool - so the algorithms shared with MLlib are just an example of what can be achieved with such a DSL. This could actually be very useful for inference on domain specific generative models that need custom-derived update steps. And probabilistic programming could be built on top of this. I stand very much corrected - this is definitely a project worth attention from the community separate from Spark itself!