4 ms·

IBM's SystemML Machine Learning – Now Apache SystemML

79 points by vasili111 11y ago

ealexhudson 11y ago
Cynic in me thinks this is just another IBM project thrown over the fence to die in the Apache graveyard. I hope that's very wrong in this case.
- chei0aiV 11y ago
  They are also late to the party. Google etc released theirs first.
  - argonaut 11y ago
    Google are also late to the party. Theano, Torch, etc released theirs first. Theano is from 2007. Torch is from 2002.
- bitmapbrother 11y ago
  I wonder why companies even bother open sourcing their software with all the critics looking for angle's to their motivation. Better to keep it private and not put up with the arm chair open source critics.
  - ealexhudson 11y ago
    Certainly, if criticism is a problem, then doing something in the open with the community is probably not going to end well.
  - chris_wot 11y ago
    So they must never be criticized?
  - ricksplat 11y ago
    The cynic in me thinks it's often a case of a product line they no longer want to support but for which an established user base already exists (Flex comes to mind). The optimist in me thinks this isn't necessarily a bad thing though, it's certainly better than just killing it altogether. The FOSS idealist in me thinks it's great to be getting all this stuff, under an Apache licence!
  - deleted 11y ago
    [deleted]
- dusenberrymw 11y ago
  I can assure you that is not the case. :) We have a very active & growing team focused on the project, and we're all quite excited about it. If you're interested in contributing, please jump in or reach out!
sandGorgon 11y ago
interesting.. so it allows R syntax. I wonder why they didnt build it on top of Renjin [1] [1] http://www.renjin.org/ http://www.renjin.org/
btown 11y ago
So what's the benefit to using SystemML on top of Spark (or another scheduler) vs. using Spark's own MLlib? From the links below, it seems MLlib supports a superset of SystemML's algorithms (with the small exception of survival analysis). Are there any plans for the projects to use insights from each other, or to merge in some way? Because it seems quite silly for Apache to sponsor two competing machine learning libraries with exactly the same goals. https://sparktc.github.io/systemml/algorithms-reference.html https://sparktc.github.io/systemml/algorithms-reference.html https://spark.apache.org/docs/latest/mllib-guide.html https://spark.apache.org/docs/latest/mllib-guide.html
- rectang 11y ago
  > Because it seems quite silly for Apache to sponsor two competing machine learning libraries with exactly the same goals. Apache is willing to provide a home for any community who abides by the Foundation's governance rules and traditions. It is acceptable to have competing projects, and there are many examples in the Foundation's history. Additionally, Apache doesn't "sponsor" projects per se -- no developers get paid through Apache. The Foundation spends what it takes to support projects, but the annual budget is tiny -- a little over a million each year, when there are ~200 projects and ~5000 committers.
- dusenberrymw 11y ago
  Great question. I'm part of the committer team for the project at IBM, so I'll leave a few comments representing our thoughts. As a quick overview, SystemML provides an R-like DSL, called DML, consisting of linear algebra primitives (vectors, matrices), built-in functions for common functions (such as sums, means, matrix construction, etc.), UDFs, etc., as well as a compiler/optimizer engine that can generate optimized runtime plans from the same DML script for a single node (laptop), Spark, or Hadoop MapReduce. We definitely have algorithms already available as production-ready examples, but the goal of the project is to allow for declarative ML using customizable scripts written at the mathematical DSL level, rather than to provide a fixed library of algorithms at the base language level (Scala, Python, etc.). MLlib (including the newer ML API) is awesome, and provides a great set of algorithms that fit in quite well with Scala, Python (& Java). SystemML is great in that it provides the ability to run customizable, linear algebra-based ML scripts (that can be automatically optimized within the engine) on Spark. Together, it's a great combo. We also have an API for Scala that lets one embed DML into a Scala program similar in manner to how an SQL script can be embedded [http://sparktc.github.io/systemml/mlcontext-programming-guide.html http://sparktc.github.io/systemml/mlcontext-programming-guid...]. Here are our new Apache links: https://systemml.apache.org https://systemml.apache.org https://github.com/apache/incubator-systemml https://github.com/apache/incubator-systemml
nfa_backward 11y ago
This looks interesting and something I will definitely watch, but at this point I think I will still stick with http://h2o.ai/ http://h2o.ai/ (another JVM based ML open source project that integrates well with 'Hadoop'). I have been really impressed with the quality of the product and even more so with the quality of the people behind the it.
chris_wot 11y ago
Ah, Apache. Where projects go to die.

79 points by vasili111 11y ago

Cynic in me thinks this is just another IBM project thrown over the fence to die in the Apache graveyard. I hope that's very wrong in this case.

chei0aiV 11y ago
They are also late to the party. Google etc released theirs first.
- argonaut 11y ago
  Google are also late to the party. Theano, Torch, etc released theirs first. Theano is from 2007. Torch is from 2002.
bitmapbrother 11y ago
I wonder why companies even bother open sourcing their software with all the critics looking for angle's to their motivation. Better to keep it private and not put up with the arm chair open source critics.
- ealexhudson 11y ago
  Certainly, if criticism is a problem, then doing something in the open with the community is probably not going to end well.
- chris_wot 11y ago
  So they must never be criticized?
- ricksplat 11y ago
  The cynic in me thinks it's often a case of a product line they no longer want to support but for which an established user base already exists (Flex comes to mind). The optimist in me thinks this isn't necessarily a bad thing though, it's certainly better than just killing it altogether. The FOSS idealist in me thinks it's great to be getting all this stuff, under an Apache licence!
- deleted 11y ago
  [deleted]
dusenberrymw 11y ago
I can assure you that is not the case. :) We have a very active & growing team focused on the project, and we're all quite excited about it. If you're interested in contributing, please jump in or reach out!

sandGorgon 11y ago

interesting.. so it allows R syntax. I wonder why they didnt build it on top of Renjin [1] [1] http://www.renjin.org/ http://www.renjin.org/

btown 11y ago

So what's the benefit to using SystemML on top of Spark (or another scheduler) vs. using Spark's own MLlib? From the links below, it seems MLlib supports a superset of SystemML's algorithms (with the small exception of survival analysis). Are there any plans for the projects to use insights from each other, or to merge in some way? Because it seems quite silly for Apache to sponsor two competing machine learning libraries with exactly the same goals. https://sparktc.github.io/systemml/algorithms-reference.html https://sparktc.github.io/systemml/algorithms-reference.html https://spark.apache.org/docs/latest/mllib-guide.html https://spark.apache.org/docs/latest/mllib-guide.html

rectang 11y ago
> Because it seems quite silly for Apache to sponsor two competing machine learning libraries with exactly the same goals. Apache is willing to provide a home for any community who abides by the Foundation's governance rules and traditions. It is acceptable to have competing projects, and there are many examples in the Foundation's history. Additionally, Apache doesn't "sponsor" projects per se -- no developers get paid through Apache. The Foundation spends what it takes to support projects, but the annual budget is tiny -- a little over a million each year, when there are ~200 projects and ~5000 committers.
dusenberrymw 11y ago
Great question. I'm part of the committer team for the project at IBM, so I'll leave a few comments representing our thoughts. As a quick overview, SystemML provides an R-like DSL, called DML, consisting of linear algebra primitives (vectors, matrices), built-in functions for common functions (such as sums, means, matrix construction, etc.), UDFs, etc., as well as a compiler/optimizer engine that can generate optimized runtime plans from the same DML script for a single node (laptop), Spark, or Hadoop MapReduce. We definitely have algorithms already available as production-ready examples, but the goal of the project is to allow for declarative ML using customizable scripts written at the mathematical DSL level, rather than to provide a fixed library of algorithms at the base language level (Scala, Python, etc.). MLlib (including the newer ML API) is awesome, and provides a great set of algorithms that fit in quite well with Scala, Python (& Java). SystemML is great in that it provides the ability to run customizable, linear algebra-based ML scripts (that can be automatically optimized within the engine) on Spark. Together, it's a great combo. We also have an API for Scala that lets one embed DML into a Scala program similar in manner to how an SQL script can be embedded [http://sparktc.github.io/systemml/mlcontext-programming-guide.html http://sparktc.github.io/systemml/mlcontext-programming-guid...]. Here are our new Apache links: https://systemml.apache.org https://systemml.apache.org https://github.com/apache/incubator-systemml https://github.com/apache/incubator-systemml

nfa_backward 11y ago

This looks interesting and something I will definitely watch, but at this point I think I will still stick with http://h2o.ai/ http://h2o.ai/ (another JVM based ML open source project that integrates well with 'Hadoop'). I have been really impressed with the quality of the product and even more so with the quality of the people behind the it.

chris_wot 11y ago

Ah, Apache. Where projects go to die.