6 ms·

Programming Languages for Machine Learning

67 points by ahalan 15y ago

djacobs 15y ago
We're using Matlab/Octave in the Stanford ML course, and it's certainly elegant but not necessarily comfortable for functional programmers. I'm reimplementing our assignments with Clojure/Incanter and am finding my code just as performant and even cleaner. In my opinion, the ML domain maps to Lisp code almost as well as AI as a whole.
- ericlavigne 15y ago
  The instructor emphasized the performance benefit of Octave's optimized matrix operations. Did you use Colt's matrix operations, or were you able to match the performance of vectorized Octave code with non-vectorized Clojure code?
  - rincewind 15y ago
    I also implemented some of the exercises in clojure, and my naive implementation without colt or incanter ran orders of magnitude slower than my octave code. Then again i did not code it with performance in mind. I overloaded algo.generic for matrix operations, so the fixnum math was probably not inlined.
  - aheilbut 15y ago
    I think the real magic comes from understanding the algorithms as matrix operations, and then implementing them by more or less just writing down the algebra.
    - billswift 15y ago
      Can you give a source for learning about "algorithms as matrix operations"? I tried searching on that phrase in Google Scholar but didn't get anything useful.
      - oscilloscope 15y ago
        If you can express the code using the functions and operators of linear algebra, you can simply write the math and get fast, elegant code. May even run on a GPU. Fortran is frequently used for physics simulation. We used Mathematica, Maple and Matlab even in non-computing classes. See http://www.netlib.org/lapack/ http://www.netlib.org/lapack/ Also http://en.wikipedia.org/wiki/Array_programming http://en.wikipedia.org/wiki/Array_programming
        billswift 15y ago
        Your first phrase captures what I assumed he meant, I was wondering if there is anything available that could help me recognize algorithms that could or should be expressed that way, and how to do it. Your Wikipedia link was helpful. I am working on a calculus refresher right now (I really need it) and am planning on working through an elementary linear algebra text once that is done, which is why I particularly noticed the comment. I have found that having an idea of applications helps retention, which is one reason I am trying to track down something more concrete.
        aheilbut 15y ago
        Most of the textbooks on machine learning present things in that way (it's really the only way). For example, check out Elements of Statistical Learning (http://www-stat.stanford.edu/~tibs/ElemStatLearn/ http://www-stat.stanford.edu/~tibs/ElemStatLearn/).
      - jaylevitt 15y ago
        If, like me, you thought "linear algebra" meant "stuff without exponents", you should start here: http://www-math.mit.edu/~gs/papers/starting2matrices.pdf http://www-math.mit.edu/~gs/papers/starting2matrices.pdf It's the most accessible starter I've found to date. There is also a linear algebra group-learning thread somewhere on HN.
  - djacobs 15y ago
    I used Colt. I'd be interested to see if Clojure parallelization and under-the-hood optimizations could match vectorization performance. Note, I haven't run official benchmarks yet. My comparisons have been more like "back propagation and gradient checking took about the same length of time in both".
  - spariev 15y ago
    I believe Incanter uses Colt/Parallel Colt under the hood
- tikhonj 15y ago
  Choosing a language for a class has different constraints than choosing one for research or production systems. I imagine performance is less important while ease of learning the language is probably worth much more (probably means not using Haskell for a class :(). I think the ML class at my university--it's in "beta" right now--is using Python following similar logic.
  - tensor 15y ago
    The place where performance really matters is generally in the core learning algorithms. Since these are all written in C or Java, you can very easily get away with programming the rest of the code in a higher level language. This is why Python is a feasible choice for machine learning at all. That said, Clojure is certainly a lot faster than Python for native code and the performance gap with Java is continuing to drop. It's also worth pointing out that you should not ignore C or C++ libraries just because your are running on the JVM. It's not very hard to interface to a core C library with JNI, although be warned, there is a slight trick to doing this via clojure rather than Java.
- option1138 15y ago
  I'm in the course as well and don't really understand what the fuss about Octave is. The help system is abysmal and third-party library support seems far behind other popular platforms. It's also quite unstable on Windows. I've been playing around with R for the past two weeks and have been more or less happy (with the exception of the memory and speed limitations in the GNU implementation). Nevertheless, this will be a really exciting field for the next decade or so. Amazing possibilities right now!
- nosignal 15y ago
  I'm doing the course as well and have been wanting to become more familiar with Clojure. I'd be very interested to see your implementations of the algorithms in a functional style. After the course is finished, I hope you consider putting up a blog or repo with the solutions.
  - djacobs 15y ago
    I'll be happy to do that. If you mail me your contact info, I'll ping you when the class is over.
cbo 15y ago
This article doesn't really describe what the title implies. It states issues and concerns for several languages, names several that have large ML libraries written for them already, and even gives his own approach for how to implement new ML algorithms, but it doesn't actually give real options for "Programming Languages for Machine Learning". I've found that Matlab/Octave is a decent substitute for a "high-level language" to sketch out new approaches with. They're significantly fast, as well as significantly suited to matrix algebra that they can give decent results, even though they have some less-than-beautiful code. Matlab appears to be the language of choice for AI at the University of Toronto. Personally, I think the best option would be to roll with a functional language (or at least, a language with functions as first-class objects), since a lot of ML algorithms can be reduced to recursion on several matrices, often using very similar functions. For example, ANNs frequently have very similar structures and training strategies, but simply use different learning functions. Everything can be done in C/C++ though, and while it'd be harder, ML is an area where the gain in speed and efficiency is so significant that extra development time pays boatloads in terms of ROI. Even basic ML examples often involve dealing with 300x236 dimensional data, so you can imagine how much that data would scale up significantly in production environments.
- wladimir 15y ago
  Everything can be done in C/C++ though, and while it'd be harder, ML is an area where the gain in speed Don't forget machine learning generally involves a lot of experimentation, and this is easier with higher-level languages. Hand-optmization always makes assumptions about specific data structures and details which can be hard to change later. You can get very close with code-generating high-level languages though. See Theano (http://deeplearning.net/software/theano/ http://deeplearning.net/software/theano/), for example. It uses a functional approach to compose the formulas for (c-)ANNs, and has advanced features such as automatic differentiation. Scalable generated GPU code can beat the pants of even the fastest hand-written C loops. Sure, hand-optimized GPU code can perform even faster but in my experience that is usually not worth the trouble.
the_cat_kittles 15y ago
scikits.learn, pyMC, scipy, numpy, milk and more in python make a pretty formidable toolkit, R is also (probably more powerful, albeit awkward) excellent! Who needs to read a huge article...
cageface 15y ago
Scala doesn't have great native libraries for this but otherwise I've found it a very good language for basic ML stuff. You can express the algorithms very succinctly and readably but still get good performance.
- mattmiller 15y ago
  Check out Mahout: http://mahout.apache.org/ http://mahout.apache.org/
  - cageface 15y ago
    Of course the great thing about Scala is that you can pretty transparently use all the Java stuff out there. Thanks for reminding me about Mahout.
- tutysara 15y ago
  There is a project which brings a matlab like environment to Scala. We can import many java numeric libraries as toolkits - check this http://code.google.com/p/scalalab/ http://code.google.com/p/scalalab/
- mindcrime 15y ago
  No, but there is a very rich assortment of ML libraries available in Java. That alone makes Scala, Clojure, Java, Groovy, etc. reasonable choices for some ML tasks. Mahout is particularly interesting since most of the Mahout algorithms are available in an implementation that's designed to run on Hadoop.
jey 15y ago
This is a five year old post. These days Python is probably the way to go with NumPy, SciPy, Theano, IPython, matplotlib, Cython, etc. I'm also really looking forward to PyPy's NumPy support -- should allow you to write in pure Python code but still get most of the optimization benefits of writing in a statically typed compiled language like C++.
- glimcat 15y ago
  Python and Haskell for development. Weka and R for saving loads of time. Some competency in Java and C helps.
  - sunkencity 15y ago
    Waffles for speed http://waffles.sourceforge.net/ http://waffles.sourceforge.net/ command line utilities and c++ lib. Beats the .... out of R on larger datasets.
  - danieldk 15y ago
    Did you ever do number crunching in Haskell [1]? Experiments by the developers of the repa matrix library show that repa (which exploits data parallelism) does not even outperform a single threaded C++ implementation when 8 CPU cores are used: http://www.cse.unsw.edu.au/~benl/papers/stencil/stencil-haskell2011-sub.pdf http://www.cse.unsw.edu.au/~benl/papers/stencil/stencil-hask... The most glaring problem is that GHC currently does not support SIMD vector instructions. A problem which is now, fortunately, being addressed: http://hackage.haskell.org/trac/ghc/wiki/SIMD http://hackage.haskell.org/trac/ghc/wiki/SIMD But currently, Haskell is not really an option if you want to write serious ML software that rely on vector or matrix operations. [1] I did, for maximum entropy modeling. Believe me, it's not near competitive to C or C++ yet. But one day, it will probably be.
    - glimcat 15y ago
      "C is faster" is a tired argument. The ability of this compiler or that to produce the most efficient possible code is almost never relevant outside of a few niche cases. Even UHF finance is such that the fractional return in computation time that you may get from using C isn't worth the development overhead. There are any number of great reasons to use C, but speed is rather outdated. In this case, you're also guilty of putting optimization before the algorithm. Squeezing out the last clock cycle doesn't matter when you have raw data in one hand and some high-level objectives in the other. Effective machine learning is more about finding ways to obtain and reduce data into a useful form, and finding ways to map between your design goals and workable algorithms. Speed is the last step, and often the only one which can be improved by throwing money at it. Besides which, if you want to have a pissing contest about speed, C is still going to lose to FPGAs. If it's that important, hire someone who knows Verilog.
sramsay 15y ago
Considering what happened to Lisp when machine learning was called AI, I'm not sure I'd want my language to win this contest. ;)
- brianobush 15y ago
  but machine learning isn't promising the world. So far ML has a good track record thus I believe we are safe.
srean 15y ago
Given that there are many lisp aficionados here at HN, some may find lush http://lush.sourceforge.net/ http://lush.sourceforge.net/ quite interesting. Lush has been discussed at HN before but here is a short summary: You get lisp syntax, optimized multi-dimensional arrays, extremely easy integration with C code, common numerical optimization libraries, and the option of translating (and then compiling) lispy code into a C dynamic library. The language features that the interpreter supports is subtly (and sometimes not so subtly) different from the compiled version though they share the same syntax. In the context of scaling machine learning code you often hear that one should/could write most of it in matlab/octave and the critical parts in C. But anyone who has done it would know it is such a butt-hurting nuisance. In comparison the C integration is a pleasure in Lush.
purplebear 15y ago
I highly recommend to consider probabilistic languages, which constitute a natural framework for probabilistic reasoning and modelling. For example, have a look at this: http://research.microsoft.com/en-us/um/cambridge/projects/infernet/ http://research.microsoft.com/en-us/um/cambridge/projects/in... or at this: http://projects.csail.mit.edu/church/wiki/Church http://projects.csail.mit.edu/church/wiki/Church Have a great day!
tutysara 15y ago
Scala could be a better choice since it is compiled and not interpreted. It would be very beneficial to run huge programs which runs for large duration as apposed to running programs which runs for smaller duration often since the java run time does a lot of optimiztion when it does the JIT. There is a even a project which brings a matlab like environment to Scala where we can import many java numeric libraries as toolkits - check this http://code.google.com/p/scalalab/ http://code.google.com/p/scalalab/

67 points by ahalan 15y ago

We're using Matlab/Octave in the Stanford ML course, and it's certainly elegant but not necessarily comfortable for functional programmers. I'm reimplementing our assignments with Clojure/Incanter and am finding my code just as performant and even cleaner. In my opinion, the ML domain maps to Lisp code almost as well as AI as a whole.

ericlavigne 15y ago
The instructor emphasized the performance benefit of Octave's optimized matrix operations. Did you use Colt's matrix operations, or were you able to match the performance of vectorized Octave code with non-vectorized Clojure code?
- rincewind 15y ago
  I also implemented some of the exercises in clojure, and my naive implementation without colt or incanter ran orders of magnitude slower than my octave code. Then again i did not code it with performance in mind. I overloaded algo.generic for matrix operations, so the fixnum math was probably not inlined.
- aheilbut 15y ago
  I think the real magic comes from understanding the algorithms as matrix operations, and then implementing them by more or less just writing down the algebra.
  - billswift 15y ago
    Can you give a source for learning about "algorithms as matrix operations"? I tried searching on that phrase in Google Scholar but didn't get anything useful.
    - oscilloscope 15y ago
      If you can express the code using the functions and operators of linear algebra, you can simply write the math and get fast, elegant code. May even run on a GPU. Fortran is frequently used for physics simulation. We used Mathematica, Maple and Matlab even in non-computing classes. See http://www.netlib.org/lapack/ http://www.netlib.org/lapack/ Also http://en.wikipedia.org/wiki/Array_programming http://en.wikipedia.org/wiki/Array_programming
      - billswift 15y ago
        Your first phrase captures what I assumed he meant, I was wondering if there is anything available that could help me recognize algorithms that could or should be expressed that way, and how to do it. Your Wikipedia link was helpful. I am working on a calculus refresher right now (I really need it) and am planning on working through an elementary linear algebra text once that is done, which is why I particularly noticed the comment. I have found that having an idea of applications helps retention, which is one reason I am trying to track down something more concrete.
        aheilbut 15y ago
        Most of the textbooks on machine learning present things in that way (it's really the only way). For example, check out Elements of Statistical Learning (http://www-stat.stanford.edu/~tibs/ElemStatLearn/ http://www-stat.stanford.edu/~tibs/ElemStatLearn/).
    - jaylevitt 15y ago
      If, like me, you thought "linear algebra" meant "stuff without exponents", you should start here: http://www-math.mit.edu/~gs/papers/starting2matrices.pdf http://www-math.mit.edu/~gs/papers/starting2matrices.pdf It's the most accessible starter I've found to date. There is also a linear algebra group-learning thread somewhere on HN.
- djacobs 15y ago
  I used Colt. I'd be interested to see if Clojure parallelization and under-the-hood optimizations could match vectorization performance. Note, I haven't run official benchmarks yet. My comparisons have been more like "back propagation and gradient checking took about the same length of time in both".
- spariev 15y ago
  I believe Incanter uses Colt/Parallel Colt under the hood
tikhonj 15y ago
Choosing a language for a class has different constraints than choosing one for research or production systems. I imagine performance is less important while ease of learning the language is probably worth much more (probably means not using Haskell for a class :(). I think the ML class at my university--it's in "beta" right now--is using Python following similar logic.
- tensor 15y ago
  The place where performance really matters is generally in the core learning algorithms. Since these are all written in C or Java, you can very easily get away with programming the rest of the code in a higher level language. This is why Python is a feasible choice for machine learning at all. That said, Clojure is certainly a lot faster than Python for native code and the performance gap with Java is continuing to drop. It's also worth pointing out that you should not ignore C or C++ libraries just because your are running on the JVM. It's not very hard to interface to a core C library with JNI, although be warned, there is a slight trick to doing this via clojure rather than Java.
option1138 15y ago
I'm in the course as well and don't really understand what the fuss about Octave is. The help system is abysmal and third-party library support seems far behind other popular platforms. It's also quite unstable on Windows. I've been playing around with R for the past two weeks and have been more or less happy (with the exception of the memory and speed limitations in the GNU implementation). Nevertheless, this will be a really exciting field for the next decade or so. Amazing possibilities right now!
nosignal 15y ago
I'm doing the course as well and have been wanting to become more familiar with Clojure. I'd be very interested to see your implementations of the algorithms in a functional style. After the course is finished, I hope you consider putting up a blog or repo with the solutions.
- djacobs 15y ago
  I'll be happy to do that. If you mail me your contact info, I'll ping you when the class is over.

cbo 15y ago

This article doesn't really describe what the title implies. It states issues and concerns for several languages, names several that have large ML libraries written for them already, and even gives his own approach for how to implement new ML algorithms, but it doesn't actually give real options for "Programming Languages for Machine Learning". I've found that Matlab/Octave is a decent substitute for a "high-level language" to sketch out new approaches with. They're significantly fast, as well as significantly suited to matrix algebra that they can give decent results, even though they have some less-than-beautiful code. Matlab appears to be the language of choice for AI at the University of Toronto. Personally, I think the best option would be to roll with a functional language (or at least, a language with functions as first-class objects), since a lot of ML algorithms can be reduced to recursion on several matrices, often using very similar functions. For example, ANNs frequently have very similar structures and training strategies, but simply use different learning functions. Everything can be done in C/C++ though, and while it'd be harder, ML is an area where the gain in speed and efficiency is so significant that extra development time pays boatloads in terms of ROI. Even basic ML examples often involve dealing with 300x236 dimensional data, so you can imagine how much that data would scale up significantly in production environments.

wladimir 15y ago
Everything can be done in C/C++ though, and while it'd be harder, ML is an area where the gain in speed Don't forget machine learning generally involves a lot of experimentation, and this is easier with higher-level languages. Hand-optmization always makes assumptions about specific data structures and details which can be hard to change later. You can get very close with code-generating high-level languages though. See Theano (http://deeplearning.net/software/theano/ http://deeplearning.net/software/theano/), for example. It uses a functional approach to compose the formulas for (c-)ANNs, and has advanced features such as automatic differentiation. Scalable generated GPU code can beat the pants of even the fastest hand-written C loops. Sure, hand-optimized GPU code can perform even faster but in my experience that is usually not worth the trouble.

the_cat_kittles 15y ago

scikits.learn, pyMC, scipy, numpy, milk and more in python make a pretty formidable toolkit, R is also (probably more powerful, albeit awkward) excellent! Who needs to read a huge article...

cageface 15y ago

Scala doesn't have great native libraries for this but otherwise I've found it a very good language for basic ML stuff. You can express the algorithms very succinctly and readably but still get good performance.

mattmiller 15y ago
Check out Mahout: http://mahout.apache.org/ http://mahout.apache.org/
- cageface 15y ago
  Of course the great thing about Scala is that you can pretty transparently use all the Java stuff out there. Thanks for reminding me about Mahout.
tutysara 15y ago
There is a project which brings a matlab like environment to Scala. We can import many java numeric libraries as toolkits - check this http://code.google.com/p/scalalab/ http://code.google.com/p/scalalab/
mindcrime 15y ago
No, but there is a very rich assortment of ML libraries available in Java. That alone makes Scala, Clojure, Java, Groovy, etc. reasonable choices for some ML tasks. Mahout is particularly interesting since most of the Mahout algorithms are available in an implementation that's designed to run on Hadoop.

jey 15y ago

This is a five year old post. These days Python is probably the way to go with NumPy, SciPy, Theano, IPython, matplotlib, Cython, etc. I'm also really looking forward to PyPy's NumPy support -- should allow you to write in pure Python code but still get most of the optimization benefits of writing in a statically typed compiled language like C++.

glimcat 15y ago
Python and Haskell for development. Weka and R for saving loads of time. Some competency in Java and C helps.
- sunkencity 15y ago
  Waffles for speed http://waffles.sourceforge.net/ http://waffles.sourceforge.net/ command line utilities and c++ lib. Beats the .... out of R on larger datasets.
- danieldk 15y ago
  Did you ever do number crunching in Haskell [1]? Experiments by the developers of the repa matrix library show that repa (which exploits data parallelism) does not even outperform a single threaded C++ implementation when 8 CPU cores are used: http://www.cse.unsw.edu.au/~benl/papers/stencil/stencil-haskell2011-sub.pdf http://www.cse.unsw.edu.au/~benl/papers/stencil/stencil-hask... The most glaring problem is that GHC currently does not support SIMD vector instructions. A problem which is now, fortunately, being addressed: http://hackage.haskell.org/trac/ghc/wiki/SIMD http://hackage.haskell.org/trac/ghc/wiki/SIMD But currently, Haskell is not really an option if you want to write serious ML software that rely on vector or matrix operations. [1] I did, for maximum entropy modeling. Believe me, it's not near competitive to C or C++ yet. But one day, it will probably be.
  - glimcat 15y ago
    "C is faster" is a tired argument. The ability of this compiler or that to produce the most efficient possible code is almost never relevant outside of a few niche cases. Even UHF finance is such that the fractional return in computation time that you may get from using C isn't worth the development overhead. There are any number of great reasons to use C, but speed is rather outdated. In this case, you're also guilty of putting optimization before the algorithm. Squeezing out the last clock cycle doesn't matter when you have raw data in one hand and some high-level objectives in the other. Effective machine learning is more about finding ways to obtain and reduce data into a useful form, and finding ways to map between your design goals and workable algorithms. Speed is the last step, and often the only one which can be improved by throwing money at it. Besides which, if you want to have a pissing contest about speed, C is still going to lose to FPGAs. If it's that important, hire someone who knows Verilog.

sramsay 15y ago

Considering what happened to Lisp when machine learning was called AI, I'm not sure I'd want my language to win this contest. ;)

brianobush 15y ago
but machine learning isn't promising the world. So far ML has a good track record thus I believe we are safe.

srean 15y ago

Given that there are many lisp aficionados here at HN, some may find lush http://lush.sourceforge.net/ http://lush.sourceforge.net/ quite interesting. Lush has been discussed at HN before but here is a short summary: You get lisp syntax, optimized multi-dimensional arrays, extremely easy integration with C code, common numerical optimization libraries, and the option of translating (and then compiling) lispy code into a C dynamic library. The language features that the interpreter supports is subtly (and sometimes not so subtly) different from the compiled version though they share the same syntax. In the context of scaling machine learning code you often hear that one should/could write most of it in matlab/octave and the critical parts in C. But anyone who has done it would know it is such a butt-hurting nuisance. In comparison the C integration is a pleasure in Lush.

purplebear 15y ago

I highly recommend to consider probabilistic languages, which constitute a natural framework for probabilistic reasoning and modelling. For example, have a look at this: http://research.microsoft.com/en-us/um/cambridge/projects/infernet/ http://research.microsoft.com/en-us/um/cambridge/projects/in... or at this: http://projects.csail.mit.edu/church/wiki/Church http://projects.csail.mit.edu/church/wiki/Church Have a great day!

tutysara 15y ago

Scala could be a better choice since it is compiled and not interpreted. It would be very beneficial to run huge programs which runs for large duration as apposed to running programs which runs for smaller duration often since the java run time does a lot of optimiztion when it does the JIT. There is a even a project which brings a matlab like environment to Scala where we can import many java numeric libraries as toolkits - check this http://code.google.com/p/scalalab/ http://code.google.com/p/scalalab/