4 ms·

Python and Haskell for development. Weka and R for saving loads of time. Some competency in Java and C helps.

by glimcat 15y ago

sunkencity 15y ago
Waffles for speed http://waffles.sourceforge.net/ http://waffles.sourceforge.net/ command line utilities and c++ lib. Beats the .... out of R on larger datasets.
danieldk 15y ago
Did you ever do number crunching in Haskell [1]? Experiments by the developers of the repa matrix library show that repa (which exploits data parallelism) does not even outperform a single threaded C++ implementation when 8 CPU cores are used: http://www.cse.unsw.edu.au/~benl/papers/stencil/stencil-haskell2011-sub.pdf http://www.cse.unsw.edu.au/~benl/papers/stencil/stencil-hask... The most glaring problem is that GHC currently does not support SIMD vector instructions. A problem which is now, fortunately, being addressed: http://hackage.haskell.org/trac/ghc/wiki/SIMD http://hackage.haskell.org/trac/ghc/wiki/SIMD But currently, Haskell is not really an option if you want to write serious ML software that rely on vector or matrix operations. [1] I did, for maximum entropy modeling. Believe me, it's not near competitive to C or C++ yet. But one day, it will probably be.
- glimcat 15y ago
  "C is faster" is a tired argument. The ability of this compiler or that to produce the most efficient possible code is almost never relevant outside of a few niche cases. Even UHF finance is such that the fractional return in computation time that you may get from using C isn't worth the development overhead. There are any number of great reasons to use C, but speed is rather outdated. In this case, you're also guilty of putting optimization before the algorithm. Squeezing out the last clock cycle doesn't matter when you have raw data in one hand and some high-level objectives in the other. Effective machine learning is more about finding ways to obtain and reduce data into a useful form, and finding ways to map between your design goals and workable algorithms. Speed is the last step, and often the only one which can be improved by throwing money at it. Besides which, if you want to have a pissing contest about speed, C is still going to lose to FPGAs. If it's that important, hire someone who knows Verilog.
  - kristoffer 15y ago
    Well, FPGAs are not very cost effective for ML stuff I'd say. You're better of with a GPGPU then since it's mostly matrix operations anyway. Still, depending on the problem C could still be the best language, no need to piss on that ...
  - Jach 15y ago
    Ssh, don't tell him about Nios et al. cores that let you write C again, or MyHDL to write Python that compiles to Verilog.
  - danieldk 15y ago
    "C is faster" is a tired argument. The ability of this compiler or that to produce the most efficient possible code is almost never relevant outside of a few niche cases. What niches? ML is exactly one of the niches where highly optimized code helps tremendously. We use ML, amongst other things, for parse disambiguation and fluency ranking. Making a modification in, say the grammar, often results in training and retraining various components that have a mutual influence (e.g. parse disambiguation, auxiliary distributions for subcategorization frames, the part of speech tagger). Evaluation is also performed using ten-fold cross validation. Since a modification can also result in regressions, the effect of modifications are usually checked individually. Being able to validate changes in one hour rather than one day, helps the development of such a system tremendously. We developed some software first in other languages (Java, Scala, Python), usually to make a prototype for getting grip on a problem. But reimplementing it in C or C++ paid of tremendously on each occasion. Even though there is enough CPU time available (we have a 3280 core cluster). Another thing to note is that most machine learning techniques are fairly generic. So, usually there is already a perfectly fine C/C++ implementation available. Sometimes with bindings and all. In this case, you're also guilty of putting optimization before the algorithm. How do you know? Most data is actually preprocessed using Prolog or Perl. But the actual machine software is written in C or C++. Speed is the last step, and often the only one which can be improved by throwing money at it. Again, there are niches where it is worth it. And machine learning is often that niche, because people use huge data sets, complex problems, or both. Besides which, if you want to have a pissing contest about speed, C is still going to lose to FPGAs. Yes, but FPGAs are not readily available to our users :). But we are very much interested in GPU computing. --- Apart from all of this, it's funny that you pointing to the price of implementing in ML software in C. It's not as if there are large amounts of Haskell programmers available. Writing such software in Haskell has about the same economic risk as using COBOL ;).

Python and Haskell for development. Weka and R for saving loads of time. Some competency in Java and C helps.

by glimcat 15y ago

Python and Haskell for development. Weka and R for saving loads of time. Some competency in Java and C helps.

Waffles for speed http://waffles.sourceforge.net/ http://waffles.sourceforge.net/ command line utilities and c++ lib. Beats the .... out of R on larger datasets.

danieldk 15y ago

Did you ever do number crunching in Haskell [1]? Experiments by the developers of the repa matrix library show that repa (which exploits data parallelism) does not even outperform a single threaded C++ implementation when 8 CPU cores are used: http://www.cse.unsw.edu.au/~benl/papers/stencil/stencil-haskell2011-sub.pdf http://www.cse.unsw.edu.au/~benl/papers/stencil/stencil-hask... The most glaring problem is that GHC currently does not support SIMD vector instructions. A problem which is now, fortunately, being addressed: http://hackage.haskell.org/trac/ghc/wiki/SIMD http://hackage.haskell.org/trac/ghc/wiki/SIMD But currently, Haskell is not really an option if you want to write serious ML software that rely on vector or matrix operations. [1] I did, for maximum entropy modeling. Believe me, it's not near competitive to C or C++ yet. But one day, it will probably be.

glimcat 15y ago
"C is faster" is a tired argument. The ability of this compiler or that to produce the most efficient possible code is almost never relevant outside of a few niche cases. Even UHF finance is such that the fractional return in computation time that you may get from using C isn't worth the development overhead. There are any number of great reasons to use C, but speed is rather outdated. In this case, you're also guilty of putting optimization before the algorithm. Squeezing out the last clock cycle doesn't matter when you have raw data in one hand and some high-level objectives in the other. Effective machine learning is more about finding ways to obtain and reduce data into a useful form, and finding ways to map between your design goals and workable algorithms. Speed is the last step, and often the only one which can be improved by throwing money at it. Besides which, if you want to have a pissing contest about speed, C is still going to lose to FPGAs. If it's that important, hire someone who knows Verilog.
- kristoffer 15y ago
  Well, FPGAs are not very cost effective for ML stuff I'd say. You're better of with a GPGPU then since it's mostly matrix operations anyway. Still, depending on the problem C could still be the best language, no need to piss on that ...
- Jach 15y ago
  Ssh, don't tell him about Nios et al. cores that let you write C again, or MyHDL to write Python that compiles to Verilog.
- danieldk 15y ago
  "C is faster" is a tired argument. The ability of this compiler or that to produce the most efficient possible code is almost never relevant outside of a few niche cases. What niches? ML is exactly one of the niches where highly optimized code helps tremendously. We use ML, amongst other things, for parse disambiguation and fluency ranking. Making a modification in, say the grammar, often results in training and retraining various components that have a mutual influence (e.g. parse disambiguation, auxiliary distributions for subcategorization frames, the part of speech tagger). Evaluation is also performed using ten-fold cross validation. Since a modification can also result in regressions, the effect of modifications are usually checked individually. Being able to validate changes in one hour rather than one day, helps the development of such a system tremendously. We developed some software first in other languages (Java, Scala, Python), usually to make a prototype for getting grip on a problem. But reimplementing it in C or C++ paid of tremendously on each occasion. Even though there is enough CPU time available (we have a 3280 core cluster). Another thing to note is that most machine learning techniques are fairly generic. So, usually there is already a perfectly fine C/C++ implementation available. Sometimes with bindings and all. In this case, you're also guilty of putting optimization before the algorithm. How do you know? Most data is actually preprocessed using Prolog or Perl. But the actual machine software is written in C or C++. Speed is the last step, and often the only one which can be improved by throwing money at it. Again, there are niches where it is worth it. And machine learning is often that niche, because people use huge data sets, complex problems, or both. Besides which, if you want to have a pissing contest about speed, C is still going to lose to FPGAs. Yes, but FPGAs are not readily available to our users :). But we are very much interested in GPU computing. --- Apart from all of this, it's funny that you pointing to the price of implementing in ML software in C. It's not as if there are large amounts of Haskell programmers available. Writing such software in Haskell has about the same economic risk as using COBOL ;).