5 ms·

Machine learning for the impatient: algorithms tuning algorithms

155 points by aelaguiz 14y ago

bencpeters 14y ago
I had a few questions about the actual implementation of this stuff. I took the coursera ML course, so much of the terminology and techniques are familiar after that, but Professor Ng structured the exercises in the course around Matlab/Octave and suggested using one of these tools for a first-pass solution when implementing machine learning problems. Have you used Matlab much in your work? How does the performance (and libraries available) compare with Python? Also, does anyone know a good resource for finding good, high performance ML libraries for other languages (Ruby, C++, etc.)?
- aelaguiz 14y ago
  I used to prototype with Octave but I actually find I"m much faster in python. I push my tests to EC2 and just use cProfile to make sure I'm not doing anything silly before hand. I parallelize with celery, but I've also used pp with some limited success. Performance is sufficient for my needs currently but not for production or real-time.
- ogrisel 14y ago
  matlab and python + numpy + scipy + matplotlib give the same kind of performance for the same level of features and ability to do quick exploratory prototyping with high level, vector based datastructures. However one is free (beer and freedom) and multipurpose (you can preprocess string data and expose your model with a HTTP API) while the other is much less so. In python + numpy you should always profile and if the bottleneck is in python interpreted code (rather than a low level numpy call) you can always rewrite the offending python loop in cython.
- punee 14y ago
  Scikit-learn for Python, great tutorials/user guide covering various ML techniques, makes prototyping very easy: http://scikit-learn.org/stable/ http://scikit-learn.org/stable/
  - deleted 14y ago
    [deleted]
  - deleted 14y ago
    [deleted]
- kilburn 14y ago
  One of the most famous collections/frameworks for development and usage of (traditional) machine learning algorithms is weka [1]. It is written in java, and actively developed by the machine learning group of the University of Waikato. They also seem to have a forum to discuss, which should be a good resource given that a lot of their users are actually researchers. The very same group is also developing another framework (also in java), MOA [2], that is oriented towards "big data": the algorithms used in there are specifically tailored for huge datasets and/or realtime (streaming) machine learning. [1] http://www.cs.waikato.ac.nz/ml/weka/ http://www.cs.waikato.ac.nz/ml/weka/ [2] http://moa.cs.waikato.ac.nz/ http://moa.cs.waikato.ac.nz/
scottfr 14y ago
Such aggressive usage of the test data set in determining the tuning parameters in effect makes your test data set part of your training data set. The more times you go back to your test data set to evaluate the effectiveness of a model, the more optimistic your error predictions will be and the greater your chance of overfitting. Several iterations of his loop will probably improve the model, but if you keep repeating it eventually the true model performance will start to degrade.
- aelaguiz 14y ago
  Yep, I've definitely seen that. I bet if I were to reserve a portion of the dataset for validation after all tuning, I'd get a much better measure of it's actual ability at prediction. Using it in the way that I did, I wouldn't be surprised if there was significant overfitting. In fact I'd be surprised if there wasn't.
- gersh 14y ago
  You can split it into training, validation, and test sets if you have enough data. Then, the validation set would be used for the training. Presumably by using a genetic algorithm, he isn't going for a perfect fit, which may help avoid overfitting.
- brador 14y ago
  Question: Why is the data for machine learning split into a training dataset and a test dataset? Wouldn't using the entire dataset to build the model result in greater accuracy of the predictions?
  - aelaguiz 14y ago
    For this I actually had sufficient data available that adding more didn't do much. You need a testing dataset in order to validate the performance of the model. If you validate against the training set really what you're doing is measuring the model's ability to fit the training data - which it will be able to do with high accuracy. That will, however, result in a much diminished ability to predict any new games - as it isn't "learning" the features of college basketball as much as it is memorizing the contents of the training set. As others pointed out, it'd probably even be better to add a third grouping which is tested against only after the algorithm has finished - as an objective validation against as of yet unseen data.
danger 14y ago
As another commenter pointed out, the accuracy really needs to be evaluated using a validation set, not the test set--the approach described in the post is training with the testing data. In the field, we call this "cheating". The basic idea of automatically tuning hyperparameters (the things this post discusses tuning with genetic algorithms) is cool, though, and is becoming a popular subject in machine learning research. A couple recent research papers on the topic are pretty readable: Algorithms for Hyper-Parameter Optimization: http://books.nips.cc/papers/files/nips24/NIPS2011_1385.pdf http://books.nips.cc/papers/files/nips24/NIPS2011_1385.pdf Practical Bayesian Optimization of Machine Learning Algorithms: http://arxiv.org/abs/1206.2944 http://arxiv.org/abs/1206.2944
- danger 14y ago
  Also, for people who are interested in the application of predicting college basketball with machine learning, there's a Google group that is worth joining: https://groups.google.com/group/machine-march-madness https://groups.google.com/group/machine-march-madness
- aelaguiz 14y ago
  Thanks for the information! I've updated the article to reflect this. Here's a question: where does "the field" hang out? Is there a cohesive community of any sort?
  - danger 14y ago
    I'd say the closest thing to a cohesive community would be the MetaOptimize Q&A forum, but maybe others have other suggestions: http://metaoptimize.com/qa http://metaoptimize.com/qa
  - gammarator 14y ago
    The Kaggle forums can be a good resource, and competing there is a good way to polish your skills. Also, you might check out Random Forest algorithms--they're high performance but still very beginner friendly, as there aren't many parameters to tune. There's a nice implementation in the excellent scikits-learn python library.
misiti3780 14y ago
This article is very interesting but I am really confused about the edit: "A few people have pointed out that using the testing set for tuning demands that final measure of effectiveness be doing using a validation test set which is not part of either the training or testing datasets. This is due for the very real potential of over fitting. Also – apparently this technique is called “Hyper-Parameter Optimization.” A helpful commenter over at Hacker News supplied the following resources" Does that mean there is definitely overfitting going on but it is acceptable for the purposes of the article. That 82.5% accuracy rate has overfitting written all over it. Good stuff regardless!

155 points by aelaguiz 14y ago

I had a few questions about the actual implementation of this stuff. I took the coursera ML course, so much of the terminology and techniques are familiar after that, but Professor Ng structured the exercises in the course around Matlab/Octave and suggested using one of these tools for a first-pass solution when implementing machine learning problems. Have you used Matlab much in your work? How does the performance (and libraries available) compare with Python? Also, does anyone know a good resource for finding good, high performance ML libraries for other languages (Ruby, C++, etc.)?

aelaguiz 14y ago
I used to prototype with Octave but I actually find I"m much faster in python. I push my tests to EC2 and just use cProfile to make sure I'm not doing anything silly before hand. I parallelize with celery, but I've also used pp with some limited success. Performance is sufficient for my needs currently but not for production or real-time.
ogrisel 14y ago
matlab and python + numpy + scipy + matplotlib give the same kind of performance for the same level of features and ability to do quick exploratory prototyping with high level, vector based datastructures. However one is free (beer and freedom) and multipurpose (you can preprocess string data and expose your model with a HTTP API) while the other is much less so. In python + numpy you should always profile and if the bottleneck is in python interpreted code (rather than a low level numpy call) you can always rewrite the offending python loop in cython.
punee 14y ago
Scikit-learn for Python, great tutorials/user guide covering various ML techniques, makes prototyping very easy: http://scikit-learn.org/stable/ http://scikit-learn.org/stable/
- deleted 14y ago
  [deleted]
- deleted 14y ago
  [deleted]
kilburn 14y ago
One of the most famous collections/frameworks for development and usage of (traditional) machine learning algorithms is weka [1]. It is written in java, and actively developed by the machine learning group of the University of Waikato. They also seem to have a forum to discuss, which should be a good resource given that a lot of their users are actually researchers. The very same group is also developing another framework (also in java), MOA [2], that is oriented towards "big data": the algorithms used in there are specifically tailored for huge datasets and/or realtime (streaming) machine learning. [1] http://www.cs.waikato.ac.nz/ml/weka/ http://www.cs.waikato.ac.nz/ml/weka/ [2] http://moa.cs.waikato.ac.nz/ http://moa.cs.waikato.ac.nz/

scottfr 14y ago

Such aggressive usage of the test data set in determining the tuning parameters in effect makes your test data set part of your training data set. The more times you go back to your test data set to evaluate the effectiveness of a model, the more optimistic your error predictions will be and the greater your chance of overfitting. Several iterations of his loop will probably improve the model, but if you keep repeating it eventually the true model performance will start to degrade.

aelaguiz 14y ago
Yep, I've definitely seen that. I bet if I were to reserve a portion of the dataset for validation after all tuning, I'd get a much better measure of it's actual ability at prediction. Using it in the way that I did, I wouldn't be surprised if there was significant overfitting. In fact I'd be surprised if there wasn't.
gersh 14y ago
You can split it into training, validation, and test sets if you have enough data. Then, the validation set would be used for the training. Presumably by using a genetic algorithm, he isn't going for a perfect fit, which may help avoid overfitting.
brador 14y ago
Question: Why is the data for machine learning split into a training dataset and a test dataset? Wouldn't using the entire dataset to build the model result in greater accuracy of the predictions?
- aelaguiz 14y ago
  For this I actually had sufficient data available that adding more didn't do much. You need a testing dataset in order to validate the performance of the model. If you validate against the training set really what you're doing is measuring the model's ability to fit the training data - which it will be able to do with high accuracy. That will, however, result in a much diminished ability to predict any new games - as it isn't "learning" the features of college basketball as much as it is memorizing the contents of the training set. As others pointed out, it'd probably even be better to add a third grouping which is tested against only after the algorithm has finished - as an objective validation against as of yet unseen data.

danger 14y ago

As another commenter pointed out, the accuracy really needs to be evaluated using a validation set, not the test set--the approach described in the post is training with the testing data. In the field, we call this "cheating". The basic idea of automatically tuning hyperparameters (the things this post discusses tuning with genetic algorithms) is cool, though, and is becoming a popular subject in machine learning research. A couple recent research papers on the topic are pretty readable: Algorithms for Hyper-Parameter Optimization: http://books.nips.cc/papers/files/nips24/NIPS2011_1385.pdf http://books.nips.cc/papers/files/nips24/NIPS2011_1385.pdf Practical Bayesian Optimization of Machine Learning Algorithms: http://arxiv.org/abs/1206.2944 http://arxiv.org/abs/1206.2944

danger 14y ago
Also, for people who are interested in the application of predicting college basketball with machine learning, there's a Google group that is worth joining: https://groups.google.com/group/machine-march-madness https://groups.google.com/group/machine-march-madness
aelaguiz 14y ago
Thanks for the information! I've updated the article to reflect this. Here's a question: where does "the field" hang out? Is there a cohesive community of any sort?
- danger 14y ago
  I'd say the closest thing to a cohesive community would be the MetaOptimize Q&A forum, but maybe others have other suggestions: http://metaoptimize.com/qa http://metaoptimize.com/qa
- gammarator 14y ago
  The Kaggle forums can be a good resource, and competing there is a good way to polish your skills. Also, you might check out Random Forest algorithms--they're high performance but still very beginner friendly, as there aren't many parameters to tune. There's a nice implementation in the excellent scikits-learn python library.

misiti3780 14y ago

This article is very interesting but I am really confused about the edit: "A few people have pointed out that using the testing set for tuning demands that final measure of effectiveness be doing using a validation test set which is not part of either the training or testing datasets. This is due for the very real potential of over fitting. Also – apparently this technique is called “Hyper-Parameter Optimization.” A helpful commenter over at Hacker News supplied the following resources" Does that mean there is definitely overfitting going on but it is acceptable for the purposes of the article. That 82.5% accuracy rate has overfitting written all over it. Good stuff regardless!