4 ms·

Unsupervised Learning with Even Less Supervision Using Bayesian Optimization

85 points by Zephyr314 11y ago

pizza 11y ago
Now just throw some compressive sensing at the problem ;)
Zephyr314 11y ago
One of the co-founders of SigOpt (YC W15) here. I'm happy to answer and questions about this post or the methods used. More info on the Bayesian methods behind this can be found at sigopt.com/research as well!
- pizza 11y ago
  Actually, out of curiosity, would there be some way to use the inverse coloring transform + a lil noise to generate some kind of equivalence class of free training examples, sort of a la skip-gram?
  - idewanck 11y ago
    Author here, you can definitely talk about augmenting your training data with some slight transforms of the labelled set you have. Common strategies are adding noise, rotation etc for images
    - pizza 11y ago
      Interesting. I guess the question then becomes what constitutes a "big" transformation that preserves relevant invariants.
- bearzoo 11y ago
  Well - just my two cents. The title feels inaccurate. You all are tuning hyper parameters with respect to the performance of the classification task. The bayesian optimization is really to optimize the unsupervised -> supervised pipeline. I was expecting some bayesian optimization of strictly unsupervised representation learning (ex. we have an autoencoder and use some bayesian optimization to tune hyper parameters in order to minimize a reconstruction error). This is really just supervised learning with even less supervision (which is quite typical).
  - Zephyr314 11y ago
    Thanks for the note! We're using Bayesian optimization to tune both the hyperparameters of the unsupervised model and the supervised model, but you are correct that they are being done in unison with the overall accuracy being the target. The lift you get from adding the unsupervised step (and tuning it) is quite substantial (and statistically significant). The idea of tuning just the unsupervised part (or doing it independently) is great though. All the code for the post is available at https://github.com/sigopt/sigopt-examples/tree/master/unsupervised-model https://github.com/sigopt/sigopt-examples/tree/master/unsupe.... It would be interesting to see if doing that would make for a better overall accuracy.
- tavert 11y ago
  No constrained optimization?
  - Zephyr314 11y ago
    SigOpt allows specifying constraint ranges for every parameter. If the parameter space isn't a tensor product you can report constraint violating suggestions as failures via our API [1] and SigOpt will take that into account as it optimizes. SigOpt isn't a constraint optimization package for solving things like k-SAT thought if that is what you were asking. [1]: https://sigopt.com/docs/endpoints/observations/create https://sigopt.com/docs/endpoints/observations/create
    - tavert 11y ago
      In addition to bounds, say I'd like to solve a problem where the variables must satisfy a linear or nonlinear system of equations. So overall there are fewer degrees of freedom than the number of variables, but the problem structure is such that it may be much more efficient to express the objective function and constraints in terms of the higher-dimensional space.
      - Zephyr314 11y ago
        This is where the reporting of "failures" as mentioned above can be helpful. Any suggestion that violates your system of equations based constraints can be instantly reported as a "failure," and this will be taken into account as SigOpt converges to the best parameters. You could also try to bake this into the objective function (with an L2 penalty for how bad it violates some constraints), depending on how hard the constraints of the problem actually are.
        tavert 11y ago
        Equality constraints means you're working in a lower-dimensional subspace, I'd have to find out more about your algorithms but I'd be surprised if you wound up evaluating many feasible points at all (esp. with nonlinear equalities). L2 penalty would likely converge to an infeasible solution. Augmented Lagrangian would be better, but then you're making users handle dual updates. At that point I'd rather use an actual constrained optimization library that does the algorithm carefully, and use primal-dual interior point. Not having this kind of thing built in counts as "no constrained optimization" IMO.
lqdc13 11y ago
Is this the first OHAAS (Optimize Hyperparameters As A Service)?
- IanCal 11y ago
  There is/was whetlab which got bought out by Twitter if I remember rightly. It's a shame as I was using them and wanted to do it more.
  - bearzoo 11y ago
    https://github.com/JasperSnoek/spearmint https://github.com/JasperSnoek/spearmint
    - Zephyr314 11y ago
      We've found that SigOpt compares very well to spearmint, as well as MOE [1], which I wrote and open sourced around the same time spearmint was open sourced. We have a paper coming out soon comparing SigOpt rigorously to standard methods like random and grid search as well as other open source Bayesian methods like MOE [1], spearmint, HyperOpt [2], and SMAC [3] with good results. [1]: https://github.com/Yelp/MOE https://github.com/Yelp/MOE [2]: https://github.com/hyperopt/hyperopt https://github.com/hyperopt/hyperopt [3]: http://www.cs.ubc.ca/labs/beta/Projects/SMAC/ http://www.cs.ubc.ca/labs/beta/Projects/SMAC/
      - bearzoo 11y ago
        With spearmint I had the ability to modify the parameters of the mcmc sampling (e.g. burn in iterations). Will sigopt expose parameters for those of us who want to manipulate them? Will there be options to use different types of function estimators to estimate the mapping between hyper params and performance (i.e. what if I would like to use a neural network or a decision tree instead of gaussian processes)? I say these things because as someone who is active in machine learning - I often want to optimize hyper parameters. The type of people that are serious about optimizing hyper parameters (i.e. people who may not like to use grid or random searches) for a model are usually some what technical. Your product seems to be catered to those who may not be too technical (very simple interface, etc). How will you balance what you expose in the future without giving away too much of your underlying algorithms?

85 points by Zephyr314 11y ago

pizza 11y ago

Now just throw some compressive sensing at the problem ;)

One of the co-founders of SigOpt (YC W15) here. I'm happy to answer and questions about this post or the methods used. More info on the Bayesian methods behind this can be found at sigopt.com/research as well!

pizza 11y ago
Actually, out of curiosity, would there be some way to use the inverse coloring transform + a lil noise to generate some kind of equivalence class of free training examples, sort of a la skip-gram?
- idewanck 11y ago
  Author here, you can definitely talk about augmenting your training data with some slight transforms of the labelled set you have. Common strategies are adding noise, rotation etc for images
  - pizza 11y ago
    Interesting. I guess the question then becomes what constitutes a "big" transformation that preserves relevant invariants.
bearzoo 11y ago
Well - just my two cents. The title feels inaccurate. You all are tuning hyper parameters with respect to the performance of the classification task. The bayesian optimization is really to optimize the unsupervised -> supervised pipeline. I was expecting some bayesian optimization of strictly unsupervised representation learning (ex. we have an autoencoder and use some bayesian optimization to tune hyper parameters in order to minimize a reconstruction error). This is really just supervised learning with even less supervision (which is quite typical).
- Zephyr314 11y ago
  Thanks for the note! We're using Bayesian optimization to tune both the hyperparameters of the unsupervised model and the supervised model, but you are correct that they are being done in unison with the overall accuracy being the target. The lift you get from adding the unsupervised step (and tuning it) is quite substantial (and statistically significant). The idea of tuning just the unsupervised part (or doing it independently) is great though. All the code for the post is available at https://github.com/sigopt/sigopt-examples/tree/master/unsupervised-model https://github.com/sigopt/sigopt-examples/tree/master/unsupe.... It would be interesting to see if doing that would make for a better overall accuracy.
tavert 11y ago
No constrained optimization?
- Zephyr314 11y ago
  SigOpt allows specifying constraint ranges for every parameter. If the parameter space isn't a tensor product you can report constraint violating suggestions as failures via our API [1] and SigOpt will take that into account as it optimizes. SigOpt isn't a constraint optimization package for solving things like k-SAT thought if that is what you were asking. [1]: https://sigopt.com/docs/endpoints/observations/create https://sigopt.com/docs/endpoints/observations/create
  - tavert 11y ago
    In addition to bounds, say I'd like to solve a problem where the variables must satisfy a linear or nonlinear system of equations. So overall there are fewer degrees of freedom than the number of variables, but the problem structure is such that it may be much more efficient to express the objective function and constraints in terms of the higher-dimensional space.
    - Zephyr314 11y ago
      This is where the reporting of "failures" as mentioned above can be helpful. Any suggestion that violates your system of equations based constraints can be instantly reported as a "failure," and this will be taken into account as SigOpt converges to the best parameters. You could also try to bake this into the objective function (with an L2 penalty for how bad it violates some constraints), depending on how hard the constraints of the problem actually are.
      - tavert 11y ago
        Equality constraints means you're working in a lower-dimensional subspace, I'd have to find out more about your algorithms but I'd be surprised if you wound up evaluating many feasible points at all (esp. with nonlinear equalities). L2 penalty would likely converge to an infeasible solution. Augmented Lagrangian would be better, but then you're making users handle dual updates. At that point I'd rather use an actual constrained optimization library that does the algorithm carefully, and use primal-dual interior point. Not having this kind of thing built in counts as "no constrained optimization" IMO.

lqdc13 11y ago

Is this the first OHAAS (Optimize Hyperparameters As A Service)?

IanCal 11y ago
There is/was whetlab which got bought out by Twitter if I remember rightly. It's a shame as I was using them and wanted to do it more.
- bearzoo 11y ago
  https://github.com/JasperSnoek/spearmint https://github.com/JasperSnoek/spearmint
  - Zephyr314 11y ago
    We've found that SigOpt compares very well to spearmint, as well as MOE [1], which I wrote and open sourced around the same time spearmint was open sourced. We have a paper coming out soon comparing SigOpt rigorously to standard methods like random and grid search as well as other open source Bayesian methods like MOE [1], spearmint, HyperOpt [2], and SMAC [3] with good results. [1]: https://github.com/Yelp/MOE https://github.com/Yelp/MOE [2]: https://github.com/hyperopt/hyperopt https://github.com/hyperopt/hyperopt [3]: http://www.cs.ubc.ca/labs/beta/Projects/SMAC/ http://www.cs.ubc.ca/labs/beta/Projects/SMAC/
    - bearzoo 11y ago
      With spearmint I had the ability to modify the parameters of the mcmc sampling (e.g. burn in iterations). Will sigopt expose parameters for those of us who want to manipulate them? Will there be options to use different types of function estimators to estimate the mapping between hyper params and performance (i.e. what if I would like to use a neural network or a decision tree instead of gaussian processes)? I say these things because as someone who is active in machine learning - I often want to optimize hyper parameters. The type of people that are serious about optimizing hyper parameters (i.e. people who may not like to use grid or random searches) for a model are usually some what technical. Your product seems to be catered to those who may not be too technical (very simple interface, etc). How will you balance what you expose in the future without giving away too much of your underlying algorithms?