3 ms·

One of the co-founders of SigOpt (YC W15) here. I'm happy to answer and questions about this post or the methods used. More info on the Bayesian methods behind

One of the co-founders of SigOpt (YC W15) here. I'm happy to answer and questions about this post or the methods used. More info on the Bayesian methods behind this can be found at sigopt.com/research as well!

pizza 11y ago
Actually, out of curiosity, would there be some way to use the inverse coloring transform + a lil noise to generate some kind of equivalence class of free training examples, sort of a la skip-gram?
- idewanck 11y ago
  Author here, you can definitely talk about augmenting your training data with some slight transforms of the labelled set you have. Common strategies are adding noise, rotation etc for images
  - pizza 11y ago
    Interesting. I guess the question then becomes what constitutes a "big" transformation that preserves relevant invariants.
bearzoo 11y ago
Well - just my two cents. The title feels inaccurate. You all are tuning hyper parameters with respect to the performance of the classification task. The bayesian optimization is really to optimize the unsupervised -> supervised pipeline. I was expecting some bayesian optimization of strictly unsupervised representation learning (ex. we have an autoencoder and use some bayesian optimization to tune hyper parameters in order to minimize a reconstruction error). This is really just supervised learning with even less supervision (which is quite typical).
- Zephyr314 11y ago
  Thanks for the note! We're using Bayesian optimization to tune both the hyperparameters of the unsupervised model and the supervised model, but you are correct that they are being done in unison with the overall accuracy being the target. The lift you get from adding the unsupervised step (and tuning it) is quite substantial (and statistically significant). The idea of tuning just the unsupervised part (or doing it independently) is great though. All the code for the post is available at https://github.com/sigopt/sigopt-examples/tree/master/unsupervised-model https://github.com/sigopt/sigopt-examples/tree/master/unsupe.... It would be interesting to see if doing that would make for a better overall accuracy.
tavert 11y ago
No constrained optimization?
- Zephyr314 11y ago
  SigOpt allows specifying constraint ranges for every parameter. If the parameter space isn't a tensor product you can report constraint violating suggestions as failures via our API [1] and SigOpt will take that into account as it optimizes. SigOpt isn't a constraint optimization package for solving things like k-SAT thought if that is what you were asking. [1]: https://sigopt.com/docs/endpoints/observations/create https://sigopt.com/docs/endpoints/observations/create
  - tavert 11y ago
    In addition to bounds, say I'd like to solve a problem where the variables must satisfy a linear or nonlinear system of equations. So overall there are fewer degrees of freedom than the number of variables, but the problem structure is such that it may be much more efficient to express the objective function and constraints in terms of the higher-dimensional space.
    - Zephyr314 11y ago
      This is where the reporting of "failures" as mentioned above can be helpful. Any suggestion that violates your system of equations based constraints can be instantly reported as a "failure," and this will be taken into account as SigOpt converges to the best parameters. You could also try to bake this into the objective function (with an L2 penalty for how bad it violates some constraints), depending on how hard the constraints of the problem actually are.
      - tavert 11y ago
        Equality constraints means you're working in a lower-dimensional subspace, I'd have to find out more about your algorithms but I'd be surprised if you wound up evaluating many feasible points at all (esp. with nonlinear equalities). L2 penalty would likely converge to an infeasible solution. Augmented Lagrangian would be better, but then you're making users handle dual updates. At that point I'd rather use an actual constrained optimization library that does the algorithm carefully, and use primal-dual interior point. Not having this kind of thing built in counts as "no constrained optimization" IMO.

pizza 11y ago

Actually, out of curiosity, would there be some way to use the inverse coloring transform + a lil noise to generate some kind of equivalence class of free training examples, sort of a la skip-gram?

idewanck 11y ago
Author here, you can definitely talk about augmenting your training data with some slight transforms of the labelled set you have. Common strategies are adding noise, rotation etc for images
- pizza 11y ago
  Interesting. I guess the question then becomes what constitutes a "big" transformation that preserves relevant invariants.

bearzoo 11y ago

Well - just my two cents. The title feels inaccurate. You all are tuning hyper parameters with respect to the performance of the classification task. The bayesian optimization is really to optimize the unsupervised -> supervised pipeline. I was expecting some bayesian optimization of strictly unsupervised representation learning (ex. we have an autoencoder and use some bayesian optimization to tune hyper parameters in order to minimize a reconstruction error). This is really just supervised learning with even less supervision (which is quite typical).

Zephyr314 11y ago
Thanks for the note! We're using Bayesian optimization to tune both the hyperparameters of the unsupervised model and the supervised model, but you are correct that they are being done in unison with the overall accuracy being the target. The lift you get from adding the unsupervised step (and tuning it) is quite substantial (and statistically significant). The idea of tuning just the unsupervised part (or doing it independently) is great though. All the code for the post is available at https://github.com/sigopt/sigopt-examples/tree/master/unsupervised-model https://github.com/sigopt/sigopt-examples/tree/master/unsupe.... It would be interesting to see if doing that would make for a better overall accuracy.

tavert 11y ago

No constrained optimization?

Zephyr314 11y ago
SigOpt allows specifying constraint ranges for every parameter. If the parameter space isn't a tensor product you can report constraint violating suggestions as failures via our API [1] and SigOpt will take that into account as it optimizes. SigOpt isn't a constraint optimization package for solving things like k-SAT thought if that is what you were asking. [1]: https://sigopt.com/docs/endpoints/observations/create https://sigopt.com/docs/endpoints/observations/create
- tavert 11y ago
  In addition to bounds, say I'd like to solve a problem where the variables must satisfy a linear or nonlinear system of equations. So overall there are fewer degrees of freedom than the number of variables, but the problem structure is such that it may be much more efficient to express the objective function and constraints in terms of the higher-dimensional space.
  - Zephyr314 11y ago
    This is where the reporting of "failures" as mentioned above can be helpful. Any suggestion that violates your system of equations based constraints can be instantly reported as a "failure," and this will be taken into account as SigOpt converges to the best parameters. You could also try to bake this into the objective function (with an L2 penalty for how bad it violates some constraints), depending on how hard the constraints of the problem actually are.
    - tavert 11y ago
      Equality constraints means you're working in a lower-dimensional subspace, I'd have to find out more about your algorithms but I'd be surprised if you wound up evaluating many feasible points at all (esp. with nonlinear equalities). L2 penalty would likely converge to an infeasible solution. Augmented Lagrangian would be better, but then you're making users handle dual updates. At that point I'd rather use an actual constrained optimization library that does the algorithm carefully, and use primal-dual interior point. Not having this kind of thing built in counts as "no constrained optimization" IMO.