4 ms·

I'm one of the co-founders of SigOpt (YC W15) and am happy to answer any questions about SigOpt or the methods we apply. More info on our research (and example

I'm one of the co-founders of SigOpt (YC W15) and am happy to answer any questions about SigOpt or the methods we apply. More info on our research (and examples) can be found at https://sigopt.com/research https://sigopt.com/research

sevensor 10y ago
Does your platform support more than one level of preemption / lexicographic goal setting? (Like the ε-constraint scalarization technique, but with a second, third, nth set of goals between the first set and the objective?)
- mccourt 10y ago
  Our customers who are working with multicriteria problems have, thus far, had primarily two criteria, thus we have been helping them manage their two criteria problems into a scalar setting. As such, we do not, at this moment, permit the layers of ordering strategy you suggest through our API. To do so internally would introduce a complicated bifurcation between problems phrased with real-valued observations (as is our standard workflow), and the less informative comparative structure you're suggesting, whereby we would only be able to make statements about the relative order of points and not the magnitude by which they differ. If we were willing to impose a magnitude, doing so would revert the problem back into the weighted combination scalarization setting (or at least some norm-scalarization setting, if not the linear setting discussed in the post). I do not foresee us implementing such a tiered preemptive ordering any time soon.
  - sevensor 10y ago
    I agree that expressing the problem can become more confusing in the presence of more levels of preemption, however it can be an effective way to organize large numbers of criteria. If your customers are mainly working with biobjective problems, I can see why you wouldn't be too interested in adding more levels!
    - mccourt 10y ago
      And I can absolutely agree that, as more criteria arise, the mechanism for linear scalarization probably becomes more fragile (subject to inconsistent behavior from the coefficients). As a result, something less sensitive but more robust, such as the tiered ordering, is probably preferable. But yeah, we just have not seen the demand yet. What actually seems to be most common is that people who have ~10 metrics spend some time thinking about it, and then realize that they mostly only cared about 1-2 so long as the rest did not cause problems/failures. That was part of the reason I wrote about the epsilon-constraint idea. We do this in one sense within our company, but it's actually not within the context of a numerical multicriteria optimization problem. We are always trying to optimize around our customer's needs, which is in some ways a multicriteria problem involving balancing: 1) the "best" parameterization of a model subject to some (usually cross-validation) metric, 2) the "cost" (number of samples) required to optimize the model quality, 3) the "robustness" of (degree to which small parameter changes impact) the resulting solution, 4) the "parallel speed" (number of simultaneous suggestions) of the optimization process. We consult with enterprise customers to understand their needs and expectations regarding these criteria to produce a sort of hierarchical ordering (as you've suggested) which helps inform our optimization procedure (maybe a customer doesn't care as much about speed but definitely cares about robustness). Obviously, it's a relatively restricted problem, and we're not considering it in a rigorous mathematical framework (just how best to serve our customers). Because these factors have no real numerical relationship, the only mechanism we can use to balance the concerns is a relative ordering, which is then manage internally. We spoke about this design at the ICML AutoML workshop this year (A Strategy for Ranking ... at https://sites.google.com/site/automl2016/accepted-papers https://sites.google.com/site/automl2016/accepted-papers)
chaoxu 10y ago
I do research in combinatorial optimization. Does your company solve any problems with a combinatorial flavor? (say, things can be optimized using combinatorial algorithms instead of going for gradient descent.
- Zephyr314 10y ago
  We don't (yet), but we are growing rapidly and always looking for ways to help our customers solve optimization problems. Fwiw, we do support categorical parameters (as well as continuous and integer) and our ensemble of Bayesian optimization techniques are able to solve this mixed type problem much more efficiently than techniques like gradient decent. Although the way we handle the purely combinatorial (only category) problem isn't as flushed out as our mixed type problems. We are looking to grow the team (and our offerings) if you're interested though [1]. [1]: https://sigopt.com/careers https://sigopt.com/careers
  - wolfgke 10y ago
    > [We] are able to solve this mixed type problem much more efficiently than techniques like gradient decent. Naive gradient descent is probably the simplest strategy that one can imagine. How do your algorithms compare to Newton methods for minimizing the primal-dual gap (interior point methods)?
    - Zephyr314 10y ago
      We compare to some standard convex optimization techniques implemented in scipy here [1]. We have comparisons to some other Bayesian methods here [2]. I'm happy to answer any questions! [1]: https://github.com/sigopt/sigopt-examples/blob/master/ipython-notebook-example/SigOpt_Introduction.ipynb https://github.com/sigopt/sigopt-examples/blob/master/ipytho... [2]: http://arxiv.org/abs/1603.09441 http://arxiv.org/abs/1603.09441
- pizza 10y ago
  What are the bread and butter of combinatorial optimization? In other words, the concepts you would first come across, at an undergrad level if possible?

sevensor 10y ago

Does your platform support more than one level of preemption / lexicographic goal setting? (Like the ε-constraint scalarization technique, but with a second, third, nth set of goals between the first set and the objective?)

mccourt 10y ago
Our customers who are working with multicriteria problems have, thus far, had primarily two criteria, thus we have been helping them manage their two criteria problems into a scalar setting. As such, we do not, at this moment, permit the layers of ordering strategy you suggest through our API. To do so internally would introduce a complicated bifurcation between problems phrased with real-valued observations (as is our standard workflow), and the less informative comparative structure you're suggesting, whereby we would only be able to make statements about the relative order of points and not the magnitude by which they differ. If we were willing to impose a magnitude, doing so would revert the problem back into the weighted combination scalarization setting (or at least some norm-scalarization setting, if not the linear setting discussed in the post). I do not foresee us implementing such a tiered preemptive ordering any time soon.
- sevensor 10y ago
  I agree that expressing the problem can become more confusing in the presence of more levels of preemption, however it can be an effective way to organize large numbers of criteria. If your customers are mainly working with biobjective problems, I can see why you wouldn't be too interested in adding more levels!
  - mccourt 10y ago
    And I can absolutely agree that, as more criteria arise, the mechanism for linear scalarization probably becomes more fragile (subject to inconsistent behavior from the coefficients). As a result, something less sensitive but more robust, such as the tiered ordering, is probably preferable. But yeah, we just have not seen the demand yet. What actually seems to be most common is that people who have ~10 metrics spend some time thinking about it, and then realize that they mostly only cared about 1-2 so long as the rest did not cause problems/failures. That was part of the reason I wrote about the epsilon-constraint idea. We do this in one sense within our company, but it's actually not within the context of a numerical multicriteria optimization problem. We are always trying to optimize around our customer's needs, which is in some ways a multicriteria problem involving balancing: 1) the "best" parameterization of a model subject to some (usually cross-validation) metric, 2) the "cost" (number of samples) required to optimize the model quality, 3) the "robustness" of (degree to which small parameter changes impact) the resulting solution, 4) the "parallel speed" (number of simultaneous suggestions) of the optimization process. We consult with enterprise customers to understand their needs and expectations regarding these criteria to produce a sort of hierarchical ordering (as you've suggested) which helps inform our optimization procedure (maybe a customer doesn't care as much about speed but definitely cares about robustness). Obviously, it's a relatively restricted problem, and we're not considering it in a rigorous mathematical framework (just how best to serve our customers). Because these factors have no real numerical relationship, the only mechanism we can use to balance the concerns is a relative ordering, which is then manage internally. We spoke about this design at the ICML AutoML workshop this year (A Strategy for Ranking ... at https://sites.google.com/site/automl2016/accepted-papers https://sites.google.com/site/automl2016/accepted-papers)

chaoxu 10y ago

I do research in combinatorial optimization. Does your company solve any problems with a combinatorial flavor? (say, things can be optimized using combinatorial algorithms instead of going for gradient descent.

Zephyr314 10y ago
We don't (yet), but we are growing rapidly and always looking for ways to help our customers solve optimization problems. Fwiw, we do support categorical parameters (as well as continuous and integer) and our ensemble of Bayesian optimization techniques are able to solve this mixed type problem much more efficiently than techniques like gradient decent. Although the way we handle the purely combinatorial (only category) problem isn't as flushed out as our mixed type problems. We are looking to grow the team (and our offerings) if you're interested though [1]. [1]: https://sigopt.com/careers https://sigopt.com/careers
- wolfgke 10y ago
  > [We] are able to solve this mixed type problem much more efficiently than techniques like gradient decent. Naive gradient descent is probably the simplest strategy that one can imagine. How do your algorithms compare to Newton methods for minimizing the primal-dual gap (interior point methods)?
  - Zephyr314 10y ago
    We compare to some standard convex optimization techniques implemented in scipy here [1]. We have comparisons to some other Bayesian methods here [2]. I'm happy to answer any questions! [1]: https://github.com/sigopt/sigopt-examples/blob/master/ipython-notebook-example/SigOpt_Introduction.ipynb https://github.com/sigopt/sigopt-examples/blob/master/ipytho... [2]: http://arxiv.org/abs/1603.09441 http://arxiv.org/abs/1603.09441
pizza 10y ago
What are the bread and butter of combinatorial optimization? In other words, the concepts you would first come across, at an undergrad level if possible?