6 ms·
The algorithm looks nice. However, the comparison to Bayesian optimization doesn't really hold as-is. The slowness of naive O(n^3) Bayesian optimization is well
by tziki 9y ago
The algorithm looks nice. However, the comparison to Bayesian optimization doesn't really hold as-is. The slowness of naive O(n^3) Bayesian optimization is well-known, and while I've never personally needed a faster solution, there's literature on eg. using approximate matrix inverse or special factorizations to speed up the process. You'll be hard pressed to find anyone actually suffering from the O(n^3) limitation.
- chroem- 9y agoHi there, That's a fair point about the time complexity, though the algorithm does at least demonstrate a real-world speedup of several orders of magnitude against fmfn's Bayesian Optimization implementation. However, I am not sure which performance optimizations he has implemented. Simple boils down to a handful of constant size matrix multiplies and a heap insertion at each iteration, so its performance is hard to beat. I will update the first paragraph of the readme.
- highd 9y agoBayesian optimization is often applied on problems where the cost of function evaluation greatly exceeds the cost of any posterior updates etc. In this case sample efficiency is the real benchmark, and you should be looking at realistic problems with d>>2.
- chroem- 9y agoHi there, Generally speaking the heuristic for applying BO to optimization problems is that the objective function must take at least as long to evaluate as an iteration of BO. For an equally sample-efficient algorithm with negligible overhead, you can now take twice as many samples and find better optima for a large class of problems. The shown examples are all 2D because plotting high dimensional space is rather difficult. Also, this is intended as an early proof of concept. A more formal description of the algorithm is coming, along with better examples.