10 ms·
Google Vizier: A Service for Black-Box Optimization
- kevlar1818 8y agoOkay, this is awesome (and easy to miss with a just a cursory skim): "5.3 Delicious Chocolate Chip Cookies Vizier is also used to solve complex black–box optimization problems arising from physical design or logistical problems. Here we present an example that highlights some additional capabilities of the system: finding the most delicious chocolate chip cookie recipe from a parameterized space of recipes... We provided recipes to contractors responsible for providing desserts for Google employees. The head chefs among the contractors were given discretion to alter parameters if (and only if) they strongly believed it to be necessary, but would carefully note what alterations were made. The cookies were baked, and distributed to the cafes for taste–testing. Cafe goers tasted the cookies and provided feedback via a survey. Survey results were aggregated and the results were sent back to Vizier. The “machine learning cookies” were provided about twice a week over several weeks... The cookies improved significantly over time; later rounds were extremely well-rated and, in the authors’ opinions, delicious."
- rnjs 8y agoI noted this section when I read the paper as well... This is cute, but "in the authors' opinions, delicious" does not contribute anything to scientific research. Yes, you can use black-box optimization for cookie recipes. No, you should not make any sort of performance claims nor talk about "significant" improvement unless you back it up.
- marksomnian 8y agoThat's the authors' sense of humour. The more relevant part for science is "extremely well-rated".
- ec109685 8y agoTheir back up: later rounds were extremely well-rated
- rnjs 8y agoHow about "My brand new image recognition architecture performs extremely well on benchmarks" But it's worse than the last 100 SOTA models...
- pm90 8y ago> in the authors' opinions, delicious Can there be/Is there an objective way to measure deliciousness? If not what can they really say?
- dekhn 8y agoTo the extent that deliciousness is a physiologically measurable phenomenon, or there are proxy measures that correlated with it, yes, there is an objective way. You give large numbers of people cookies and have them rate it. Tastes vary but generally if you make something with umami, crunch, fat, and sugar, people will say it is delicious.
- rnjs 8y ago> Cafe goers tasted the cookies and provided feedback via a survey. Survey results were aggregated and the results were sent back to Vizier. This, from the paper, is a great start on measuring deliciousness. But if you go to that effort, why not provide a benchmark? Such as the existing chocolate chip cookie recipe or the average scores from 10 random chocolate chip cookie recipes? As a reader, I might suspect that performance was not that great compared to baselines but that the authors were able to mask it by making this qualitative claim ("the cookies were delicious") -- which I'm sure was still true!
- AstralStorm 8y agoThis technique is known as anchoring and should be used alongside a descriptive scale when making subjective comparisons.
- tripzilch 8y agoYep. There's a lot of pre-existing science that this research ignored ...
- tripzilch 8y agoBut it really is not a very great start on measuring deliciousness. Like said elsewhere in this thread, if after a few batches the algorithm optimizes into a local maximum gully that some people don't like, but others really love, then the ones that dislike it will stop taking the cookies and thus stop rating them. But does that mean they're less delicious? Nobody knows! Because certainly the first and foremost great start on measuring deliciousness, is doing research on what is deliciousness, and are there any good methods to measure deliciousness that can be compared with other literature and research, and if that fails, either measure something else or defend why you chose a target metric that can't be measured properly. The latter being valuable if your way of measuring is novel and you think it's better than what has been used so far in the field. But you gotta provide a good argument for that, and the method used in this experiment wasn't particularly novel, simply flawed. I mean you can be all giggly-googly about it, but food science is a thing.
- halbritt 8y agoDid they post the recipe?
- halbritt 8y agoThey didn't. I am very disappointed. I, for one, welcome our new robot overlords and also want to try their chocolate chip cookies.
- joshuamorton 8y agoHere you go: https://www.blog.google/technology/research/makings-smart-cookie/ https://www.blog.google/technology/research/makings-smart-co...
- pliny 8y agoThese are not the vizier cookies.
- joshuamorton 8y agoOh darn you're right (to clarify, I'm pretty sure those are vizier cookies, just not the vizier cookies). I have the recipe sitting on my counter at home. I thought it was public, but it doesn't' seem to exist online anywhere.
- deleted 8y ago[deleted]
- sitkack 8y agoThe secret formula is love!
- pliny 8y agohttps://research.google.com/pubs/archive/46507.pdf https://research.google.com/pubs/archive/46507.pdf
- dekhn 8y agoIt's great that they published this - I wasn't sure if I could share this externally. I see they left the ingredient that really improved the scores unexpectedly out, presumbly for proprietary reasons.
- hn_throwaway_99 8y agoWeed?
- sp332 8y agoYou're not thinking of these gluten-free cardamom cookies? https://www.blog.google/technology/research/makings-smart-cookie/ https://www.blog.google/technology/research/makings-smart-co...
- patejam 8y agoThat ingredient list is missing the "secret ingredient" that your parent post is talking about, so nah.
- Itsdijital 8y agoThey most likely are holding it back for a bigger media event where they will show them off and every tech blog on the internet will run a story about them.
- joefkelley 8y agoAre you talking about orange extract? or cayenne pepper? The post is here: https://ai.google/research/pubs/pub46507 https://ai.google/research/pubs/pub46507 and direct link to paper is here: https://storage.googleapis.com/pub-tools-public-publication-data/pdf/f52ac33bc9d1adecd3a8037a7009b185fd934f0e.pdf https://storage.googleapis.com/pub-tools-public-publication-... And the recipe I found internally is "The Real California Cookie" in that paper: Bake at 163C (325◦ F) until brown: • 167 grams of all-purpose flour. • 245 grams of milk chocolate chips. • 0.60 tsp. baking soda. • 0.50 tsp. salt. • 0.125 tsp. cayenne pepper. • 127 grams of sugar (31% medium brown, 69% white). • 25.7 grams of egg. • 81.3 grams of butter. • 0.12 tsp. orange extract. • 0.75 tsp. vanilla extract.
- jcims 8y agoGood application for Ben Krasnow's Cookie Perfection machine - http://benkrasnow.blogspot.com/2014/01/cookie-perfection-machine.html http://benkrasnow.blogspot.com/2014/01/cookie-perfection-mac...
- meinstream 8y agoI worked for Google at the time they went through the cookie iterations in the cafes. Personally I disliked all the cookies that ever came out of this project, mainly because they contained spices, I heard the same thing from many others. After you got handed a cookie you were asked to fill out a feedback form on a tablet, which almost always is annoying. But more importantly, the process of optimizing the ingredients left out many parts of what makes food enjoyable, temperature and texture for example. I know this was meant to be a fun showcase of ML but to me this is still my favourite example of explaining misuse of ML technology where a simpler statistical model supported by expert opinion would have outperformed the taste of the cookies. Eg, any cookie expert might know that the number of people who like spicy cookies is only a small subset of all cookie lovers.
- joshuamorton 8y agoIf I recall, the spices were an intentional choice of a kind of pathological input to the optimizer, and iirc it did what you might expect (minimize the amount of cayenne to the smallest in the acceptable range).
- shmageggy 8y agoSounds like you got the less tasty side of the explore/exploit tradeoff.
- oh-kumudo 8y agoI think this is exactly the case how ML could go wrong. I would assume the people who hate the cookies probably will not bother to submit a survey and try again. The sample might be biased and not representative of the general population, thus overfitting, performs badly on a boarder test set.
- whazor 8y agoOur dystopian future where we will be forced fed cookies in order for accurate optimization.
- 8y ago
- iamgopal 8y agoWhy I have an idea about receipe optimisation as a service ?
- dallen33 8y agoSeems to be missing the key ingredient of the best tasting chocolate chip cookies: espresso powder.
- sannee 8y agoIs this possibly related to the cookie machine by Ben Krasnow from several years ago? [1] [1] https://www.youtube.com/watch?v=8YEdHjGMeho https://www.youtube.com/watch?v=8YEdHjGMeho
- indiesolver 8y agoA related service currently in beta: https://indiesolver.com/access https://indiesolver.com/access
- malkia 8y agoThis site was forbidden to look at from the company I'm working in?
- indiesolver 8y agoThanks for reporting! The site is hosted on Google Cloud Platform. If some of GCP's IPs are banned in your company, then it might be the reason.
- mark212 8y agoChrome on Windows 10 gave me this error (and no, Google is not blocked on my ISP): "The page you are trying to view cannot be shown because the authenticity of the received data could not be verified."
- beejiu 8y agoIt is referenced in the paper, but it's worth pointing out Yelp's MOE. This is an open source (Apache 2.0) implementation for black-box optimization. It doesn't look like it is well maintained, but it is interesting nonetheless. https://github.com/Yelp/MOE https://github.com/Yelp/MOE
- Zephyr314 8y agoHi, I'm one of the co-authors of MOE (it was a fork of my PhD thesis). We haven't been actively maintaining the open source version of MOE because we built it into a SaaS service 4 years ago as SigOpt (YC W15). Since then we've also had some of the authors of BayesOpt and HyperOpt work with us. Let me know if you'd like to give it a shot. It is also completely free for academics [1]. If you'd like to see some of the extensions we've made to our original approaches as we've built out our ensemble of optimization algorithms check out our research page [2]. [1]: https://sigopt.com/edu https://sigopt.com/edu [2]: https://sigopt.com/research https://sigopt.com/research
- pm90 8y agoTotally calling the next GCP AI Service: VaaS (Vizier as a Service)
- coredog64 8y agoWill the OSS version be called VaaSecotomy?
- usefulcat 8y agoI know a guy who told me 8 years ago about an idea he had for a "black box" optimization consulting service. You give him a pile of data, he uses ML to optimize for whatever.
- minimaxir 8y agoThat's known nowadays as AutoML. Google recently released AutoML products for image and text classification. https://cloud.google.com/automl/ https://cloud.google.com/automl/
- abhgh 8y agoFYI there is also a different "automl" suite of libraries maintained by Frank Hutters lab at the University of Freiburg - [1]. His group has been leading a lot of research in the area of hyperparameter tuning. [1] https://www.automl.org/ https://www.automl.org/
- dekhn 8y agoThis is already part of Google's Cloud ML Engine product.
- minimaxir 8y agoThis product is publicly available as a part of the hyperparameter turning in Cloud ML Engine (although I haven't played with it): https://cloud.google.com/ml-engine/docs/tensorflow/hyperparameter-tuning-overview https://cloud.google.com/ml-engine/docs/tensorflow/hyperpara...
- riku_iki 8y agoWhy do you think it is vizier, and not something developed specifically for clous?..
- minimaxir 8y agoIf you click through the explanation in the documentation: > Cloud Machine Learning Engine is a managed service that enables you to easily build machine learning models that work on any type of data, of any size. And one of its most powerful capabilities is HyperTune, which is hyperparameter tuning as a service using Google Vizier. https://cloud.google.com/blog/products/gcp/hyperparameter-tuning-cloud-machine-learning-engine-using-bayesian-optimization https://cloud.google.com/blog/products/gcp/hyperparameter-tu...
- sevensor 8y agoOnly up to 64 variables? Why such small problems? And why are their results are averaged across all of the different test functions? I'd like to see the performance difference between Rosenbrock and Rastrigin, thank you very much. And they have a weird fixation on stopping rules, when in general your stopping rule is how many evaluations you can afford. Was this written by interns? There's no discussion of the retrogression by CMAES on small problems? What parameters does their algorithm have and how sensitive is it to its parameterization? What parameters did they use for the competing algorithms? If I were reviewing this paper I'd have these and other pointed questions for them. It's as if they did a cursory lit review and then decided to reinvent the entire field. Their one contribution seems to be not allowing you to run it on your own hardware.
- indiesolver 8y agoHow long it will take CMAES to solve 10-dimensional Rosenbrock? Here, it takes about 700 function evaluations: https://indiesolver.com/tutorials/rosenbrock_python https://indiesolver.com/tutorials/rosenbrock_python Please note that their work is primarily for expensive optimization where you cannot afford millions of function evaluations like you would need on rotated Rastrigin to solve it exactly for n>64.
- sevensor 8y ago> How long it will take CMAES to solve 10-dimensional Rosenbrock? Here, it takes about 700 function evaluations It really depends on parameterization though. Did they use off-the-shelf parameters? Did they make any attempt to tune CMAES for their test functions? Did they make any attempt to tune their own optimizer for the test functions? It's really easy to put your thumb on the scale when you're writing a paper about a black-box optimizer, and my default reaction is to assume that any paper claiming to equal or outdo CMAES is pure B.S. This paper did nothing to convince me otherwise.
- ScoutOrgo 8y agoIt is 64 hyper-parameters, not variables, which is a huge amount. Stopping rules are also a requirement of tuning. Can't just have a while True:. If an objective function doesn't improve after 100 (or any #), there needs to be logic to stop the trials since this system is apparently serving all of Alphabet.
- btown 8y agoWhetlab, a startup founded by some former colleagues of mine, provided a service just like this 4 years ago (in fact, it's referenced in the Vizier paper as its open-source variant Spearmint), but unfortunately, it was acquired and shut down by Twitter: https://venturebeat.com/2015/06/17/twitter-acquires-machine-learning-startup-whetlab-will-kill-the-service-on-july-15/ https://venturebeat.com/2015/06/17/twitter-acquires-machine-... Black-box optimization is a hugely important problem to solve, especially when experiments require real wet-work (i.e. medicine, chemistry, etc.). Kudos to Google for commercializing this - I expect it will see a lot of use in those fields. But it's bittersweet to know that it's taken this long for this type of application to be promoted like this.
- indiesolver 8y agoWhetlab was the first startup in this domain. Then, followed by SigOpt (https://sigopt.com/ https://sigopt.com/) which is doing a great job at popularizing the concept.
- levesque 8y agoYeah SigOpt clearly has taken the space left by Whetlab. They must not be too pleased about this announcement, I mean generally the worst thing that can happen to your startup is a tech giant launching in your space. Then again the problem of hyperparameter/black-box optimization is so ubiquitous that there should be enough space for them both.
- btown 8y agoI wasn't familiar with SigOpt - this is really cool. I actually think SigOpt will see this as great publicity telling people about the usefulness of the concept - and whereas Google won't lift a finger for (insert hedge fund or pharmaceutical giant here) beyond maintaining uptime, SigOpt can provide those customers with customized advice about how to integrate, what pitfalls to watch for, how to design experiments to maximally take advantage of their technology. And they're competitive with Spearmint (though not necessarily the closed-source versions of it used at Whetlab), though Vizier remains to be seen: https://arxiv.org/pdf/1603.09441.pdf https://arxiv.org/pdf/1603.09441.pdf
- richardliaw 8y agoVery related open source project from the RISE Lab at UC Berkeley: http://ray.readthedocs.io/en/latest/tune.html http://ray.readthedocs.io/en/latest/tune.html
- pinhead 8y agoVery cool project! Glad to see support for PBT and HyperBand. Also related to https://www.microsoft.com/en-us/research/publication/hyperdrive-exploring-hyperparameters-pop-scheduling/ https://www.microsoft.com/en-us/research/publication/hyperdr...
- deleted 8y ago[deleted]
- abhgh 8y agoI had come across this library - [1] - a while ago that claims to be the open source equivalent of Vizier. Does anyone know how well justified the claim is? [1] https://github.com/tobegit3hub/advisor https://github.com/tobegit3hub/advisor
- deleted 8y ago[deleted]
- Zephyr314 8y agoSigOpt (YC W15) is a related service that performs a superset of these features as a SaaS API (I am one of the co-founders). We've been solving this problem for customers around the world for the last 4 years and have extended a lot of the original research I started at Yelp with MOE [1]. We employ an ensemble of optimization techniques and provide seamless integration with any pipeline. I'm happy to answer any questions about the product or technology. We're completely free for academics [2] and publish research like the above at ICML, NIPS, AISTATS regularly [3]. [1]: https://github.com/yelp/MOE https://github.com/yelp/MOE [2]: https://sigopt.com/edu https://sigopt.com/edu [3]: https://sigopt.com/research https://sigopt.com/research
- ddutta 8y agoAnother open source tool for hyperparameter optimization inspired by Vizier https://github.com/kubeflow/katib https://github.com/kubeflow/katib
- yuzhu01052010 8y agoThanks for sharing.