12 ms·
Successful machine learning models: lessons learned at Booking.com
- woliveirajr 7y ago> Model performance is not the same as business performance This is interesting. Sometimes some people from business side consider that AI is the solution to all problems (as if there was just one catch-them-all AI solution) and some academic people think that the top-performance model for some classification task is the must-go, and all they forget that the goal is to earn money.
- ghaff 7y agoThat was an interesting result from the original Netflix challenge. First of all it turned out that the winner wasn't actually all that useful for various reasons such as computational intensity. But, more interestingly, it also turned out that the goals of the model--"best" recommendations--isn't actually the goal of Netflix at all which is much more interested in customer retention and similar metrics. The two things may be correlated but they're certainly not the same thing. I don't remember all the details but I thought it was a really good insight at the time.
- joker3 7y agoThe big problem for Netflix was that their data was all from DVD rentals, but by the end of the contest, their business model was very streaming oriented. As you might imagine, people have beliefs about what they'll want to watch in a few days that don't exactly match up with what they want to watch now. That difference killed the model.
- mv4 7y agoanother problem was that their "user" was actually a household, with a range of (sometimes conflicting) likes and interests - which could result in strange recommendations. They finally added profiles to fix that data problem.
- modeless 7y agoThanks, this makes more sense than any other explanation I've heard for why they didn't use the Netflix prize model in production.
- deng 7y ago> the goal is to earn money Yep, from my experience with booking.com it seems that instead of using highly trained AIs the decision was made to simply slap every dark pattern known to man onto the site and auto-subscribe every customer to a dozen newsletters.
- ignoramous 7y ago> ... they forget that the goal is to earn money. Relevant news.yc discussion from a month ago: https://news.ycombinator.com/item?id=20876158 https://news.ycombinator.com/item?id=20876158
- deleted 7y ago[deleted]
- carapace 7y agoAs an aside, > developing an organisational capability to design, build, and deploy successful machine learned models in user-facing contexts is, in my opinion, as fundamental to an organisation’s competitiveness You hear that, right? In 2019 already you have to have AI and do it well to be competitive. I just wanted to point out how cyberpunk that is.
- minxomat 7y agoThat gave me a flashback to the early 2000s AIML chatbot craze. "What, you have an online store and no customer chatbot? What are you doing?!"
- nerdponx 7y agoFlipside is that this ML (whether you consider it AI or not) really is delivering huge value to the businesses that deploy them.
- otabdeveloper4 7y agoNot really. Today's ML is really good at speech and image recogniziton, which makes for some very eye-popping layman demos. Whereas what businesses really want is time series prediction, and modern ML really sucks balls at solving this problem.
- nerdponx 7y agoDeep learning or machine learning? I agree on the former, and quite strongly disagree on the latter, even if it means redefining ML to be dressed-up statistics.
- otabdeveloper4 7y agoForecasting in 2019 is still using techniques from 1950. You can call that "machine learning", but only if you really want to make people confused by marketing speak.
- anthony_doan 7y agoExperimental design is just a t-test? At least accord to that picture it seems that way. There are no ANOVA or interaction test? Do websites usually just use t-test only? Like adding one feature at a time?
- joker3 7y agoPeople should be using more than one design, but that's not as commonly taught as it should be. I'm going to give a talk about that to my company's ML group in about a month, and hopefully that'll improve things where I am.
- jj12345 7y agoNot sure if you'd be willing, but I'd love a quick rundown of the high level takeaways if you'd be willing to drop them here. Are you talking about more than one experimental design in terms of comparing the exp/control distributions or something else?
- joker3 7y agoI'm just going to present the absolute basics of the topic. It'll be a high-level overview of something along the lines of chapters 1-4 of https://www.amazon.com/Analysis-Experiments-Chapman-Statistical-Science/dp/1439868131/ https://www.amazon.com/Analysis-Experiments-Chapman-Statisti....
- jj12345 7y agoThanks for the reference!
- benmaraschino 7y agoNot the OP, but I work on similar problems, albeit in a different setting (healthcare, millions+ of patients). The gist is that you have to bake experimental design into the deployment of your ML model, but in many cases a simple RCT or A/B test just won't cut it. This is largely because when you deploy a model, no matter how sophisticated or accurate, there's no guarantee that it'll actually move the needle in terms of the outcomes you care about—hence you need to run some kind of trial. At the same time, you want to maximize overall utility by not having to allocate more subjects to your control arm (or harmful, or resource-intensive and ineffective treatment arms) than you need to. This latter point is much more of a problem in medicine than it is in other settings, as you can imagine. These considerations point to adaptive designs that balance exploration/exploitation, e.g. those based on multi-armed bandits. Currently working on some cool (in my opinion) variations of MABs that incorporate domain-specific knowledge, so I could talk about this all day!
- deleted 7y ago[deleted]
- Odenwaelder 7y ago"Content Overload: Accommodations have very rich content, e.g.descriptions, pictures, reviews and ratings." Laughed at that one. Booking.com is so full of dark patterns that I dread using it.
- Bootwizard 7y agoWhat do you mean by "dark patterns"? I'm not familiar with that term
- kennethologist 7y agoDark Patterns are tricks used in websites and apps that make you do things that you didn't mean to, like buying or signing up for something. See more examples here: https://www.darkpatterns.org/ https://www.darkpatterns.org/
- jwc1 7y agoIt's amazing how once you put a label on something, you start noticing it _everywhere_. Someone pointed out "confirmshaming" to me a few years ago...and since then I feel like it shows up on > 50% of the sites I visit. https://www.darkpatterns.org/types-of-dark-pattern/confirmshaming https://www.darkpatterns.org/types-of-dark-pattern/confirmsh...
- cm2012 7y agoHow is confirm shaming a dark pattern? It puts more factual detail near the yes or no options so people know what they're agreeing to or declining.
- lbotos 7y agoIt's emotional manipulation. Confirm shaming uses fear. If I'm cancelling amazon prime because it "costs too much" but you say "are you sure you want to miss out on all the fast shipping" someone who is easily manipulated may continue to subscribe because they are weak willed. You could instead ask: "Why are you cancelling?": - Cost - Don't use it enough - ... - Other (please specify:)
- orasis 7y agoI’m glad they highlighted inference latency. This is a big issue that I’ve started running into at scale.
- romaaeterna 7y agoComing from the math side, I don't really get this. Isn't all of the latency introduced by the learning side of things? Shouldn't the answer side be entirely decoupled from the learning, and simply be plugging data into an equation with a bunch of constant parameters (with values discovered by your learning system, updated at a less than realtime frequency)?
- alexhutcheson 7y agoThe "equation with a bunch of constant parameters" generated by an ML model can be huge, with thousands of inputs or more. Evaluating that equation for a specific observation can require a huge number of computations, which is why there's a boom in ML inference hardware right now.
- romaaeterna 7y agoYeah, I don't buy that. It's still just a matrix multiplication (for the linear bits). Incredibly fast. Besides, the old physics rule of thumb is that any real world equation with a bunch of parameters only has 5-7 that actually matter, and only 3 that matter a lot. Everything else can be set to zero without noticeable change in the result.
- orasis 7y agoIf you’re making decisions that involve multiple variables you may be doing hundreds to thousands of inferences for a single page load. Keeping latency under 50ms becomes a real challenge.
- romaaeterna 7y ago
- djp518 7y ago> Once deployed, beyond the immediate business benefit they often go on to become a foundation for further product development. This is one of the reasons I am a big believer in having a system to track model research and deployment lineage. (I personally use Domino Data Lab for this. I also work for Domino, but use it in my own modeling work and that of others I mentor.) No matter which system you use to track lineage, I've found it important to have a strict history of retraining, versioning, and experimentation. When models are used in downstream systems from the one they were originally intended, it becomes even more critical to able to explain and reproduce the 'research' that led up to deployment.
- sandGorgon 7y ago>>Booking.com go to some lengths to minimise the latency introduced by models, including horizontally scaled distributed copies of models, a in-house developed custom linear prediction engine, favouring models with fewer parameters, batching requests, and pre-computation and/or caching. Any idea what these are ? especially the pre-computation/caching and batching. I'm not able to see what advantage does batching bring...or how you can really cache a prediction request
- Falcorian 7y agoHere is an overly simple example: Pre-compute the recommended hotels for my top users every night. Now when that user comes back, they see a slightly stale recommendation, but it's lightning fast. You can also pre-compute and cache some of the inputs cheer model, like maybe a vector representation of the description of a hotel.
- ovi256 7y agoBatching just uses hardware capabilities efficiently. Either vector instructions in CPUs or batch capabilities in GPUs. For the same hardware load, you can process several samples instead of just one. Pre-computation means running your model on samples in advance, before the model result is needed, so it's ready to use instantly when needed. Caching works probably because there are model results that are reused again and again, so it makes sense to cache them. For example, there are deep models that process room pictures, room and customer characteristics. Only customer characteristics change between customers, so it makes sense to cache the features output by the deep CNN that processes the room pictures. Once you start doing prediction at scale, there are lots of these optimizations to pick up.
- chudi 7y agoIn my company we use this approach, instead of infer online, we just run all our models overnight and save the results on a database that we serve trough an API, that gives you constant time on the latency, its a shotgun approach as much of the recommendations aren't served (specially if are user facing and not item facing ones) but works really well. All our models are balanced using multi-armed bandits, so for our recommendations engine, we run lots of arms that depends on the incoming channel, were in the app is being shown the recommendation, etc and just combine the outputs of the models.
- beefield 7y agoNice to see that I am not only one hating booking.com with a passion. But what really amazes me is the market failure that hotels and other accomodation providers can't come up with a co-op booking site. I am sure there are issues that are difficult to solve from competition point of view, but are they really so difficult to solve that the rent seeking fees of current booking sites are justified?
- kfk 7y agoYes it is. You are maybe not considering the challenge of putting all those accommodations across the globe in one place. Accommodation providers don’t care about how they fill up their rooms as long as they get filled. Booking, contrary to many other tech companies, is very successful financially with 14.5b of revenues and something like 25+% EBIT. I find hard to believe that a coop is going to be able to build such service worldwide.
- DJHenk 7y agoSometimes you see initiatives where the tourist board of a city or a region provides a convenient site for booking a stay. But of course they don't come near the SEO power of Booking.com, so the accommodations also put their rooms there. For the traveler, it is easier to just always use Booking.com. And people like me and you, who really don't want to book at Booking.com and make an effort to book elsewhere, are often in bad luck, because they own a bunch of other booking sites too.
- villux 7y agoAny other good resources on production level deep learning practices?