5 ms·
I always wondered how some companies can rely exclusively on data like A/B testing to make decisions. Isn't that incredibly slow? In my experience there are tho
by blobster 8y ago
I always wondered how some companies can rely exclusively on data like A/B testing to make decisions. Isn't that incredibly slow? In my experience there are thousands of decisions to make every year in a high growth company and there is just not enough time to test everything. This is why the best software companies that we are all familiar with revolve around an individual with incredible intuition/vision. If it was possible/practical to just rely on a bunch of A/B tests, every company would be successful.
- kbyatnal 8y agoNot really. With the amount of users these companies have, you can get statistically valid results very fast (and you can run hundreds of tests in parallel).
- blobster 8y agoMaybe, but I'm talking about getting to that level in the first place.
- nostrademons 8y agoStartups that are still searching for product/market fit don't really do much A/B testing - the user populations are too small for anything statistically relevant. They rely more on intuition, or as some would call it, blind luck. In some ways, what an entrepreneur does is very coarse-grained A/B testing anyways. If you launch 50 different startup ideas and they all get zero traffic, and then you launch the 51st and it has 100 users sign up over a weekend, then there's a good indication that you should go with the 51st idea.
- quanticle 8y agoIt's also what venture capitalists do at the level of companies. Fund a hundred companies, and then watch them. Pour more money into the ones who appear to be succeeding.
- deleted 8y ago[deleted]
- masonic 8y agoThen why is their search such utter crap? When I search on a specific title, I get anything that roughly matches any keywords in combination, then also any title that remotely associates (even conceptually):in any possible way with any individual keyword. But what it won't tell me is "we don't have the title you want". Specific example from earlier: look for the 2007 film "Sunshine". Then, go plug "sunshine film" into duckduckgo and see what comes up as the very first item.
- homero 8y agoThat upsets me too. They hope I'll forget what i wanted and use a recommendation but I'm left confused and leave the app.
- mongol 8y agoLet me guess: It has proved to work well for retention.
- masonic 8y agoProbably not so much toward retention (I'd have quit over this were I not getting it free) as toward redirecting you to content they already control at little or no variable cost.
- groestl 8y agoProbably causes some customers to watch into some of the crappy search result titles, to see if it's the thing they wanted. Hence, the "hours watched" metric goes up.
- shalmanese 8y agoI think the way they do it is very elegant. They surmise the film you want and if they don't have it, they put it in the top bar as "find films related to X". That way, they can push you into consuming something else even if they don't have what you were originally looking for.
- masonic 8y ago
- mehrdada 8y agoI doubt there are companies exclusively rely on A/B tests to make decisions, but even if there were, someone has to think about hypothesis and design the right experiment to then confirm with data. You can't just have a computer running around and make experiments and confirm them for youº. -- º: you kind of can, if you have enough fresh data, to search for and derive statistically significant results automatically.
- blobster 8y agoMaybe it's just me, but I don't think the secret behind this kind of success is just properly designing experiments. Again, if this were true, the best founders would be data scientists. But they're not. They are visionaries. So I reckon it must be the vision.
- morningcovfefe 8y agoVision is obviously necessary - but a big problem folks run into is failing to recognize when something is not working. A/B testing is one way to keep yourself from falling into the trap of thinking 'I know better than my users what they should want'
- blobster 8y agoI agree, but I think this doesn't apply to innovation. When is comes to innovation, people always want faster horses.
- rv-de 8y agothat depends on the validity and accuracy of your test metric.
- TylerE 8y agoAlso a good way to find a local maximum when there are much higher maxima not far removed.
- andrepd 8y ago>Again, if this were true, the best founders would be data scientists. But they're not. They are visionaries. [citation needed]
- revicon 8y agoRemember that Netflix has huge amounts of traffic. They get enough data from a/b tests deployed to only a tiny fraction of their users.