9 ms·

Netflix’s Secret Special Algorithm Is a Human

214 points by donohoe 12y ago

raverbashing 12y ago
As much as people (especially here) think that everything can be solved by Big Data/Machine Learning, human experience and intuition are still very important. No, you can't A/B test your new logo, or the design of your page from scratch. Big Data can't tell you your product sucks. Option A may be better than Option B but this is in the context of both options (and not considering all other possibilities) I see companies firehosing every tiny bit of consumer data hoping to be able to make sense of it all and find something there. Meanwhile they're missing whatever their competitor is doing and what their consumers are liking about it.
- jobposter1234 12y ago
  Those companies are just cargo-culting into the Big Data cult, like everyone else. It doesn't mean they aren't getting useful answers from their black box. Heck, sometimes it's even useful to have a Magic 8 ball make a decision for me.
scott_s 12y ago
In other realms, we don't call this data but intelligence. Humans will still make the decisions, but in more and more places, the decisions are more likely to be informed by data (intelligence). Consider that Netflix's data is tiny compared to the amount of data that any government must sift through. Algorithms don't make government decisions, people do. But they (hopefully) base those decisions based on intelligence (some of) which was gathered and filtered by algorithms, then synthesized by humans.
- SquidMagnet 12y ago
  I am not entirely sure what you are saying here, but the way I have always understood it is: data is knowledge; the ability to apply knowledge is intelligence. Netflix has a considerable amount of data (knowledge) and its algorithms exemplify some efforts to apply that knowledge (intelligence). As it stands presently, though, humans still tend to be more intelligent than any algorithms we have created. (Generally speaking, of course.) I think we are trying to say the same things here, right?
  - muaddirac 12y ago
    I assumed they were using the other meaning of intelligence, as in "signal intelligence"
  - scott_s 12y ago
    I'm trying to reframe what Netflix does as something that is already done - we just tend to use different names for it in those other domains. What's new is that we're doing this old thing - processing massive amounts of data, extracting the relevant facts, synthesizing those facts into a coherent story, and presenting that story to decision makers - in new contexts. In the Netflix context, what is mostly called "data" is called "intelligence" in, say, government decision making.
- digisth 12y ago
  This is another facet of the "amplified-teams" trend that's been happening over the past few years. This book review of 'Average is over' has some good information about it: http://ieet.org/index.php/IEET/more/searle20150109 http://ieet.org/index.php/IEET/more/searle20150109 "In his vision intelligent machines will revolutionize everything from medicine to education to business management and negotiation to love. The human beings who will best thrive in this new environment will be those whose work best complements that of intelligent machines, and this will be the case all the way from the factory floor to the classroom." Very interesting times ahead.
  - scott_s 12y ago
    Perhaps another way of stating that is: The people who will best thrive are those whose work cannot be automated. Which, in some ways, has always been true.
- tel 12y ago
  In Health Informatics at least this is taken even further Data X of Y people have some disease Information Based on data, I can predict disease likelihood given some environmental and personal factors of a patient Knowledge Using informed predictions, I make good inferences about how to proceed with diagnosis Wisdom Using knowledge and experience I choose the right approach for treating and diagnosing a patient which is efficacious, healthy, and works with the patient's actual needs It's easy to draw these lines in other places or to call the tower a lot of woo woo able to be reduced into inferences atop raw data all combined correctly... but it serves to remind just how difficult it is to combine the right data in the right way to make the right decisions at the right times. It also serves as a sharp counterpoint to the idea of, say, machine learning patient diagnoses. It turns out that diagnostic accuracy is terrible, but not because people are directly bad at it (even if they are) but instead because knowledge/wisdom dictates that perfect accuracy isn't that valuable---perfect care is and that can involve chasing down treatment and care avenues that would never be predicted or acting on information that is not currently in your model.
ljd 12y ago
I see this argument akin to something along the lines of, "A website is really a human because someone needs to maintain it and make decisions on it's appearance." You are describing a tool. People don't say hammers are humans, either. Netflix's special algorithm is indeed an algorithm.
- abandonliberty 12y ago
  I was really hoping Netflix had a secret warehouse filled with hundreds of workers whose job was to provide recommendations based on what you like, a la 'invisible boyfriend' [1] Sadly, they just have people on the quest for the perfect hammer. [1]http://www.washingtonpost.com/news/the-intersect/wp/2015/01/22/i-paid-25-for-an-invisible-boyfriend-and-i-think-i-might-be-in-love/ http://www.washingtonpost.com/news/the-intersect/wp/2015/01/...
stillsut 12y ago
I see how netflix's dataset helps them produce an addictive serialized drama like House of Cards. There are many examples to learn from, and the shows are basically built out of a formula. What I don't see is how this would help you make a critically well received documentary about Nina Simone. Sure, you might be able to predict a lot of people will stream it, but how does that make it well received at Cannes? Nothing in their dataset is telling them about the art of film-making.
- Bluestrike2 12y ago
  No, but their data allows filmmakers to gain a better idea of what the consequences of certain decisions are likely to be. It's not so much as allowing algorithms to dictate creative decisions as it is enabling those decisions to be better informed. As for targeting film festivals in particular, I've no doubt that Netflix has sufficient data to look at sort of festival-goers (and those like them) in particular. While it can't provide a guaranteed formula for festival success, it should still give a filmmaker another tool that helps them do their work.
- smacktoward 12y ago
  The argument of the article is that Netflix's dataset doesn't really help them make decisions like what the content of their next documentary should be. It argues that what happens instead is that Netflix's programming honcho, Ted Sarandos, picks projects based on the person who's putting them together, looking for creators with a solid creative track record and an existing cult following: > I began to sense that their biggest bets always seemed ultimately driven by faith in a particular cult creator, like David Fincher (“House of Cards”), Jenji Leslie Kohan (“Orange is the New Black”), Ricky Gervais (“Derek”), John Fusco (“Marco Polo”), or Mitchell Hurwitz (“Arrested Development”)... I do think that there is a sophisticated algorithm at work here—but I think his name is Ted Sarandos. So take the case of the Nina Simone doc. It was directed by Liz Garbus (http://en.wikipedia.org/wiki/Liz_Garbus http://en.wikipedia.org/wiki/Liz_Garbus), whose previous films have been Sundance- and Oscar- fare for a few years now. So it could just be that Sarandos looked at her track record and decided that Garbus was an up-and-coming talent, and that mattered more than the specific content of whatever her next project was going to be. He was just betting that a Liz Garbus documentary was going to be great, no matter what story it told.
JeremyMorgan 12y ago
Hollywood Video's corporate office had a guy, one guy whose job it was to just watch movies. He'd watch them, make sure it was categorized properly, then create associations with other movies that "you might like" if you like this movie, and vice versa. All kept in a spreadsheet that we later plugged in. We also thought there should be an algorithm, but he was pretty dang good at what he did.
- boomzilla 12y ago
  It's simple ROI. There is not that many movies this is a task that one doesn't need to scale with computers. Hiring a guy is likely much cheaper than investing on the software. Plus I am sure there are plenty of people who would like to watch movies for their day jobs. On the other hand, I am pretty sure there is at least some degree of automated video classification at Youtube.
  - RogerL 12y ago
    I think it does need to scale. What happens when after a bus event? Someone else has to watch every film in existence? It is not enough to just watch the new films - you have to have a memory of all other films in order to make that association.
    - Crito 12y ago
      Seems reasonable to me that you could have a team of categorizers/critics. That'd be several times more expensive of course, depending on how many people you added, but I think it also might improve the results if the team was picked well. Finding associations between movies is probably something a group of people can do more effectively than one person, since recall will be better. There might be an issue of disagreements within the team, but I think at least finding associations between movies would tend to be fairly non-controversial. We might disagree over whether or not The Italian Job is a good movie, but we probably both agree that it is a heist movie.
    - zeeshanm 12y ago
      What about aggregating movie reviews and classifying movies based on their reviews. Would be an interesting problem regardless of accuracy.
minthd 12y ago
This article looks at single point in the industry with regards to automation. But i believe the important point is whether the addition of data provides more efficient movie production, with less failure , and by how much.No need to automate a movie you don't produce.
dtjones 12y ago
>Even Google, the champion of algorithms, employs substantial human adjustments to make its search engines perform just right. The author doesn't have a solid grasp on machine learning. The 'human adjustments' provide feedback to the algorithm, which the algorithm then uses to update and improve performance. His tone implies its a bad thing to use human feedback.
- mostly_harmless 12y ago
  Ideally, it would be a completely automated system. You don't need to understand machine learning to know that it would be better for the computer to do everything itself without human intervention
  - dtjones 12y ago
    It is completely automated. It automatically incorporates human/user feedback into ranking.
    - unknownian 12y ago
      Even the startingly accurate Chrome predictions?
      - DannyBee 12y ago
        yes
  - DannyBee 12y ago
    ?? It doesn't require human intervention. Simplified: Machine learning algorithm constructs page to show human. Humans click on this or that on page. Click data is fed into algorithm. Algorithm uses this data decide how to show better page to humans You are taking human input. It is still completely automated, it requires no human intervention.
  - CHY872 12y ago
    How so? If the cost of making the computer able to do everything itself is greater than the cost of an equivalent system that takes human input, then you should use the human input version every time.
j2kun 12y ago
> featuring the television star Melissa Raunch Whoa there! It's Rauch. Kind of a Freudian slip there, considering the actress is being named in relation to a sex scene.
- ScottBurson 12y ago
  Here's another: > Television studios have Nielson ratings It's "Nielsen".
  - j2kun 12y ago
    Looks like they've fixed it. But seriously don't they have editors reading this stuff before it goes live?
zkhalique 12y ago
In other news, Pandora is built on human input, among other things. The Music Genome Project.
ctdonath 12y ago
Drive such a thing thru Amazon Turk perhaps? Might just be the "killer app" for Turks: get paid to watch movies!
- sireat 12y ago
  MTurk has already been used to classify and tag videos. I spent a month as Turker (as an experiment) a few years ago. The best paying HITs were provided by a porn company to classify and tag their videos. You were given a sequence of images from the video presumably to save bandwith. I even debated scripting a helper tool which would take a few keystrokes (for example dvda) and generate a suitable description.
domdip 12y ago
What a weird strawman article. Did the author really think Netflix produced television shows in the same way that Siri suggests gas stations? Any blockbuster-finding algorithm (or any other component of this decision) is going to have false positives and require human intervention.
wallflower 12y ago
See also Shazam's Secret > The hunt keeps Mr. Slomovitz on his toes. Every morning, he skims dozens of music blogs, checking for new releases he might have missed, as well as the iTunes, Amazon.com and Billboard charts, and blog aggregators like the Hype Machine. Most weeks he also goes to local record stores to see if there is something in stock he has not heard of, or if older albums are being remastered or reissued. And he listens to local radio stations, especially near universities. "Shazam’s Search for Songs Creates New Music Jobs" http://www.nytimes.com/2011/02/14/technology/14shazam.html http://www.nytimes.com/2011/02/14/technology/14shazam.html
- skrebbel 12y ago
  Shazam actually has 2000 music geeks in a call center who listen to tiny audio fragments and guess which song it is. The algorithm just makes sure the rock geek gets the rock songs, and so on.
  - afro88 12y ago
    Really? I'm genuinely interested in this. Do you have an article or interview or something with more info?
hotgoldminer 12y ago
Dao Nguyen @Buzzfeed, "Data should not dictate your strategy," Nguyen says, "But you should understand what data tells you and also what its limits are."[0]. Heard this a few weeks back and still resonating with me. 0 -http://www.marketplace.org/topics/tech/buzzfeed-wizard-who-changed-media-we-know-it http://www.marketplace.org/topics/tech/buzzfeed-wizard-who-c...
karmacondon 12y ago
The lesson here isn't "Human Judgement > Algorithmic Judgement". It also isn't "Humans are good at some things and computers are good at others". It's that there isn't a good reason to make the investment in solving this particular problem algorithmically (yet). Algorithms are great when you need scale, especially in situations where a 10% improvement in prediction accuracy can make a big improvement in the bottom line. Netflix and other studios might greenlight several new shows in a year, out of dozens that receive consideration. And the Pareto Distribution is in full effect. Most of the profits and awards come from one or two big hits. Algorithmic decision making just doesn't make a lot of sense in situations with a small sample size and uneven reward structure. It doesn't mean that it isn't possible, though. If someone were to make the massive investment necessary to do a more thorough analysis of the content creators, the actors, the scripts and potential audiences and all of the other possible inputs then algorithms could probably do as good a job as humans, if not better. Netflix and others have only taken baby steps in this direction, working with data that is readily available and using predictive techniques that are well tested and understood. Given the nature of the problem, it doesn't make sense for them to approach it any other way at this time. But when it comes to making billions of recommendations to millions of people per day, they still rely heavily on data and algorithmic prediction. There's a time and place for everything. The time and place for algorithms in our daily lives is changing and expanding, but very slowly.
rurban 12y ago
I also rather trust the algorithm to see what is a good movie and what is not. In my case the collected ratings of the experts, not the users. Current example: http://cannes-rurban.rhcloud.com/Sundance2015 http://cannes-rurban.rhcloud.com/Sundance2015 There are some outliers, typical "festival hits", which only relate to festival specialists, but they are easily detectable, with a "10%" human bullshit detector. Like last years Godard at Cannes, 2012 Leos Carax, 2011 the Kaurismaki and 2010 both the Godard and the Jury winner Weerasethakul. Those outliers are even statistically detectable. One thing is clear, you can trust the collected experts more than the juries. So I can fully confirm the story.
- cpeterso 12y ago
  Rotten Tomatoes' rating, mentioned in the article as scoring THE BRONZE as 10% (though the RT now says 18%), is also a collection of professional reviewers' ratings.
  - rurban 12y ago
    Exactly. Almost everybody agrees that The Bronze is shit, you can trust the collected experts in big numbers. But you also need the standard deviation to filter "special audience" flicks, like festival masturbation, horror or erotic movies, which are targeted to a very narrow audience.
- abruzzi 12y ago
  The problem I've found is the algorithms for detecting taste seem to struggle greatly the further ones taste deviates from the norm. I find Netflix's recommendations to be fundamentally useless and is always reccomending things in which I have zero interest. But I've also found Rotten Tomatoes to be borderline useless as well (the second worst film I've seen in the last decade--Snowpiercer--has a very high RT score...go figure.)
  - keypusher 12y ago
    Do you actually rate the things you watch on Netflix? Or go back and rate a bunch of popular movies you loved and hated? Because if you don't, you will just keep getting the same mainstream recommendations.
otakucode 12y ago
It always amazes me how the people in the content creation industries keep their jobs. Even though comprehensive studies have shown that the public responds randomly to entertainment products, they still skate by claiming to 'know what the public wants.' Analysis of the decisions of past executives at movie companies showed that executives who were said to be "on a hot streak" were mostly benefitting from the results of projects started by their predecessors, and then when their "streak" ended and they left, their replacement got the same benefit from the projects they left in the pipeline. The book 'A Drunkard's Walk' analyses various different industries and situations and shows where randomness shows up. The success of media is one of the strongest ones. No factor correllates with success. Not budget, star-power, genre, directors, nothing. For every runaway hit there are exactly as many abyssmal failures. Before Titanic hit theaters, film critics and industry insiders were dead certain that it would prove to be the most monumental theatrical failure in history, making Kevin Costner's Waterworld look like a walk in the park. Of course, they were wrong. They will always be wrong as often as they are correct. Their entire careers are built on, quite literally, absolutely nothing. These executives choose what gets produced, and they are incompetent at it. Yet they are paid millions of dollars. It's astonishing that they get so far with unmitigated bullshit. This is why large movie production houses will die. Their performance has always been random in terms of producing successful content. But they always made up for it by having total control of the distribution. Now, distribution is worthless. Any 12 year old with an Internet connection can distribute media better than large corporations can. Take away that control and profit from distribution, and those companies will end up simply fading into bankruptcy after enough failed projects pile up.
- marketforlemmas 12y ago
  You might also be interested in Duncan Watts' book "Everything is Obvious", which explores this tendency we have to assume that observed success equates to some skill. Its a nice read.
- roasm 12y ago
  Large movie production houses won't die in the near future, because predicting a financial winner isn't as hard as you make it sound. We currently have a movie economy that is driving the big producers to make big budget comic book sequels that do well internationally (generally based on the universal appeal of liking suspense and action). They've boxed themselves into a very profitable corner of funding high budget movies that independent producers can't fund and then profiting internationally. They make a movie for $200M then make $300M+ in sales. I do agree that more independent (but still highly funded) producers are rolling dice, then manufacturing winners with their marketing power.
redRobo 12y ago
The title is completely misleading, seems like it's talking to the recommendation feature on Netflix, but it isn't. Everyone with common sense knows that big executive decisions aren't made solely by computers, yet the author seems amazed that big executive decisions are ultimately made by big executives. Talk about clickbait, huh?

214 points by donohoe 12y ago

As much as people (especially here) think that everything can be solved by Big Data/Machine Learning, human experience and intuition are still very important. No, you can't A/B test your new logo, or the design of your page from scratch. Big Data can't tell you your product sucks. Option A may be better than Option B but this is in the context of both options (and not considering all other possibilities) I see companies firehosing every tiny bit of consumer data hoping to be able to make sense of it all and find something there. Meanwhile they're missing whatever their competitor is doing and what their consumers are liking about it.

jobposter1234 12y ago
Those companies are just cargo-culting into the Big Data cult, like everyone else. It doesn't mean they aren't getting useful answers from their black box. Heck, sometimes it's even useful to have a Magic 8 ball make a decision for me.

scott_s 12y ago

In other realms, we don't call this data but intelligence. Humans will still make the decisions, but in more and more places, the decisions are more likely to be informed by data (intelligence). Consider that Netflix's data is tiny compared to the amount of data that any government must sift through. Algorithms don't make government decisions, people do. But they (hopefully) base those decisions based on intelligence (some of) which was gathered and filtered by algorithms, then synthesized by humans.

SquidMagnet 12y ago
I am not entirely sure what you are saying here, but the way I have always understood it is: data is knowledge; the ability to apply knowledge is intelligence. Netflix has a considerable amount of data (knowledge) and its algorithms exemplify some efforts to apply that knowledge (intelligence). As it stands presently, though, humans still tend to be more intelligent than any algorithms we have created. (Generally speaking, of course.) I think we are trying to say the same things here, right?
- muaddirac 12y ago
  I assumed they were using the other meaning of intelligence, as in "signal intelligence"
- scott_s 12y ago
  I'm trying to reframe what Netflix does as something that is already done - we just tend to use different names for it in those other domains. What's new is that we're doing this old thing - processing massive amounts of data, extracting the relevant facts, synthesizing those facts into a coherent story, and presenting that story to decision makers - in new contexts. In the Netflix context, what is mostly called "data" is called "intelligence" in, say, government decision making.
digisth 12y ago
This is another facet of the "amplified-teams" trend that's been happening over the past few years. This book review of 'Average is over' has some good information about it: http://ieet.org/index.php/IEET/more/searle20150109 http://ieet.org/index.php/IEET/more/searle20150109 "In his vision intelligent machines will revolutionize everything from medicine to education to business management and negotiation to love. The human beings who will best thrive in this new environment will be those whose work best complements that of intelligent machines, and this will be the case all the way from the factory floor to the classroom." Very interesting times ahead.
- scott_s 12y ago
  Perhaps another way of stating that is: The people who will best thrive are those whose work cannot be automated. Which, in some ways, has always been true.
tel 12y ago
In Health Informatics at least this is taken even further Data X of Y people have some disease Information Based on data, I can predict disease likelihood given some environmental and personal factors of a patient Knowledge Using informed predictions, I make good inferences about how to proceed with diagnosis Wisdom Using knowledge and experience I choose the right approach for treating and diagnosing a patient which is efficacious, healthy, and works with the patient's actual needs It's easy to draw these lines in other places or to call the tower a lot of woo woo able to be reduced into inferences atop raw data all combined correctly... but it serves to remind just how difficult it is to combine the right data in the right way to make the right decisions at the right times. It also serves as a sharp counterpoint to the idea of, say, machine learning patient diagnoses. It turns out that diagnostic accuracy is terrible, but not because people are directly bad at it (even if they are) but instead because knowledge/wisdom dictates that perfect accuracy isn't that valuable---perfect care is and that can involve chasing down treatment and care avenues that would never be predicted or acting on information that is not currently in your model.

ljd 12y ago

I see this argument akin to something along the lines of, "A website is really a human because someone needs to maintain it and make decisions on it's appearance." You are describing a tool. People don't say hammers are humans, either. Netflix's special algorithm is indeed an algorithm.

abandonliberty 12y ago
I was really hoping Netflix had a secret warehouse filled with hundreds of workers whose job was to provide recommendations based on what you like, a la 'invisible boyfriend' [1] Sadly, they just have people on the quest for the perfect hammer. [1]http://www.washingtonpost.com/news/the-intersect/wp/2015/01/22/i-paid-25-for-an-invisible-boyfriend-and-i-think-i-might-be-in-love/ http://www.washingtonpost.com/news/the-intersect/wp/2015/01/...

stillsut 12y ago

I see how netflix's dataset helps them produce an addictive serialized drama like House of Cards. There are many examples to learn from, and the shows are basically built out of a formula. What I don't see is how this would help you make a critically well received documentary about Nina Simone. Sure, you might be able to predict a lot of people will stream it, but how does that make it well received at Cannes? Nothing in their dataset is telling them about the art of film-making.

Bluestrike2 12y ago
No, but their data allows filmmakers to gain a better idea of what the consequences of certain decisions are likely to be. It's not so much as allowing algorithms to dictate creative decisions as it is enabling those decisions to be better informed. As for targeting film festivals in particular, I've no doubt that Netflix has sufficient data to look at sort of festival-goers (and those like them) in particular. While it can't provide a guaranteed formula for festival success, it should still give a filmmaker another tool that helps them do their work.
smacktoward 12y ago
The argument of the article is that Netflix's dataset doesn't really help them make decisions like what the content of their next documentary should be. It argues that what happens instead is that Netflix's programming honcho, Ted Sarandos, picks projects based on the person who's putting them together, looking for creators with a solid creative track record and an existing cult following: > I began to sense that their biggest bets always seemed ultimately driven by faith in a particular cult creator, like David Fincher (“House of Cards”), Jenji Leslie Kohan (“Orange is the New Black”), Ricky Gervais (“Derek”), John Fusco (“Marco Polo”), or Mitchell Hurwitz (“Arrested Development”)... I do think that there is a sophisticated algorithm at work here—but I think his name is Ted Sarandos. So take the case of the Nina Simone doc. It was directed by Liz Garbus (http://en.wikipedia.org/wiki/Liz_Garbus http://en.wikipedia.org/wiki/Liz_Garbus), whose previous films have been Sundance- and Oscar- fare for a few years now. So it could just be that Sarandos looked at her track record and decided that Garbus was an up-and-coming talent, and that mattered more than the specific content of whatever her next project was going to be. He was just betting that a Liz Garbus documentary was going to be great, no matter what story it told.

JeremyMorgan 12y ago

Hollywood Video's corporate office had a guy, one guy whose job it was to just watch movies. He'd watch them, make sure it was categorized properly, then create associations with other movies that "you might like" if you like this movie, and vice versa. All kept in a spreadsheet that we later plugged in. We also thought there should be an algorithm, but he was pretty dang good at what he did.

boomzilla 12y ago
It's simple ROI. There is not that many movies this is a task that one doesn't need to scale with computers. Hiring a guy is likely much cheaper than investing on the software. Plus I am sure there are plenty of people who would like to watch movies for their day jobs. On the other hand, I am pretty sure there is at least some degree of automated video classification at Youtube.
- RogerL 12y ago
  I think it does need to scale. What happens when after a bus event? Someone else has to watch every film in existence? It is not enough to just watch the new films - you have to have a memory of all other films in order to make that association.
  - Crito 12y ago
    Seems reasonable to me that you could have a team of categorizers/critics. That'd be several times more expensive of course, depending on how many people you added, but I think it also might improve the results if the team was picked well. Finding associations between movies is probably something a group of people can do more effectively than one person, since recall will be better. There might be an issue of disagreements within the team, but I think at least finding associations between movies would tend to be fairly non-controversial. We might disagree over whether or not The Italian Job is a good movie, but we probably both agree that it is a heist movie.
  - zeeshanm 12y ago
    What about aggregating movie reviews and classifying movies based on their reviews. Would be an interesting problem regardless of accuracy.

minthd 12y ago

This article looks at single point in the industry with regards to automation. But i believe the important point is whether the addition of data provides more efficient movie production, with less failure , and by how much.No need to automate a movie you don't produce.

dtjones 12y ago

>Even Google, the champion of algorithms, employs substantial human adjustments to make its search engines perform just right. The author doesn't have a solid grasp on machine learning. The 'human adjustments' provide feedback to the algorithm, which the algorithm then uses to update and improve performance. His tone implies its a bad thing to use human feedback.

mostly_harmless 12y ago
Ideally, it would be a completely automated system. You don't need to understand machine learning to know that it would be better for the computer to do everything itself without human intervention
- dtjones 12y ago
  It is completely automated. It automatically incorporates human/user feedback into ranking.
  - unknownian 12y ago
    Even the startingly accurate Chrome predictions?
    - DannyBee 12y ago
      yes
- DannyBee 12y ago
  ?? It doesn't require human intervention. Simplified: Machine learning algorithm constructs page to show human. Humans click on this or that on page. Click data is fed into algorithm. Algorithm uses this data decide how to show better page to humans You are taking human input. It is still completely automated, it requires no human intervention.
- CHY872 12y ago
  How so? If the cost of making the computer able to do everything itself is greater than the cost of an equivalent system that takes human input, then you should use the human input version every time.

j2kun 12y ago

> featuring the television star Melissa Raunch Whoa there! It's Rauch. Kind of a Freudian slip there, considering the actress is being named in relation to a sex scene.

ScottBurson 12y ago
Here's another: > Television studios have Nielson ratings It's "Nielsen".
- j2kun 12y ago
  Looks like they've fixed it. But seriously don't they have editors reading this stuff before it goes live?

zkhalique 12y ago

In other news, Pandora is built on human input, among other things. The Music Genome Project.

ctdonath 12y ago

Drive such a thing thru Amazon Turk perhaps? Might just be the "killer app" for Turks: get paid to watch movies!

sireat 12y ago
MTurk has already been used to classify and tag videos. I spent a month as Turker (as an experiment) a few years ago. The best paying HITs were provided by a porn company to classify and tag their videos. You were given a sequence of images from the video presumably to save bandwith. I even debated scripting a helper tool which would take a few keystrokes (for example dvda) and generate a suitable description.

domdip 12y ago

What a weird strawman article. Did the author really think Netflix produced television shows in the same way that Siri suggests gas stations? Any blockbuster-finding algorithm (or any other component of this decision) is going to have false positives and require human intervention.

wallflower 12y ago

See also Shazam's Secret > The hunt keeps Mr. Slomovitz on his toes. Every morning, he skims dozens of music blogs, checking for new releases he might have missed, as well as the iTunes, Amazon.com and Billboard charts, and blog aggregators like the Hype Machine. Most weeks he also goes to local record stores to see if there is something in stock he has not heard of, or if older albums are being remastered or reissued. And he listens to local radio stations, especially near universities. "Shazam’s Search for Songs Creates New Music Jobs" http://www.nytimes.com/2011/02/14/technology/14shazam.html http://www.nytimes.com/2011/02/14/technology/14shazam.html

skrebbel 12y ago
Shazam actually has 2000 music geeks in a call center who listen to tiny audio fragments and guess which song it is. The algorithm just makes sure the rock geek gets the rock songs, and so on.
- afro88 12y ago
  Really? I'm genuinely interested in this. Do you have an article or interview or something with more info?

hotgoldminer 12y ago

Dao Nguyen @Buzzfeed, "Data should not dictate your strategy," Nguyen says, "But you should understand what data tells you and also what its limits are."[0]. Heard this a few weeks back and still resonating with me. 0 -http://www.marketplace.org/topics/tech/buzzfeed-wizard-who-changed-media-we-know-it http://www.marketplace.org/topics/tech/buzzfeed-wizard-who-c...

karmacondon 12y ago

The lesson here isn't "Human Judgement > Algorithmic Judgement". It also isn't "Humans are good at some things and computers are good at others". It's that there isn't a good reason to make the investment in solving this particular problem algorithmically (yet). Algorithms are great when you need scale, especially in situations where a 10% improvement in prediction accuracy can make a big improvement in the bottom line. Netflix and other studios might greenlight several new shows in a year, out of dozens that receive consideration. And the Pareto Distribution is in full effect. Most of the profits and awards come from one or two big hits. Algorithmic decision making just doesn't make a lot of sense in situations with a small sample size and uneven reward structure. It doesn't mean that it isn't possible, though. If someone were to make the massive investment necessary to do a more thorough analysis of the content creators, the actors, the scripts and potential audiences and all of the other possible inputs then algorithms could probably do as good a job as humans, if not better. Netflix and others have only taken baby steps in this direction, working with data that is readily available and using predictive techniques that are well tested and understood. Given the nature of the problem, it doesn't make sense for them to approach it any other way at this time. But when it comes to making billions of recommendations to millions of people per day, they still rely heavily on data and algorithmic prediction. There's a time and place for everything. The time and place for algorithms in our daily lives is changing and expanding, but very slowly.

rurban 12y ago

I also rather trust the algorithm to see what is a good movie and what is not. In my case the collected ratings of the experts, not the users. Current example: http://cannes-rurban.rhcloud.com/Sundance2015 http://cannes-rurban.rhcloud.com/Sundance2015 There are some outliers, typical "festival hits", which only relate to festival specialists, but they are easily detectable, with a "10%" human bullshit detector. Like last years Godard at Cannes, 2012 Leos Carax, 2011 the Kaurismaki and 2010 both the Godard and the Jury winner Weerasethakul. Those outliers are even statistically detectable. One thing is clear, you can trust the collected experts more than the juries. So I can fully confirm the story.

cpeterso 12y ago
Rotten Tomatoes' rating, mentioned in the article as scoring THE BRONZE as 10% (though the RT now says 18%), is also a collection of professional reviewers' ratings.
- rurban 12y ago
  Exactly. Almost everybody agrees that The Bronze is shit, you can trust the collected experts in big numbers. But you also need the standard deviation to filter "special audience" flicks, like festival masturbation, horror or erotic movies, which are targeted to a very narrow audience.
abruzzi 12y ago
The problem I've found is the algorithms for detecting taste seem to struggle greatly the further ones taste deviates from the norm. I find Netflix's recommendations to be fundamentally useless and is always reccomending things in which I have zero interest. But I've also found Rotten Tomatoes to be borderline useless as well (the second worst film I've seen in the last decade--Snowpiercer--has a very high RT score...go figure.)
- keypusher 12y ago
  Do you actually rate the things you watch on Netflix? Or go back and rate a bunch of popular movies you loved and hated? Because if you don't, you will just keep getting the same mainstream recommendations.

otakucode 12y ago

It always amazes me how the people in the content creation industries keep their jobs. Even though comprehensive studies have shown that the public responds randomly to entertainment products, they still skate by claiming to 'know what the public wants.' Analysis of the decisions of past executives at movie companies showed that executives who were said to be "on a hot streak" were mostly benefitting from the results of projects started by their predecessors, and then when their "streak" ended and they left, their replacement got the same benefit from the projects they left in the pipeline. The book 'A Drunkard's Walk' analyses various different industries and situations and shows where randomness shows up. The success of media is one of the strongest ones. No factor correllates with success. Not budget, star-power, genre, directors, nothing. For every runaway hit there are exactly as many abyssmal failures. Before Titanic hit theaters, film critics and industry insiders were dead certain that it would prove to be the most monumental theatrical failure in history, making Kevin Costner's Waterworld look like a walk in the park. Of course, they were wrong. They will always be wrong as often as they are correct. Their entire careers are built on, quite literally, absolutely nothing. These executives choose what gets produced, and they are incompetent at it. Yet they are paid millions of dollars. It's astonishing that they get so far with unmitigated bullshit. This is why large movie production houses will die. Their performance has always been random in terms of producing successful content. But they always made up for it by having total control of the distribution. Now, distribution is worthless. Any 12 year old with an Internet connection can distribute media better than large corporations can. Take away that control and profit from distribution, and those companies will end up simply fading into bankruptcy after enough failed projects pile up.

marketforlemmas 12y ago
You might also be interested in Duncan Watts' book "Everything is Obvious", which explores this tendency we have to assume that observed success equates to some skill. Its a nice read.
roasm 12y ago
Large movie production houses won't die in the near future, because predicting a financial winner isn't as hard as you make it sound. We currently have a movie economy that is driving the big producers to make big budget comic book sequels that do well internationally (generally based on the universal appeal of liking suspense and action). They've boxed themselves into a very profitable corner of funding high budget movies that independent producers can't fund and then profiting internationally. They make a movie for $200M then make $300M+ in sales. I do agree that more independent (but still highly funded) producers are rolling dice, then manufacturing winners with their marketing power.

redRobo 12y ago

The title is completely misleading, seems like it's talking to the recommendation feature on Netflix, but it isn't. Everyone with common sense knows that big executive decisions aren't made solely by computers, yet the author seems amazed that big executive decisions are ultimately made by big executives. Talk about clickbait, huh?