5 ms·

> Sometimes it works. Sometimes it’s critical. But sometimes it fails, or results in unintended consequences that we may not notice for years. > Data-driven jo

by baddox 12y ago

> Sometimes it works. Sometimes it’s critical. But sometimes it fails, or results in unintended consequences that we may not notice for years. > Data-driven journalism gave us Buzzfeed > Data-driven music gave us X-Factor and Pop Idol > Data-driven movies gave us 25 Hollywood sequels planned for this year > Data-driven education gave us Key Performance Indicators and Teaching to the test These first three examples are awful examples. A ton of people love all of those things. The only failure of data-drivenness here is the failure to generate content that the author wants. Now, the author can attempt to make some argument about how websites, TV shows, and movies have a moral obligation to strive for whatever objectives the author prefers, but that's a separate issue to settle. The fourth example is a little different, because we're talking about mandatory education programs for children, rather than products that people choose to pay for or consume. Also, I don't think that data-drivenness itself is a significant contributor to those problems in education.

api 12y ago
The failure is the failure to generate novelty or adequately explore alternatives. Enough people might reliable go see 25 sequels, but none of those films will be memorable. None will advance the art of film-making. None will change anyone's life or mentality or affect the culture in any meaningful way. Being data driven means chasing the biggest, loudest signal in your data set. It means pandering to that signal, because it swamps all others. A data-driven approach is not going to lead you anywhere new. In machine learning / AI we refer to such algorithms as "greedy." A classic example would be simple hill climbing a.k.a. gradient descent. These algorithms are known to be very good at optimizing within the bounds of a simple well-behaved and regular fitness landscape, but they readily become stuck at local maxima when presented with any solution space with any complex structure. We're living in what I'm tempted to call the dark age of the local maximum, the age of gradient descent.
- baddox 12y ago
  > The failure is the failure to generate novelty or adequately explore alternatives. Perhaps you value novelty more than the average person. What might not be "adequate" for you might very well be adequate for a huge portion of people. Throughout this comment you mention several potential goals of content producers (being memorable, advancing the art, changing someone's life, etc.), but you make no argument for why those goals ought to be prioritized over other goals, other than the implication that you personally prefer those goals.
  - alansammarone 12y ago
    It's seems clear that any content that advances the art is better than a content doesn't advance the art. I guess that what he meant is that global maxima is better than local maxima (which is clearly true), and data-driven vision is a hill climbing strategy, thus locking you into a local maxima.
    - baddox 12y ago
      > It's seems clear that any content that advances the art is better than a content doesn't advance the art. Perhaps that's true with all else being equal, but clearly all else isn't equal. > I guess that what he meant is that global maxima is better than local maxima (which is clearly true), and data-driven vision is a hill climbing strategy, thus locking you into a local maxima. My issue is that none of those examples are backed by any evidence that they are not doing a decent job of finding a global maximum.
      - TheOtherHobbes 12y ago
        It's true in the sense that what makes technology and culture interesting is invention, not endless recycling. You're pretty much suggesting that using strong feedback to force culture to stay within a tiny area of the total possible cultural phase space is just as interesting as allowing chaotic exploration of the entire space. It's not just an argument against creativity, it's an argument against invention in general. >My issue is that none of those examples are backed by any evidence that they are not doing a decent job of finding a global maximum. That's the thing about global maxima - you only know that you've found a global maximum if you've explored the entire space. Otherwise you've just stumbled across a local attractor, and you're stuck in a loop around it. This isn't even a good analogy, because cultural attractors are contingent, and they vary over time. They're also unpredictable. The reason they capture attention isn't because they're maxima in some analytic sense. They become found maxima because they summarise some aspect of human experience, so they appeal to a lot of people at once. The spectrum of possible maxima is mysterious and not understood, which is how you get - say - Harry Potter coming out of nowhere and captivating a generation. Culture is music, not a sine wave. You don't just want a single signal - you want a mix of related-but-different signals running all the time.
- HaloZero 12y ago
  You can make exceptions to all those rules though. Empire Strikes Back is probably the most obvious example of a sequel that was far and beyond the first and changed cinema. If you're talking about money-making sequels than Toy Story 2 and 3 were memorable and interesting.
- Double_Cast 12y ago
  > Being data driven means chasing the biggest, loudest signal in your data set. This reminds me of how one time, a professor of mine was discussing a pie-chart. He pointed towards the smallest sliver and said "Know what that is? That's opportunity!" He was most interested in the smallest signal because it represented untapped potential. (IIRC the topic was related to replacing oil with renewable energy.)
- karmacondon 12y ago
  Feh. What you're saying is only true if 100% of decisions are data driven. Just because Hollywood has chosen to make 25 sequels doesn't mean that Sundance no longer exists. It, and countless events and organizations like it, are celebrated inside and outside of Hollywood. The implications of the hypothetical extension of this trend are clear to a lot of people. As long as some percent of people are pushing the boundary, culture will be fine. Let everyone else enjoy their sequels and reality shows.
- puranjay 12y ago
  Data is useless for art because art is expected to explore the boundaries of what's acceptable and what's conventional. There is no data for things that haven't even been done yet. Historical data on sales of paintings would've never told Picasso to pursue Cubism, or for Salinger to write 'Catcher in the Rye'. It was data that causes so many publishing houses to reject future bestsellers. Sales figures for past children's books told publishers that "Harry Potter" would never work. We all know how that played out. As for business, existing data would have never told Jobs to build an iPhone - simply because there really was no data on touchscreen devices.
  - lmm 12y ago
    > Historical data on sales of paintings would've never told Picasso to pursue Cubism, or for Salinger to write 'Catcher in the Rye'. In that case I wish art would be data-driven, and produce more of the pre-Picasso paintings that are fun to look at and pre-Salinger books that are fun to read.
- Tycho 12y ago
  disagree about sequels not advancing the art of filmmaking although the Star Wars prequels were not great IMO, they certainly brought a ton of technological innovations which have had enduring impact on filmmaking in general
- cableshaft 12y ago
  > ...25 sequels, but none of those films will be memorable. * Back to the Future 2 * Indiana Jones and the Temple of Doom * Army of Darkness * Goldeneye * Evil Dead 2 * Terminator 2 * Ghostbusters 2 * Captain America: Winter Soldier * Batman: The Dark Knight Yeah, none of those sequels are memorable. We should stop making sequels and only make original films so we don't waste money on travesties such as these.
  - koberstein 12y ago
    Those were not sequels planned for this year.
frandroid 12y ago
> Data-driven movies gave us 25 Hollywood sequels planned for this year You mean, data-driven movie investment. Movie producers don't want the same thing as cinephiles.
- baddox 12y ago
  True, and given the tiny portion of people who are cinephiles, that seems reasonable.
  - collyw 12y ago
    I am by no means a cinephile, but the Hollywood churn is certainly not what I want to see in movies.
curiously 12y ago
its funny how all of those shows are the ones that are hated by any one with a bit of intellect.
cwyers 12y ago
I struggle to call Buzzfeed "data-driven journalism." It's certainly SEO-driven, but I don't think I would call it journalism, first off.
- cableshaft 12y ago
  Yeah, not journalism. But I'd call it "data-driven media".
anigbrowl 12y ago
People love candy and ice cream, but that's not a good reason to stop eating vegetables. You're overlooking the fact that it's quite easy to condition people's wants, and there's a compelling economic incentive to feed people the cultural equivalent of refined sugar regardless of how it affects their long-term health. I don't think all this sort of stuff is bad. I'm looking forward to the next Avengers movie, in fact I love a good superhero yarn. At the same time, it's a sad fact that big-budget sequels crowd out investment in small ($10 million or less) indie movies, and there's general downward pressure on budgets at the low end of the market.
justizin 12y ago
> A ton of people love all of those things. This is exactly the problem with completely data-driven decisions. First, do people really love those things, or given an ever-increasing trend toward those formulaic things, are those the particular formulaic things they have chosen? Obviously, it's important to use whatever data is available, but isn't the entire notion of a startup to throw away _ALL_ of the prior data and do something that entrenched players won't tend to risk? I think the idea is not to throw away what's fundamentally important to whatever you do because numbers. BuzzFeed is doing much better at _terrible_ and mostly worthless journalism than they would if they hadn't invented, or at least largely propagated, clickbait, sure. Will they ever, possibly, following this course, reach the value of real journalists? They're a distraction. Sometimes the data just tells us what the easiest ways to distract people are, but following that principle alone is creating a world with lots of cruft and very little actual value.
- lmm 12y ago
  Journalism was always a distraction.
HillRat 12y ago
Back in the '90s, I met a guy who made quite a bit of money off specialized construction projects in the developing world. He invested a fair amount of that into a rather popular religious children's series, doing very well in the process. Unfortunately, this led him to assume that he was "in the entertainment business." He hired a company to research the direct-to-video market and tell him what types of product had the best return per dollar. After a lot of time and money, they came back with a report that said, in essence, that the two top performers in the DTV space were science fiction and vampire horror. The research was sound. The data were irrefutable. He immediately sank millions of dollars into producing a direct-to-video trilogy about vampires from space. A few years later, I saw him running one of those hot dog concessions outside a Home Depot.[1] The commonplace version of this are SMB executives who, primed by HBR articles about data-driven decisionmaking, ascribe magical powers to simple linear regressions, believe that a single model can tell them exactly how to unvaryingly run heterogeneous business lines, think overfit just means you can make the model even better, and try to force-feed explanatory variables down the throats of their stats consultants. Analytics is a powerful and necessary toolset, but it's become harder to keep the c-suite from spiking the sauce. [1] Hell of a work ethic, though. Not long after that he had a few dozen of those carts operating throughout the region. Never did see him again, but you've got to respect a guy who doesn't understand the meaning of "abject failure."
adrusi 12y ago
The fact that they're successful is selection bias. Of course the examples he give will be successful, because otherwise they would be obscure and readers wouldn't no what he was talking about. But while he argues that focusing on data is probably not the most effective way of running a startup, the main point of the article is stated in the title: that even if a data-driven startup is successful, it'll have no character. Obviously not every startup needs character, so take from that what you will.
saraid216 12y ago
People complain about the tyranny of the mob, and then they obey it. Capitalism.
coldtea 12y ago
>These first three examples are awful examples. A ton of people love all of those things. So what? A ton of people loved smoking cigarettes, segregation, creationism, and tons of other BS stuff too. The idea about culture is also making value judgements, not just bending over to whatever "people like" as if mere enjoyment is the be all end all of arguments. Especially since what people "like" is not the objective, "own choice", we make it to be, but can be manipulated by advertising, conditioning, lack (or less coverage) of alternatives and finally lack of education.
- icebraining 12y ago
  The idea about culture is also making value judgements, not just bending over to whatever "people like" as if mere enjoyment is the be all end all of arguments. But isn't that what the article is about? The song it points as the thing to strive for is Bohemian Rapsody (and not, say, Enter, Evening) exactly because people loved it.
  - coldtea 12y ago
    While not Enter, Evening, Bohemian Rapsody has a musical sofistication quite above the examples of statistics and crowd-research driven stuff mentioned above.

by baddox 12y ago

api 12y ago

The failure is the failure to generate novelty or adequately explore alternatives. Enough people might reliable go see 25 sequels, but none of those films will be memorable. None will advance the art of film-making. None will change anyone's life or mentality or affect the culture in any meaningful way. Being data driven means chasing the biggest, loudest signal in your data set. It means pandering to that signal, because it swamps all others. A data-driven approach is not going to lead you anywhere new. In machine learning / AI we refer to such algorithms as "greedy." A classic example would be simple hill climbing a.k.a. gradient descent. These algorithms are known to be very good at optimizing within the bounds of a simple well-behaved and regular fitness landscape, but they readily become stuck at local maxima when presented with any solution space with any complex structure. We're living in what I'm tempted to call the dark age of the local maximum, the age of gradient descent.

baddox 12y ago
> The failure is the failure to generate novelty or adequately explore alternatives. Perhaps you value novelty more than the average person. What might not be "adequate" for you might very well be adequate for a huge portion of people. Throughout this comment you mention several potential goals of content producers (being memorable, advancing the art, changing someone's life, etc.), but you make no argument for why those goals ought to be prioritized over other goals, other than the implication that you personally prefer those goals.
- alansammarone 12y ago
  It's seems clear that any content that advances the art is better than a content doesn't advance the art. I guess that what he meant is that global maxima is better than local maxima (which is clearly true), and data-driven vision is a hill climbing strategy, thus locking you into a local maxima.
  - baddox 12y ago
    > It's seems clear that any content that advances the art is better than a content doesn't advance the art. Perhaps that's true with all else being equal, but clearly all else isn't equal. > I guess that what he meant is that global maxima is better than local maxima (which is clearly true), and data-driven vision is a hill climbing strategy, thus locking you into a local maxima. My issue is that none of those examples are backed by any evidence that they are not doing a decent job of finding a global maximum.
    - TheOtherHobbes 12y ago
      It's true in the sense that what makes technology and culture interesting is invention, not endless recycling. You're pretty much suggesting that using strong feedback to force culture to stay within a tiny area of the total possible cultural phase space is just as interesting as allowing chaotic exploration of the entire space. It's not just an argument against creativity, it's an argument against invention in general. >My issue is that none of those examples are backed by any evidence that they are not doing a decent job of finding a global maximum. That's the thing about global maxima - you only know that you've found a global maximum if you've explored the entire space. Otherwise you've just stumbled across a local attractor, and you're stuck in a loop around it. This isn't even a good analogy, because cultural attractors are contingent, and they vary over time. They're also unpredictable. The reason they capture attention isn't because they're maxima in some analytic sense. They become found maxima because they summarise some aspect of human experience, so they appeal to a lot of people at once. The spectrum of possible maxima is mysterious and not understood, which is how you get - say - Harry Potter coming out of nowhere and captivating a generation. Culture is music, not a sine wave. You don't just want a single signal - you want a mix of related-but-different signals running all the time.
HaloZero 12y ago
You can make exceptions to all those rules though. Empire Strikes Back is probably the most obvious example of a sequel that was far and beyond the first and changed cinema. If you're talking about money-making sequels than Toy Story 2 and 3 were memorable and interesting.
Double_Cast 12y ago
> Being data driven means chasing the biggest, loudest signal in your data set. This reminds me of how one time, a professor of mine was discussing a pie-chart. He pointed towards the smallest sliver and said "Know what that is? That's opportunity!" He was most interested in the smallest signal because it represented untapped potential. (IIRC the topic was related to replacing oil with renewable energy.)
karmacondon 12y ago
Feh. What you're saying is only true if 100% of decisions are data driven. Just because Hollywood has chosen to make 25 sequels doesn't mean that Sundance no longer exists. It, and countless events and organizations like it, are celebrated inside and outside of Hollywood. The implications of the hypothetical extension of this trend are clear to a lot of people. As long as some percent of people are pushing the boundary, culture will be fine. Let everyone else enjoy their sequels and reality shows.
puranjay 12y ago
Data is useless for art because art is expected to explore the boundaries of what's acceptable and what's conventional. There is no data for things that haven't even been done yet. Historical data on sales of paintings would've never told Picasso to pursue Cubism, or for Salinger to write 'Catcher in the Rye'. It was data that causes so many publishing houses to reject future bestsellers. Sales figures for past children's books told publishers that "Harry Potter" would never work. We all know how that played out. As for business, existing data would have never told Jobs to build an iPhone - simply because there really was no data on touchscreen devices.
- lmm 12y ago
  > Historical data on sales of paintings would've never told Picasso to pursue Cubism, or for Salinger to write 'Catcher in the Rye'. In that case I wish art would be data-driven, and produce more of the pre-Picasso paintings that are fun to look at and pre-Salinger books that are fun to read.
Tycho 12y ago
disagree about sequels not advancing the art of filmmaking although the Star Wars prequels were not great IMO, they certainly brought a ton of technological innovations which have had enduring impact on filmmaking in general
cableshaft 12y ago
> ...25 sequels, but none of those films will be memorable. * Back to the Future 2 * Indiana Jones and the Temple of Doom * Army of Darkness * Goldeneye * Evil Dead 2 * Terminator 2 * Ghostbusters 2 * Captain America: Winter Soldier * Batman: The Dark Knight Yeah, none of those sequels are memorable. We should stop making sequels and only make original films so we don't waste money on travesties such as these.
- koberstein 12y ago
  Those were not sequels planned for this year.

frandroid 12y ago

> Data-driven movies gave us 25 Hollywood sequels planned for this year You mean, data-driven movie investment. Movie producers don't want the same thing as cinephiles.

baddox 12y ago
True, and given the tiny portion of people who are cinephiles, that seems reasonable.
- collyw 12y ago
  I am by no means a cinephile, but the Hollywood churn is certainly not what I want to see in movies.

curiously 12y ago

its funny how all of those shows are the ones that are hated by any one with a bit of intellect.

cwyers 12y ago

I struggle to call Buzzfeed "data-driven journalism." It's certainly SEO-driven, but I don't think I would call it journalism, first off.

cableshaft 12y ago
Yeah, not journalism. But I'd call it "data-driven media".

anigbrowl 12y ago

People love candy and ice cream, but that's not a good reason to stop eating vegetables. You're overlooking the fact that it's quite easy to condition people's wants, and there's a compelling economic incentive to feed people the cultural equivalent of refined sugar regardless of how it affects their long-term health. I don't think all this sort of stuff is bad. I'm looking forward to the next Avengers movie, in fact I love a good superhero yarn. At the same time, it's a sad fact that big-budget sequels crowd out investment in small ($10 million or less) indie movies, and there's general downward pressure on budgets at the low end of the market.

justizin 12y ago

> A ton of people love all of those things. This is exactly the problem with completely data-driven decisions. First, do people really love those things, or given an ever-increasing trend toward those formulaic things, are those the particular formulaic things they have chosen? Obviously, it's important to use whatever data is available, but isn't the entire notion of a startup to throw away _ALL_ of the prior data and do something that entrenched players won't tend to risk? I think the idea is not to throw away what's fundamentally important to whatever you do because numbers. BuzzFeed is doing much better at _terrible_ and mostly worthless journalism than they would if they hadn't invented, or at least largely propagated, clickbait, sure. Will they ever, possibly, following this course, reach the value of real journalists? They're a distraction. Sometimes the data just tells us what the easiest ways to distract people are, but following that principle alone is creating a world with lots of cruft and very little actual value.

lmm 12y ago
Journalism was always a distraction.

HillRat 12y ago

Back in the '90s, I met a guy who made quite a bit of money off specialized construction projects in the developing world. He invested a fair amount of that into a rather popular religious children's series, doing very well in the process. Unfortunately, this led him to assume that he was "in the entertainment business." He hired a company to research the direct-to-video market and tell him what types of product had the best return per dollar. After a lot of time and money, they came back with a report that said, in essence, that the two top performers in the DTV space were science fiction and vampire horror. The research was sound. The data were irrefutable. He immediately sank millions of dollars into producing a direct-to-video trilogy about vampires from space. A few years later, I saw him running one of those hot dog concessions outside a Home Depot.[1] The commonplace version of this are SMB executives who, primed by HBR articles about data-driven decisionmaking, ascribe magical powers to simple linear regressions, believe that a single model can tell them exactly how to unvaryingly run heterogeneous business lines, think overfit just means you can make the model even better, and try to force-feed explanatory variables down the throats of their stats consultants. Analytics is a powerful and necessary toolset, but it's become harder to keep the c-suite from spiking the sauce. [1] Hell of a work ethic, though. Not long after that he had a few dozen of those carts operating throughout the region. Never did see him again, but you've got to respect a guy who doesn't understand the meaning of "abject failure."

adrusi 12y ago

The fact that they're successful is selection bias. Of course the examples he give will be successful, because otherwise they would be obscure and readers wouldn't no what he was talking about. But while he argues that focusing on data is probably not the most effective way of running a startup, the main point of the article is stated in the title: that even if a data-driven startup is successful, it'll have no character. Obviously not every startup needs character, so take from that what you will.

saraid216 12y ago

People complain about the tyranny of the mob, and then they obey it. Capitalism.

coldtea 12y ago

>These first three examples are awful examples. A ton of people love all of those things. So what? A ton of people loved smoking cigarettes, segregation, creationism, and tons of other BS stuff too. The idea about culture is also making value judgements, not just bending over to whatever "people like" as if mere enjoyment is the be all end all of arguments. Especially since what people "like" is not the objective, "own choice", we make it to be, but can be manipulated by advertising, conditioning, lack (or less coverage) of alternatives and finally lack of education.

icebraining 12y ago
The idea about culture is also making value judgements, not just bending over to whatever "people like" as if mere enjoyment is the be all end all of arguments. But isn't that what the article is about? The song it points as the thing to strive for is Bohemian Rapsody (and not, say, Enter, Evening) exactly because people loved it.
- coldtea 12y ago
  While not Enter, Evening, Bohemian Rapsody has a musical sofistication quite above the examples of statistics and crowd-research driven stuff mentioned above.