5 ms·

I work as a data scientist on an extremely data driven team at an extremely data driven company, and I couldn't agree more strongly. Everyone I work with would

I work as a data scientist on an extremely data driven team at an extremely data driven company, and I couldn't agree more strongly. Everyone I work with would agree too. "Numbers aren't everything!" says the person whose job it is to deal with business problems via numbers. But this advice is so hard to implement. It's like the whole null hypothesis p-value debacle going on in academic publishing right now. Sure we can agree it's no good, but there's no standard alternative. Standards are powerful. They break lots of ties. All else equal, if you have a hunch and I have a measurement tied to objective reality, people will side with me. Nobody ever got fired for choosing Java. Nobody ever got fired for doing the thing that made The Metrics go up. It's so reassuring to just say "We still disagree, so let's just test it." We're submitting our dispute to adjudication via Science by resolving via The Metrics! It's much harder to be objective without these kinds of numbers, and I like being objective! I want the scientific method to tell me I did the right thing and I'm a good person. Because without that, when we're trying to figure out what to do, it's just us, you know? It feels like guessing, and that's hard.

jbob2000 7y ago
This is a recipe for mediocrity. Great things are created by people following their gut, not their metrics. Jobs didn’t have a data scientist telling him to make the iPhone. A data scientist would have told Notch to make a Call of Duty clone instead of Minecraft. There was no data scientist behind Gates, Warren, Zuckerberg, Musk, or anyone of their ilk. Looking back in history, some great discoveries and inventions were held back because the “data” said it wasn’t possible. Everybody thought ulcers were caused by stress and bad food, then Barry Marshall came along and literally trusted his gut by eating a bunch of bacteria to prove that’s how ulcers were formed. At best, data science helps you squeeze a couple extra bucks from someone else’s ingenuity. Once Jobs died, the data scientists got their hands on the iPhone and chopped it up into different models. And it was the data scientists who made your Facebook feed ad-ridden junk. And at worst, data science is destroying our society. It’s the data scientists who are driving this insane data vacuum that is plaguing the internet and our daily lives. You thwarted the 2016 election by weaponizing data against America. You did it again with brexit. No doubt you will be out in full force for every major political event for the next few decades, sullying the discourse of the electorate. Bit of a rant... sorry. I hate this over-reliance on data and the smugness it’s captors carry.
- mdorazio 7y ago
  I would say it's a recipe for continuation rather than innovation. You're unlikely to invent anything revolutionary just by looking at metrics, but you're also unlikely to keep a business going without them. It's easy to look at something like the iPhone and say, "metrics didn't invent that!" but metrics absolutely put millions of phones in the hands of customers, built an effective production and supply chain, optimized the whole experience, and turned Apple into a trillion dollar company. As with most things, there's a balance and a place for each.
  - jbob2000 7y ago
    I disagree that metrics helped Apple scale, but we’d have to wade in to a semantic debate to settle this. When you have a killer idea, scaling is obvious; just do more. More people, more machines, more materials, etc. You don’t need a number to tell you how much more to scale, because you always want the max anyways. Tim Cook was told there weren’t enough CNC machines in the world to manufacture the unibody MacBook. So he bought the company that makes CNC machines. What’s the point of metrics when you can shift the entire world to meet your needs? Great things are like that. Metrics are for mediocre products and services. We use them in banking because people fucking hate the banking system and we have to pull teeth to get customers to spend money. McDonald’s uses metrics to gauge how shitty they can make their beef without people noticing. Hollywood uses metrics to decide which movie franchise to reboot or make a sequel of. Metrics are a sign that you are done innovating and you are squeezing the sponge before someone else’s innovation dethrones yours.
    - nitwit005 7y ago
      > You don’t need a number to tell you how much more to scale, because you always want the max anyways. And you bankrupted the company building factories you didn't need, because iPhone demand has its limits.
    - leeoniya 7y ago
      i mostly agree with you. great intuition and experience can largely reduce the need for metrics. the problem is that not every business decision is clear cut and people who have to make them are not always Jobs or Gates. what metrics give you is a way to give the average joe (which is most of us) a way to take calculated risks when there are multiple stakeholders each with their own opinion of what should be done. metrics - like statistics - can be used effectively as a guide or can be used obsessively like dogma.
hef19898 7y ago
For me that depends on the level you are looking at: Tactical, Operational, Strategic. The first two have to be data and metrics driven, that's where you make or loose money right now. On the strategic level your performance metrics are giving only a baseline for your capabilities, strength and weaknesses. That should only be affecting the implementation of a given strategy and never the definition of it. And as always, if a metrics system is ill defined from the start it will steer you in the wrong direction. Like a broken speedometer in a car. And we didn't even start to talk about the corresponding incentive and bonus program yet...
- heymijo 7y ago
  Schlitz beer is a great example of the downfall of managing metrics from a strategic level. They began cutting cost/quality of ingredients in their beer to compete with Anheuser Busch. "Salami slicing the company to death."[0] >* the steps from A to B and from B to C might have been tiny and unnoticeable, but the steps from A to M added up to a big leap.* [1] The metrics looked great, getting more beer out of less ingredients, in less time. Until the 70's when they hit a tipping point and drinkers stopped buying, tanking the company. [0] https://en.wikipedia.org/wiki/Salami_slicing https://en.wikipedia.org/wiki/Salami_slicing [1] https://beerconnoisseur.com/articles/how-milwaukees-famous-beer-became-infamous https://beerconnoisseur.com/articles/how-milwaukees-famous-b...
heisenbit 7y ago
Nobody has ever been fired for believing data.
- username90 7y ago
  Unless it is data about political stuff like religion, race or gender. I bet someone has even gotten fired for believing in evolution.
  - concordDance 7y ago
    Bringing up Culture War ideas like this in unrelated topics is a good way to derail them
plaidfuji 7y ago
This speaks to a concept in data science I like to call "measurability bias" - the entire field of data science is biased toward making decisions on things that are, well, measurable. If you try to apply data science to a problem whose objective function doesn't have robust representative measurements tied to it, you are bound to fail. Being able to read subtle facial cues, measure the energy in a room, basically all of the subtle empathic interactions required to gauge customer satisfaction... we're not there yet.
tryitnow 7y ago
I think the whole "data-driven" ideology is getting worn out. It's especially bad on a lot of data science teams because there can be a lack of serious management experience and know-how if the team has a background mostly in academia. Academia is full of extremely dysfunctional behavior, even moreso than corporations, which is shockingly hard to pull off! Everything described in this article is old hat for experienced, knowledgeable executives, people in the know have been talking about these ideas for at least half a century (dating back to Drucker, Deming and others). While I agree with the authors' recommendations, I disagree about their diagnosis. I think Wells Fargo's main problem was a lack of internal controls, which is a boring, but necessary part of any large business.

jbob2000 7y ago

This is a recipe for mediocrity. Great things are created by people following their gut, not their metrics. Jobs didn’t have a data scientist telling him to make the iPhone. A data scientist would have told Notch to make a Call of Duty clone instead of Minecraft. There was no data scientist behind Gates, Warren, Zuckerberg, Musk, or anyone of their ilk. Looking back in history, some great discoveries and inventions were held back because the “data” said it wasn’t possible. Everybody thought ulcers were caused by stress and bad food, then Barry Marshall came along and literally trusted his gut by eating a bunch of bacteria to prove that’s how ulcers were formed. At best, data science helps you squeeze a couple extra bucks from someone else’s ingenuity. Once Jobs died, the data scientists got their hands on the iPhone and chopped it up into different models. And it was the data scientists who made your Facebook feed ad-ridden junk. And at worst, data science is destroying our society. It’s the data scientists who are driving this insane data vacuum that is plaguing the internet and our daily lives. You thwarted the 2016 election by weaponizing data against America. You did it again with brexit. No doubt you will be out in full force for every major political event for the next few decades, sullying the discourse of the electorate. Bit of a rant... sorry. I hate this over-reliance on data and the smugness it’s captors carry.

mdorazio 7y ago
I would say it's a recipe for continuation rather than innovation. You're unlikely to invent anything revolutionary just by looking at metrics, but you're also unlikely to keep a business going without them. It's easy to look at something like the iPhone and say, "metrics didn't invent that!" but metrics absolutely put millions of phones in the hands of customers, built an effective production and supply chain, optimized the whole experience, and turned Apple into a trillion dollar company. As with most things, there's a balance and a place for each.
- jbob2000 7y ago
  I disagree that metrics helped Apple scale, but we’d have to wade in to a semantic debate to settle this. When you have a killer idea, scaling is obvious; just do more. More people, more machines, more materials, etc. You don’t need a number to tell you how much more to scale, because you always want the max anyways. Tim Cook was told there weren’t enough CNC machines in the world to manufacture the unibody MacBook. So he bought the company that makes CNC machines. What’s the point of metrics when you can shift the entire world to meet your needs? Great things are like that. Metrics are for mediocre products and services. We use them in banking because people fucking hate the banking system and we have to pull teeth to get customers to spend money. McDonald’s uses metrics to gauge how shitty they can make their beef without people noticing. Hollywood uses metrics to decide which movie franchise to reboot or make a sequel of. Metrics are a sign that you are done innovating and you are squeezing the sponge before someone else’s innovation dethrones yours.
  - nitwit005 7y ago
    > You don’t need a number to tell you how much more to scale, because you always want the max anyways. And you bankrupted the company building factories you didn't need, because iPhone demand has its limits.
  - leeoniya 7y ago
    i mostly agree with you. great intuition and experience can largely reduce the need for metrics. the problem is that not every business decision is clear cut and people who have to make them are not always Jobs or Gates. what metrics give you is a way to give the average joe (which is most of us) a way to take calculated risks when there are multiple stakeholders each with their own opinion of what should be done. metrics - like statistics - can be used effectively as a guide or can be used obsessively like dogma.

hef19898 7y ago

For me that depends on the level you are looking at: Tactical, Operational, Strategic. The first two have to be data and metrics driven, that's where you make or loose money right now. On the strategic level your performance metrics are giving only a baseline for your capabilities, strength and weaknesses. That should only be affecting the implementation of a given strategy and never the definition of it. And as always, if a metrics system is ill defined from the start it will steer you in the wrong direction. Like a broken speedometer in a car. And we didn't even start to talk about the corresponding incentive and bonus program yet...

heymijo 7y ago
Schlitz beer is a great example of the downfall of managing metrics from a strategic level. They began cutting cost/quality of ingredients in their beer to compete with Anheuser Busch. "Salami slicing the company to death."[0] >* the steps from A to B and from B to C might have been tiny and unnoticeable, but the steps from A to M added up to a big leap.* [1] The metrics looked great, getting more beer out of less ingredients, in less time. Until the 70's when they hit a tipping point and drinkers stopped buying, tanking the company. [0] https://en.wikipedia.org/wiki/Salami_slicing https://en.wikipedia.org/wiki/Salami_slicing [1] https://beerconnoisseur.com/articles/how-milwaukees-famous-beer-became-infamous https://beerconnoisseur.com/articles/how-milwaukees-famous-b...

heisenbit 7y ago

Nobody has ever been fired for believing data.

username90 7y ago
Unless it is data about political stuff like religion, race or gender. I bet someone has even gotten fired for believing in evolution.
- concordDance 7y ago
  Bringing up Culture War ideas like this in unrelated topics is a good way to derail them

plaidfuji 7y ago

This speaks to a concept in data science I like to call "measurability bias" - the entire field of data science is biased toward making decisions on things that are, well, measurable. If you try to apply data science to a problem whose objective function doesn't have robust representative measurements tied to it, you are bound to fail. Being able to read subtle facial cues, measure the energy in a room, basically all of the subtle empathic interactions required to gauge customer satisfaction... we're not there yet.

tryitnow 7y ago

I think the whole "data-driven" ideology is getting worn out. It's especially bad on a lot of data science teams because there can be a lack of serious management experience and know-how if the team has a background mostly in academia. Academia is full of extremely dysfunctional behavior, even moreso than corporations, which is shockingly hard to pull off! Everything described in this article is old hat for experienced, knowledgeable executives, people in the know have been talking about these ideas for at least half a century (dating back to Drucker, Deming and others). While I agree with the authors' recommendations, I disagree about their diagnosis. I think Wells Fargo's main problem was a lack of internal controls, which is a boring, but necessary part of any large business.