5 ms·

"Managers will say they want to make data-driven decisions, but they really want decision-driven data" Ooofff. This is too true. How often is the case that dat

"Managers will say they want to make data-driven decisions, but they really want decision-driven data" Ooofff. This is too true. How often is the case that data is collected to test hypotheses vs confirming priors?

phkahler 4y ago
>> "Managers will say they want to make data-driven decisions, but they really want decision-driven data" Ooofff. This is too true. How often is the case that data is collected to test hypotheses vs confirming priors? Find me some evidence of WMDs in Iraq! Yessss Sir!
A4ET8a8uTh0 4y ago
That one stood out to me as well, but, to be fair, this predated current 'fashionable trend' for data driven decisions. It is, sadly, not a new development, but something to still be overcome.
Reimersholme 4y ago
Rather than wanting to confirm priors, I believe this usually is a problem with neither the PM nor the data scientist ensuring that the problem formulation is good enough before diving in. I.e., what data would be needed to actually test the hypothesis? Do we have that data or not? Is the hypothesis even formulated in a way to be falsified in theory? I've seen so many analysis tasks where data scientists without questioning went away for a few weeks to crunch data and come back with some random graphs and statistics that are completely useless as decision support.
- nisegami 4y ago
  You're overthinking it. Executives and managers quite literally want to see data that confirms their existing convictions and beliefs so they can act on those beliefs under the guise of it being "data-driven".
cobbzilla 4y ago
So true. This article accurately describes DS at many companies. The preceding sentence is a hilariously cynical zinger: “Those who have seen my Twitter posts know that I believe the role of the data scientist in a scenario of insane management is not to provide real, honest consultation, but to launder these insane ideas as having some sort of basis in objective reality even if they don’t.”
lajosbacs 4y ago
This especially sucks if you are the middle manager. You know that what you are asked to do is a complete BS but you have to somehow communicate it to you underlings (who see through the BS) without using sarcasm or snarky remarks.
yamtaddle 4y ago
I've found this to be the rule, not the exception. Pointing out extremely-obvious (to me? Maybe I'm just unusually good at it? I don't even have much formal science training, and hell, barely any math training by the standards of HN folks, though) damning errors in experimental construction that should invalidate the whole thing won't earn you any friends, even if you do it before the work is undertaken, and even if you're telling the person who's claiming to want good data and useful results. Everyone seems to just want a veneer of science to what they're doing, not actually good efforts at it. As long as you have a paper-thin layer of justification that falls apart if anyone looks at it long enough, that's considered good enough and people will sit around in meetings nodding along. Of course, in many situations the business totally lacks what it needs to correctly do the "data-driven" stuff they want to, and it'd take a good deal of up-front effort by competent people to get it, amounting to entire new projects or deep modification of existing projects. So, given the choice between: going without that stuff and acknowledging that a lot of what they're doing is guesswork and gut decision making, or simply arbitrary; putting a smaller but still-large amount of work into finding out what they can glean from what's available; spending the time and money to collect what they need, the right way, to do the data-driven decision making they claim to want to do; and insisting they're doing things "data driven" but having all their data hopelessly ruined by e.g. selection bias and comically-bad experimental construction that can't possibly be yielding reliable results, so they can cheap out and get no actual "data-driven" benefits aside from falsely claiming that's what they're doing—they tend to go with that last option, nearly every time!

phkahler 4y ago

>> "Managers will say they want to make data-driven decisions, but they really want decision-driven data" Ooofff. This is too true. How often is the case that data is collected to test hypotheses vs confirming priors? Find me some evidence of WMDs in Iraq! Yessss Sir!

A4ET8a8uTh0 4y ago

That one stood out to me as well, but, to be fair, this predated current 'fashionable trend' for data driven decisions. It is, sadly, not a new development, but something to still be overcome.

Reimersholme 4y ago

Rather than wanting to confirm priors, I believe this usually is a problem with neither the PM nor the data scientist ensuring that the problem formulation is good enough before diving in. I.e., what data would be needed to actually test the hypothesis? Do we have that data or not? Is the hypothesis even formulated in a way to be falsified in theory? I've seen so many analysis tasks where data scientists without questioning went away for a few weeks to crunch data and come back with some random graphs and statistics that are completely useless as decision support.

nisegami 4y ago
You're overthinking it. Executives and managers quite literally want to see data that confirms their existing convictions and beliefs so they can act on those beliefs under the guise of it being "data-driven".

cobbzilla 4y ago

So true. This article accurately describes DS at many companies. The preceding sentence is a hilariously cynical zinger: “Those who have seen my Twitter posts know that I believe the role of the data scientist in a scenario of insane management is not to provide real, honest consultation, but to launder these insane ideas as having some sort of basis in objective reality even if they don’t.”

lajosbacs 4y ago

This especially sucks if you are the middle manager. You know that what you are asked to do is a complete BS but you have to somehow communicate it to you underlings (who see through the BS) without using sarcasm or snarky remarks.

yamtaddle 4y ago

I've found this to be the rule, not the exception. Pointing out extremely-obvious (to me? Maybe I'm just unusually good at it? I don't even have much formal science training, and hell, barely any math training by the standards of HN folks, though) damning errors in experimental construction that should invalidate the whole thing won't earn you any friends, even if you do it before the work is undertaken, and even if you're telling the person who's claiming to want good data and useful results. Everyone seems to just want a veneer of science to what they're doing, not actually good efforts at it. As long as you have a paper-thin layer of justification that falls apart if anyone looks at it long enough, that's considered good enough and people will sit around in meetings nodding along. Of course, in many situations the business totally lacks what it needs to correctly do the "data-driven" stuff they want to, and it'd take a good deal of up-front effort by competent people to get it, amounting to entire new projects or deep modification of existing projects. So, given the choice between: going without that stuff and acknowledging that a lot of what they're doing is guesswork and gut decision making, or simply arbitrary; putting a smaller but still-large amount of work into finding out what they can glean from what's available; spending the time and money to collect what they need, the right way, to do the data-driven decision making they claim to want to do; and insisting they're doing things "data driven" but having all their data hopelessly ruined by e.g. selection bias and comically-bad experimental construction that can't possibly be yielding reliable results, so they can cheap out and get no actual "data-driven" benefits aside from falsely claiming that's what they're doing—they tend to go with that last option, nearly every time!