5 ms·

Anytime I see 'evidence based', I can't help but think "my opinion for which I've cherry picked data".

Shugarl 6y ago
I also find it quite strange. I remember my teachers telling that we're making a little bit of progress in software engineering research, but comparing it to the state of research in fields like Math or Physics would be like comparing a fully grown man and a toddler. In other words, we hardly know anything
- pmiller2 6y ago
  We may hardly know anything, but, comparing any field that involves humans to math or physics is an apples to oranges comparison. All it takes to do math research is time, knowledge, writing material, and, preferably, the support of an institution which allows one to do research full time. There are no pesky experiments to run, except possibly inside a computer. Physics can be the same way, but, even when it involves "pesky experiments," those experiments only depend on the physical laws of the universe, and the labor of a fleet of graduate students who have great incentives to see these experiments through. SWE research has either of these properties.
st1x7 6y ago
What's the alternative?
- dpc_pw 6y ago
  Stop pretending that there's an easy empirical answer to everything, and accept the nature of complex domains.
mooreds 6y ago
Have you checked out the book (PDF here: http://knosof.co.uk/ESEUR/ESEUR.pdf http://knosof.co.uk/ESEUR/ESEUR.pdf )? There are 2000 footnotes referencing published studies. It's not perfect, but hopefully there are enough different views all triangulating on "truth" to provide value.
monocasa 6y ago
Better than "my opinion for which I've not even attempted to see if the data can be contorted to support", which is still the stage we're at for a lot of our industry since it's such a new field.
- yen223 6y ago
  My pet peeve with internet forums is when we counter n=1 studies with n=0 opinions
- bigbubba 6y ago
  I'm not convinced that's right. Somebody who incorrectly believes their belief is backed by data may be less inclined to change their mind than somebody who recognizes that their belief isn't backed by data and consequently has a good chance of being wrong. In other words, the chance that your belief is correct or incorrect matters, but your inclination to reevaluate your belief is also something to consider. Cherrypicking data to support your beliefs or various forms of inadvertent p-hacking may make people feel quite certain of things that actually aren't true.
deleted 6y ago
[deleted]
IshKebab 6y ago
I don't, but maybe that's because I've mostly heard in a medical context to distinguish policies based on any evidence at all from policies based on gut feeling, of which there are a lot!
gameswithgo 6y ago
It is a common cognitive approach to argue that things are unknowable, especially if evidence is currently pointing to a truth we wish were not so. Certainly it is very, very hard to get quality evidence in some domains, and software engineering is one of the hard ones. That does not mean it is impossible, and we should never dismiss things out of hand. We should have specific critiques for claims we think are wrong, and explain why.
gwd 6y ago
Do you have any evidence for that? ;-) The world is a complicated place and the truth is hard to find. Unintentional confirmation bias is a thing. Intentional (at some level) cherry-picking to support your preferred answer is a thing too. But that doesn't mean that nobody ever learns anything from looking at data. If someone claims to be "evidence-based", then their intention should be given the benefit of the doubt. They may be affected by confirmation bias, but that doesn't mean their conclusions are wholly baseless. You should take a look and see how they're doing before dismissing them out of hand.
mekoka 6y ago
It works both ways though. When the data presented to us suit our current view or agenda, we somehow feel that it's valid. I remember when discussions about "open plans" or "estimates" were still a thing, without fail "evidence based" works such as Peopleware or The Mythical Man Month would be mentioned. If this book turned out to agree with the view that "scrum is not really more productive", or "less meetings means more productive", or "the interview process is broken", then we'd all be really glad to refer to its data.
TameAntelope 6y ago
I completely agree with this -- obviously we have to made decisions and build systems based on the data we observe and collect, but the idea that it's some incorruptible or unbiased exercise is one that I think is both appealing and naïve. By assuming there's a "right" way to arrive at conclusions, people run the very dangerous risk of believing they've got the One True Answer, and everyone else are Wrong. I think reality is a lot closer to the idea that there is no objective/rational way to decide what information is relevant and what information isn't, and we have to constantly adjust that dial in everything we think about and do. There is an article about a group of people who have greatly reduced emotional capacity (not sociopaths, something more severe), and are therefore more or less rendered incapable of making decisions. It's a fascinating read, but I can't find it in my brief searches. Maybe someone else knows what I'm talking about? Edit: I found the relevant anecdote! From an NPR interview https://www.npr.org/templates/story/story.php?storyId=101334645 https://www.npr.org/templates/story/story.php?storyId=101334... about a book called "How We Decide": "And I think one of the best examples of this comes from the work of a neurologist named Antonio Demasio, who in the early 1980s was studying patients who, because of a brain tumor, lost the ability to experience their emotions. So they didn't feel the everyday feelings of fear and pleasure. And you'd think, if you were Plato, that these people would be philosopher-kings, that they would be perfectly rational creatures, they'd make the best set of decisions possible. And instead, what you find is that they are like me in the cereal aisle, that they're pathologically indecisive, that they would spend all day trying to figure out where to eat lunch. They'd spend five hours choosing between a blue pen or a black pen or a red pen, that all these everyday decisions we take for granted, they couldn't make. And that's because they were missing these subtle, visceral signals that were telling them to just choose the black pen or to eat the tuna fish sandwich or whatever. And then when we're cut off from these emotional signals, the most basic decisions become all but impossible."

I also find it quite strange. I remember my teachers telling that we're making a little bit of progress in software engineering research, but comparing it to the state of research in fields like Math or Physics would be like comparing a fully grown man and a toddler. In other words, we hardly know anything

pmiller2 6y ago
We may hardly know anything, but, comparing any field that involves humans to math or physics is an apples to oranges comparison. All it takes to do math research is time, knowledge, writing material, and, preferably, the support of an institution which allows one to do research full time. There are no pesky experiments to run, except possibly inside a computer. Physics can be the same way, but, even when it involves "pesky experiments," those experiments only depend on the physical laws of the universe, and the labor of a fleet of graduate students who have great incentives to see these experiments through. SWE research has either of these properties.

st1x7 6y ago

What's the alternative?

dpc_pw 6y ago
Stop pretending that there's an easy empirical answer to everything, and accept the nature of complex domains.

mooreds 6y ago

Have you checked out the book (PDF here: http://knosof.co.uk/ESEUR/ESEUR.pdf http://knosof.co.uk/ESEUR/ESEUR.pdf )? There are 2000 footnotes referencing published studies. It's not perfect, but hopefully there are enough different views all triangulating on "truth" to provide value.

monocasa 6y ago

Better than "my opinion for which I've not even attempted to see if the data can be contorted to support", which is still the stage we're at for a lot of our industry since it's such a new field.

yen223 6y ago
My pet peeve with internet forums is when we counter n=1 studies with n=0 opinions
bigbubba 6y ago
I'm not convinced that's right. Somebody who incorrectly believes their belief is backed by data may be less inclined to change their mind than somebody who recognizes that their belief isn't backed by data and consequently has a good chance of being wrong. In other words, the chance that your belief is correct or incorrect matters, but your inclination to reevaluate your belief is also something to consider. Cherrypicking data to support your beliefs or various forms of inadvertent p-hacking may make people feel quite certain of things that actually aren't true.

deleted 6y ago

[deleted]

IshKebab 6y ago

I don't, but maybe that's because I've mostly heard in a medical context to distinguish policies based on any evidence at all from policies based on gut feeling, of which there are a lot!

gameswithgo 6y ago

It is a common cognitive approach to argue that things are unknowable, especially if evidence is currently pointing to a truth we wish were not so. Certainly it is very, very hard to get quality evidence in some domains, and software engineering is one of the hard ones. That does not mean it is impossible, and we should never dismiss things out of hand. We should have specific critiques for claims we think are wrong, and explain why.

gwd 6y ago

Do you have any evidence for that? ;-) The world is a complicated place and the truth is hard to find. Unintentional confirmation bias is a thing. Intentional (at some level) cherry-picking to support your preferred answer is a thing too. But that doesn't mean that nobody ever learns anything from looking at data. If someone claims to be "evidence-based", then their intention should be given the benefit of the doubt. They may be affected by confirmation bias, but that doesn't mean their conclusions are wholly baseless. You should take a look and see how they're doing before dismissing them out of hand.

mekoka 6y ago

It works both ways though. When the data presented to us suit our current view or agenda, we somehow feel that it's valid. I remember when discussions about "open plans" or "estimates" were still a thing, without fail "evidence based" works such as Peopleware or The Mythical Man Month would be mentioned. If this book turned out to agree with the view that "scrum is not really more productive", or "less meetings means more productive", or "the interview process is broken", then we'd all be really glad to refer to its data.

TameAntelope 6y ago

I completely agree with this -- obviously we have to made decisions and build systems based on the data we observe and collect, but the idea that it's some incorruptible or unbiased exercise is one that I think is both appealing and naïve. By assuming there's a "right" way to arrive at conclusions, people run the very dangerous risk of believing they've got the One True Answer, and everyone else are Wrong. I think reality is a lot closer to the idea that there is no objective/rational way to decide what information is relevant and what information isn't, and we have to constantly adjust that dial in everything we think about and do. There is an article about a group of people who have greatly reduced emotional capacity (not sociopaths, something more severe), and are therefore more or less rendered incapable of making decisions. It's a fascinating read, but I can't find it in my brief searches. Maybe someone else knows what I'm talking about? Edit: I found the relevant anecdote! From an NPR interview https://www.npr.org/templates/story/story.php?storyId=101334645 https://www.npr.org/templates/story/story.php?storyId=101334... about a book called "How We Decide": "And I think one of the best examples of this comes from the work of a neurologist named Antonio Demasio, who in the early 1980s was studying patients who, because of a brain tumor, lost the ability to experience their emotions. So they didn't feel the everyday feelings of fear and pleasure. And you'd think, if you were Plato, that these people would be philosopher-kings, that they would be perfectly rational creatures, they'd make the best set of decisions possible. And instead, what you find is that they are like me in the cereal aisle, that they're pathologically indecisive, that they would spend all day trying to figure out where to eat lunch. They'd spend five hours choosing between a blue pen or a black pen or a red pen, that all these everyday decisions we take for granted, they couldn't make. And that's because they were missing these subtle, visceral signals that were telling them to just choose the black pen or to eat the tuna fish sandwich or whatever. And then when we're cut off from these emotional signals, the most basic decisions become all but impossible."