4 ms·

This is probably the most annoying problem in my daily life (yeah I know, first-world problems). I have daily conversations with biologists where I've analysed

by Blahah 13y ago

This is probably the most annoying problem in my daily life (yeah I know, first-world problems). I have daily conversations with biologists where I've analysed some data and associated (say) a posterior probability with each condition in the model. They insist I give them p-values or something they can present as though they were p-values. At the beginning of my PhD I complied. Then they throw out all the nuance in the data, put one asterisk for p < 0.05, two asterisks for p < 0.01, etc., and (this is the horrifying part) _believe_ that an asterisk indicates that something is true. They put stupid asterisks all over my beautiful plots and then think their arbitrary cutoffs mean something biologically meaningful. I die a little inside every time I see an asterisk on a plot. Now I refuse to use p-values and deliberately construct analyses that are incompatible with Fisherian statistics. And rather than giving people raw numbers, I produce a massive document of interpretation. Takes a huge amount of time, but I'm hoping it will mean my publishing track will contain significantly (ha!) fewer false results than most biologists'.

km3k 13y ago
> _believe_ that an asterisk indicates that something is true Attitudes like this bug me so much. From my experience it is a big part of what makes people that are not familiar with science not trust scientists.
- vacri 13y ago
  The story is so much simpler than that. People not familiar with science don't trust scientists because they want to stick to their own known story. Change is scary, and people generally don't like it unless it significantly improves their living conditions. It doesn't matter how rigorous scientists are with their methodology, my 'new age' mother isn't going to stop believing in fairies in the trees because scientists say there's no proof. Or when she talks about higher and lower vibrations, she is puzzled when I ask her "vibrations of what? what is it that's vibrating?". This isn't important to her story, and is merely a detail that makes life less enjoyable, so she ignores it. A less obnoxious version of ICP's "scientists are ruining life by explaining things" during their 'how do magnets work' phase. Saying that people don't trust scientists because of p values is a bit like saying people don't trust plumbers when they use blue plumber's tape instead of gray. People don't like plumbers because they don't show up on time and charge through the nose - things that change your own story.
  - km3k 13y ago
    That's a good point too, but not what I was trying to get at. I realize now I wasn't clear enough. I was talking more about people that recognize science as a good thing and tend to follow what is presented as scientific facts, but aren't specifically trained in it. The problem I see is that some scientists are taking data that points toward a conclusion, even if it isn't well supported or rigorous data and talking as if it is absolutely certain and proven. This degrades people's trust in scientists, and likely science in general. It's also a problem of the media too, since they're the ones promoting the scientists that act in this way.
mtdewcmu 13y ago
Noble, but it sounds risky. Don't you need false results to pad your CV to keep up?
- gwern 13y ago
  He may not be long for this (scientific) world. As they say, _p_-ublish or perish.
  - Blahah 13y ago
    Publishing is easy. It's just a bit easier for the people who are (perhaps unconsciously) cheating. I'm pretty confident my loudly-refuse-to-cheat strategy will prevent me perishing before my time.
- Blahah 13y ago
  No.
Estragon 13y ago
"Once the two hypotheses have been defined, the first hypothesis is scarcely mentioned again -- attention focuses solely on the null hypothesis. It makes me laugh to write this, but it's true! The null hypothesis is accepted or rejected purely on the basis of how unexpected the data were to H0, not on how much better H1 predicted the data." http://www.inference.phy.cam.ac.uk/mackay/itprnn/ps/457.466.ps.gz http://www.inference.phy.cam.ac.uk/mackay/itprnn/ps/457.466....
GFK_of_xmaspast 13y ago
You're going to get a reputation as the kind of person nobody wants to work with.
- Blahah 13y ago
  Actually I have an ever increasing queue of collaboration requests. My reputation is (I am told) as someone who can rigorously get the story from the data, and who can solve problems. The people who don't want the rigour are the people I reject, not the other way around.
capnrefsmmat 13y ago
My favorite term for that behavior is "stargazing," because they're staring at the little asterisks instead of thinking about the meaning of the results. I have been working on a guide for practicing scientists which attacks common statistical misconceptions: http://www.refsmmat.com/statistics/ http://www.refsmmat.com/statistics/ It's on its way to publication, so you'll be able to hit your biologists upside the head with it in due time. (Although it'll be slim, so it won't hurt too much.)
benrapscallion 13y ago
As a biologist I would argue that we also cannot dwell too much on p values. The true test is to design follow up experiments to test causal predictions from many of these borderline p value phenomena, for example: analyzing microarrays data to find interesting GO categories. Whether it's p=.001 or p=.051 is less important than whether or not it provides a good clue to the underlying mechanistic biology, and there is no way to address that without designing specific follow up experiments. That's why experimentalists don't dwell too much upon p values. They're, for the most part, a clue.
- Blahah 13y ago
  In my experience, experimentalists tend to dwell on p-values and especially on cutoffs in their molecular experimental data as much as they do in high throughput data.

by Blahah 13y ago

km3k 13y ago

> _believe_ that an asterisk indicates that something is true Attitudes like this bug me so much. From my experience it is a big part of what makes people that are not familiar with science not trust scientists.

vacri 13y ago
The story is so much simpler than that. People not familiar with science don't trust scientists because they want to stick to their own known story. Change is scary, and people generally don't like it unless it significantly improves their living conditions. It doesn't matter how rigorous scientists are with their methodology, my 'new age' mother isn't going to stop believing in fairies in the trees because scientists say there's no proof. Or when she talks about higher and lower vibrations, she is puzzled when I ask her "vibrations of what? what is it that's vibrating?". This isn't important to her story, and is merely a detail that makes life less enjoyable, so she ignores it. A less obnoxious version of ICP's "scientists are ruining life by explaining things" during their 'how do magnets work' phase. Saying that people don't trust scientists because of p values is a bit like saying people don't trust plumbers when they use blue plumber's tape instead of gray. People don't like plumbers because they don't show up on time and charge through the nose - things that change your own story.
- km3k 13y ago
  That's a good point too, but not what I was trying to get at. I realize now I wasn't clear enough. I was talking more about people that recognize science as a good thing and tend to follow what is presented as scientific facts, but aren't specifically trained in it. The problem I see is that some scientists are taking data that points toward a conclusion, even if it isn't well supported or rigorous data and talking as if it is absolutely certain and proven. This degrades people's trust in scientists, and likely science in general. It's also a problem of the media too, since they're the ones promoting the scientists that act in this way.

mtdewcmu 13y ago

Noble, but it sounds risky. Don't you need false results to pad your CV to keep up?

gwern 13y ago
He may not be long for this (scientific) world. As they say, _p_-ublish or perish.
- Blahah 13y ago
  Publishing is easy. It's just a bit easier for the people who are (perhaps unconsciously) cheating. I'm pretty confident my loudly-refuse-to-cheat strategy will prevent me perishing before my time.
Blahah 13y ago
No.

Estragon 13y ago

"Once the two hypotheses have been defined, the first hypothesis is scarcely mentioned again -- attention focuses solely on the null hypothesis. It makes me laugh to write this, but it's true! The null hypothesis is accepted or rejected purely on the basis of how unexpected the data were to H0, not on how much better H1 predicted the data." http://www.inference.phy.cam.ac.uk/mackay/itprnn/ps/457.466.ps.gz http://www.inference.phy.cam.ac.uk/mackay/itprnn/ps/457.466....

GFK_of_xmaspast 13y ago

You're going to get a reputation as the kind of person nobody wants to work with.

Blahah 13y ago
Actually I have an ever increasing queue of collaboration requests. My reputation is (I am told) as someone who can rigorously get the story from the data, and who can solve problems. The people who don't want the rigour are the people I reject, not the other way around.

capnrefsmmat 13y ago

My favorite term for that behavior is "stargazing," because they're staring at the little asterisks instead of thinking about the meaning of the results. I have been working on a guide for practicing scientists which attacks common statistical misconceptions: http://www.refsmmat.com/statistics/ http://www.refsmmat.com/statistics/ It's on its way to publication, so you'll be able to hit your biologists upside the head with it in due time. (Although it'll be slim, so it won't hurt too much.)

benrapscallion 13y ago

As a biologist I would argue that we also cannot dwell too much on p values. The true test is to design follow up experiments to test causal predictions from many of these borderline p value phenomena, for example: analyzing microarrays data to find interesting GO categories. Whether it's p=.001 or p=.051 is less important than whether or not it provides a good clue to the underlying mechanistic biology, and there is no way to address that without designing specific follow up experiments. That's why experimentalists don't dwell too much upon p values. They're, for the most part, a clue.

Blahah 13y ago
In my experience, experimentalists tend to dwell on p-values and especially on cutoffs in their molecular experimental data as much as they do in high throughput data.