4 ms·

As a layman, your three points terrify me about the state of science. Well, numbers two and three do. There are still people addicted to video games. WoW and Mi

As a layman, your three points terrify me about the state of science. Well, numbers two and three do. There are still people addicted to video games. WoW and Minecraft probably moreso than Asheron's Call back in 2003. As to point number two: Are these questionnaires or something that are used to get psychological data? Why would those not be accessible--don't you need to know the questions asked to determine if the results are valid? As to point three: Data cleaning?! I would understand throwing out invalid results (i.e.-you find out somebody's lying, errors with data collection, etc), but to throw out results because they look like they don't belong feels disingenuous. I've done a lot of work with public company financials and operational systems. I cannot, on my worst day, imagine telling the SEC, "Yeah, well that return was really abnormal and was an outlier, so we decided not to record it." Why are these data points thrown out? To me, again, a layman, it appears that if this is standard practice, these experiments are starting with a conclusion and then just going through the motions to get that conclusion published. Why are data cleansing and secrecy normal practices? What about the scientific method don't I understand?

iandanforth 14y ago
The line between an invalid result and one that looks like it doesn't belong is fuzzy. I believe I was removing the former, but if an independent replicator doesn't have access to the raw data they would never discover that I had been fooling myself. As to secrecy, there is nothing being hidden, instead it's that nothing in the system requires you to publish all your data, and so because you're pressed for time you don't. Never ascribe to malice what laziness can explain :)
- endersshadow 14y ago
  I'm not ascribing this to malice. It strikes me as a paragon of negligence if that's the way that the science of psychology is being done. In fact, it strikes me as an affront to science. Take this in contrast to the neutrinos that were going faster than light at CERN. Instead of going, "Okay, well, those can't happen, so those are outliers," they reported their results, published their methods and data, and said to the world, "Help us validate this." It seems to me like the modern psychology scientific method is the antithesis of this--"These are the results of my massaged data, and no you can't see how I got them, just trust me." I'm not attributing this to malice, I'm attributing this to negligence and laziness. Putting your work in front of people for critique is hard. Really hard. But, it's part of science. If psychology as a field has convinced themselves that they're above that reproach, then I think that's a huge condemnation of the field. When I first read the article, I thought it was a bit sensationalist to call psychology "under attack," but now I'm not so sure. I've assumed that reproducability is the standard litmus test of all science, but I guess not, and that leaves me with a bitter taste in my mouth toward the field of psychology.
  - jmilloy 14y ago
    Outlier detection is an important part of basic statistics, and has been for a long time. It isn't about just deleting a few data points that "look" strange. Some statistical tests are robust to outliers and some are not... it's always important to use the appropriate tests in the appropriate way and be open about them.
pnathan 14y ago
You can clean the data according to certain metrics: e.g., "We removed all results that were greater than 6 standard deviations from the result". There are other metrics of dropping data that is "out there". As someone finishing my Master's, I will be releasing all my work, including test inputs and reproducibility results, as a Mercurial repository. Some of it depends on an exterior compiler, but the interesting ideas don't. I feel it is absolutely critical and honest to release source code and all data. See the CRAPL: http://matt.might.net/articles/crapl/ http://matt.might.net/articles/crapl/.

iandanforth 14y ago

The line between an invalid result and one that looks like it doesn't belong is fuzzy. I believe I was removing the former, but if an independent replicator doesn't have access to the raw data they would never discover that I had been fooling myself. As to secrecy, there is nothing being hidden, instead it's that nothing in the system requires you to publish all your data, and so because you're pressed for time you don't. Never ascribe to malice what laziness can explain :)

endersshadow 14y ago
I'm not ascribing this to malice. It strikes me as a paragon of negligence if that's the way that the science of psychology is being done. In fact, it strikes me as an affront to science. Take this in contrast to the neutrinos that were going faster than light at CERN. Instead of going, "Okay, well, those can't happen, so those are outliers," they reported their results, published their methods and data, and said to the world, "Help us validate this." It seems to me like the modern psychology scientific method is the antithesis of this--"These are the results of my massaged data, and no you can't see how I got them, just trust me." I'm not attributing this to malice, I'm attributing this to negligence and laziness. Putting your work in front of people for critique is hard. Really hard. But, it's part of science. If psychology as a field has convinced themselves that they're above that reproach, then I think that's a huge condemnation of the field. When I first read the article, I thought it was a bit sensationalist to call psychology "under attack," but now I'm not so sure. I've assumed that reproducability is the standard litmus test of all science, but I guess not, and that leaves me with a bitter taste in my mouth toward the field of psychology.
- jmilloy 14y ago
  Outlier detection is an important part of basic statistics, and has been for a long time. It isn't about just deleting a few data points that "look" strange. Some statistical tests are robust to outliers and some are not... it's always important to use the appropriate tests in the appropriate way and be open about them.

pnathan 14y ago

You can clean the data according to certain metrics: e.g., "We removed all results that were greater than 6 standard deviations from the result". There are other metrics of dropping data that is "out there". As someone finishing my Master's, I will be releasing all my work, including test inputs and reproducibility results, as a Mercurial repository. Some of it depends on an exterior compiler, but the interesting ideas don't. I feel it is absolutely critical and honest to release source code and all data. See the CRAPL: http://matt.might.net/articles/crapl/ http://matt.might.net/articles/crapl/.