7 ms·

In 2003 I did the work that lead to this paper (http://bit.ly/I1qvuI http://bit.ly/I1qvuI). Could it be replicated? Probably not. Let me outline a few challenge

In 2003 I did the work that lead to this paper (http://bit.ly/I1qvuI http://bit.ly/I1qvuI). Could it be replicated? Probably not. Let me outline a few challenges that don't have to do with a result being true or not. 1. The passage of time Unlike physics and biology, culture has a huge impact on the field of psychology, and cultures change. Additionally, my work was about addiction to online games. I have no idea if the same type of people exist today that did then. I do know that the game I was studying (Asheron's Call) does not exist in anywhere near the same form as it did then. 2. Copyright While I was lucky enough to find psychological measures (aka tests) that were in the public domain, most are not. To replicate a result you may have to pay to access the tests and then most likely cannot re-publish them. At the very least this makes replicating work inconvenient and expensive. 3. Data Because psychology is largely driven by statistics, to replicate a result you should really start by re-analyzing the data. The reality? This data simply isn't available. It is not published alongside the results, and I don't have it anymore. I might be able to find the cleaned data, but that's not important. Error and bias can just as easily be introduced during data cleaning. For example I removed many data points that appeared as outliers, but perhaps these should have been included. No one will ever know. I hope barriers such as these are addressed in this investigation. It would be disappointing to impune the work of scientists when it is the process that could really use reform.

k4st 15y ago
Note: I am not a psychologist. What do you mean by "some tests are copyrighted"? Are you talking about a statistical method being copyrighted, an implementation thereof, or something else entirely?
- ttt_ 15y ago
  He's refering to this (example): https://en.wikipedia.org/wiki/Beck_Depression_Inventory#Limitations https://en.wikipedia.org/wiki/Beck_Depression_Inventory#Limi... The BDI is copyrighted, a fee must be paid for each copy used, and photocopying it is a violation of copyright.
  - iandanforth 15y ago
    Correct. A portion of the original study involved a personality measure like big five (http://en.wikipedia.org/wiki/Big_Five_personality_traits http://en.wikipedia.org/wiki/Big_Five_personality_traits). The tests for this are almost all under copyright.
- dubya 15y ago
  Probably something like the Rorschach tests being copyrighted until recently. The actual test questions and the scoring rubric that, applied to a subject, yield a number or set of numbers.
endersshadow 15y ago
As a layman, your three points terrify me about the state of science. Well, numbers two and three do. There are still people addicted to video games. WoW and Minecraft probably moreso than Asheron's Call back in 2003. As to point number two: Are these questionnaires or something that are used to get psychological data? Why would those not be accessible--don't you need to know the questions asked to determine if the results are valid? As to point three: Data cleaning?! I would understand throwing out invalid results (i.e.-you find out somebody's lying, errors with data collection, etc), but to throw out results because they look like they don't belong feels disingenuous. I've done a lot of work with public company financials and operational systems. I cannot, on my worst day, imagine telling the SEC, "Yeah, well that return was really abnormal and was an outlier, so we decided not to record it." Why are these data points thrown out? To me, again, a layman, it appears that if this is standard practice, these experiments are starting with a conclusion and then just going through the motions to get that conclusion published. Why are data cleansing and secrecy normal practices? What about the scientific method don't I understand?
- iandanforth 15y ago
  The line between an invalid result and one that looks like it doesn't belong is fuzzy. I believe I was removing the former, but if an independent replicator doesn't have access to the raw data they would never discover that I had been fooling myself. As to secrecy, there is nothing being hidden, instead it's that nothing in the system requires you to publish all your data, and so because you're pressed for time you don't. Never ascribe to malice what laziness can explain :)
  - endersshadow 15y ago
    I'm not ascribing this to malice. It strikes me as a paragon of negligence if that's the way that the science of psychology is being done. In fact, it strikes me as an affront to science. Take this in contrast to the neutrinos that were going faster than light at CERN. Instead of going, "Okay, well, those can't happen, so those are outliers," they reported their results, published their methods and data, and said to the world, "Help us validate this." It seems to me like the modern psychology scientific method is the antithesis of this--"These are the results of my massaged data, and no you can't see how I got them, just trust me." I'm not attributing this to malice, I'm attributing this to negligence and laziness. Putting your work in front of people for critique is hard. Really hard. But, it's part of science. If psychology as a field has convinced themselves that they're above that reproach, then I think that's a huge condemnation of the field. When I first read the article, I thought it was a bit sensationalist to call psychology "under attack," but now I'm not so sure. I've assumed that reproducability is the standard litmus test of all science, but I guess not, and that leaves me with a bitter taste in my mouth toward the field of psychology.
mcguire 15y ago
"It would be disappointing to impune the work of scientists" I hate to point this out, but I've never even heard of any attempt to "impune the work" of any scientist that is anywhere near as damning as what you wrote above. You've described your work as temporary, local, and proprietary, and you've discarded the actual data. Being unable to replicate it is the least of the issues.
- deleted 15y ago
  [deleted]

k4st 15y ago

Note: I am not a psychologist. What do you mean by "some tests are copyrighted"? Are you talking about a statistical method being copyrighted, an implementation thereof, or something else entirely?

ttt_ 15y ago
He's refering to this (example): https://en.wikipedia.org/wiki/Beck_Depression_Inventory#Limitations https://en.wikipedia.org/wiki/Beck_Depression_Inventory#Limi... The BDI is copyrighted, a fee must be paid for each copy used, and photocopying it is a violation of copyright.
- iandanforth 15y ago
  Correct. A portion of the original study involved a personality measure like big five (http://en.wikipedia.org/wiki/Big_Five_personality_traits http://en.wikipedia.org/wiki/Big_Five_personality_traits). The tests for this are almost all under copyright.
dubya 15y ago
Probably something like the Rorschach tests being copyrighted until recently. The actual test questions and the scoring rubric that, applied to a subject, yield a number or set of numbers.

endersshadow 15y ago

As a layman, your three points terrify me about the state of science. Well, numbers two and three do. There are still people addicted to video games. WoW and Minecraft probably moreso than Asheron's Call back in 2003. As to point number two: Are these questionnaires or something that are used to get psychological data? Why would those not be accessible--don't you need to know the questions asked to determine if the results are valid? As to point three: Data cleaning?! I would understand throwing out invalid results (i.e.-you find out somebody's lying, errors with data collection, etc), but to throw out results because they look like they don't belong feels disingenuous. I've done a lot of work with public company financials and operational systems. I cannot, on my worst day, imagine telling the SEC, "Yeah, well that return was really abnormal and was an outlier, so we decided not to record it." Why are these data points thrown out? To me, again, a layman, it appears that if this is standard practice, these experiments are starting with a conclusion and then just going through the motions to get that conclusion published. Why are data cleansing and secrecy normal practices? What about the scientific method don't I understand?

iandanforth 15y ago
The line between an invalid result and one that looks like it doesn't belong is fuzzy. I believe I was removing the former, but if an independent replicator doesn't have access to the raw data they would never discover that I had been fooling myself. As to secrecy, there is nothing being hidden, instead it's that nothing in the system requires you to publish all your data, and so because you're pressed for time you don't. Never ascribe to malice what laziness can explain :)
- endersshadow 15y ago
  I'm not ascribing this to malice. It strikes me as a paragon of negligence if that's the way that the science of psychology is being done. In fact, it strikes me as an affront to science. Take this in contrast to the neutrinos that were going faster than light at CERN. Instead of going, "Okay, well, those can't happen, so those are outliers," they reported their results, published their methods and data, and said to the world, "Help us validate this." It seems to me like the modern psychology scientific method is the antithesis of this--"These are the results of my massaged data, and no you can't see how I got them, just trust me." I'm not attributing this to malice, I'm attributing this to negligence and laziness. Putting your work in front of people for critique is hard. Really hard. But, it's part of science. If psychology as a field has convinced themselves that they're above that reproach, then I think that's a huge condemnation of the field. When I first read the article, I thought it was a bit sensationalist to call psychology "under attack," but now I'm not so sure. I've assumed that reproducability is the standard litmus test of all science, but I guess not, and that leaves me with a bitter taste in my mouth toward the field of psychology.

mcguire 15y ago

"It would be disappointing to impune the work of scientists" I hate to point this out, but I've never even heard of any attempt to "impune the work" of any scientist that is anywhere near as damning as what you wrote above. You've described your work as temporary, local, and proprietary, and you've discarded the actual data. Being unable to replicate it is the least of the issues.

deleted 15y ago
[deleted]