4 ms·

>"When you only have 2-5 people that _actually_ read a paper" Even when a paper goes through a vigilant, rigorous peer review, it relies upon the data that the

>"When you only have 2-5 people that _actually_ read a paper" Even when a paper goes through a vigilant, rigorous peer review, it relies upon the data that the research team supplied. Over and over again cases like this have encountered manipulated data. Humans are flawed creatures, and if you spent a lot of time and professional credibility on a hypothesis, there is a strong motive to find what you sought. Doubly so if the results are salacious or contrarian in some way and thus get viral attention. Just convince yourself that it's the data that was wrong somehow and you know your assumptions are right so just this One Time you need to do a little manipulation. A study I have seen cited on here countless times is the "honesty pledge" one by Ariely et al. It was the one that claimed that when a person signs a form at the beginning, they're more honest. It was complete and utter BS, based entirely on fabricated data. It joins an infamous list of studies that have had enormous influence (especially if they have an "aha!" factor -- if it is the sort of thing that Malcolm Gladwell would talk about, consider it suspect) but were the creation of someone making up data in Excel.

bumby 3y ago
If I'm not mistaken, Ariely is asked about this on the "Armchair Expert" podcast. His claim was that he did not manipulate the data personally and it was someone further upstream whom he trusted. His point was that at some point, trust has to enter the process, except in the rare cases where a single person is doing all the research themselves (apologies if I'm misremembering this, but I think the following point still stands.) IMO the researcher still has some responsibility because ultimately, it's their research. So the questions to me are: 1) How much due diligence is reasonable? Does it change depending on the source? For example, is it more/less reasonable to accept government-provided data at face value vs. data collected by an undergraduate? 2) What processes can be implemented to safeguard data manipulation? I know there is a movement to provide data with peer-reviewed submittals, but it's still a low probability that a peer-reviewer has the time or inclination to really dive into the data to assess the claims.
- nvm0n2 3y ago
  Ariely tried to frame both his assistant and the insurance company he got the data from. The insurers provided the same data to a journalist who found there had been very substantial alterations made. The assistant showed that the Excel metadata indicated Ariely was the last to edit it.
  - bumby 3y ago
    Yeah, I read the same digging a bit more after I made that original comment. It’s much more damning than what he made it sound like in the interview, although it didn’t say how substantial his edits were.
    - nvm0n2 3y ago
      Very substantial, to the extent that the dataset was made up. The original data obtained from the insurer showed no effect. The more interesting thing though is why he chose to investigate this question in the first place and why he chose to do fraud to make it seem true. The hypothesis is a very weird one and there's no reason to think it would hold. Unless that is you think of people as being child-like lumps of Playdough, so easily manipulated that trivialities like where exactly something appears on a form can yield huge behavioural differences. That belief is the only reason you'd ever come up with such a hypothesis, and I think it's not really surprising that someone like that would engage in fraud. After all they have spent months (or years?) on trying to prove that people's levels of honesty are trivially controlled by psychologists like yourself. If you believe that's true then why wouldn't you commit fraud? After all you can easily manipulate people into not noticing it.
      - bumby 3y ago
        The manipulation seems substantial, but the point I was alluding to was that there wasn’t a smoking gun (at least by the amounts that I’ve heard and read) that Ariely made those substantial changes. I’m not sure what was included in Excel metadata, but it’s at least conceivable that the copy/paste + random change edits were done by someone else prior to Ariely edits. Where it gets damning is that he was the one the original dataset was sent to and the last to edit it. At the very least, it shows a lack of due diligence in not catching there were somehow many more datapoints added. Just conjecture of course, but this came at a time when governmental “nudges” were very en vogue. I could see where successful research could be thought of AAA a pathway to influence, prestige, and money through government grants and appointments. And there were some highly regarded behavioral psychologists who were substantiating its effectiveness.
robswc 3y ago
Oh for sure. It would be very easy to determine if the data is bad (in most cases) if someone tells you where to look. It is almost impossible to expect people to put their mental resources to trying to "crack" every paper that comes across their desk tho... so we have a bit of a problem, to say the least.

bumby 3y ago

If I'm not mistaken, Ariely is asked about this on the "Armchair Expert" podcast. His claim was that he did not manipulate the data personally and it was someone further upstream whom he trusted. His point was that at some point, trust has to enter the process, except in the rare cases where a single person is doing all the research themselves (apologies if I'm misremembering this, but I think the following point still stands.) IMO the researcher still has some responsibility because ultimately, it's their research. So the questions to me are: 1) How much due diligence is reasonable? Does it change depending on the source? For example, is it more/less reasonable to accept government-provided data at face value vs. data collected by an undergraduate? 2) What processes can be implemented to safeguard data manipulation? I know there is a movement to provide data with peer-reviewed submittals, but it's still a low probability that a peer-reviewer has the time or inclination to really dive into the data to assess the claims.

nvm0n2 3y ago
Ariely tried to frame both his assistant and the insurance company he got the data from. The insurers provided the same data to a journalist who found there had been very substantial alterations made. The assistant showed that the Excel metadata indicated Ariely was the last to edit it.
- bumby 3y ago
  Yeah, I read the same digging a bit more after I made that original comment. It’s much more damning than what he made it sound like in the interview, although it didn’t say how substantial his edits were.
  - nvm0n2 3y ago
    Very substantial, to the extent that the dataset was made up. The original data obtained from the insurer showed no effect. The more interesting thing though is why he chose to investigate this question in the first place and why he chose to do fraud to make it seem true. The hypothesis is a very weird one and there's no reason to think it would hold. Unless that is you think of people as being child-like lumps of Playdough, so easily manipulated that trivialities like where exactly something appears on a form can yield huge behavioural differences. That belief is the only reason you'd ever come up with such a hypothesis, and I think it's not really surprising that someone like that would engage in fraud. After all they have spent months (or years?) on trying to prove that people's levels of honesty are trivially controlled by psychologists like yourself. If you believe that's true then why wouldn't you commit fraud? After all you can easily manipulate people into not noticing it.
    - bumby 3y ago
      The manipulation seems substantial, but the point I was alluding to was that there wasn’t a smoking gun (at least by the amounts that I’ve heard and read) that Ariely made those substantial changes. I’m not sure what was included in Excel metadata, but it’s at least conceivable that the copy/paste + random change edits were done by someone else prior to Ariely edits. Where it gets damning is that he was the one the original dataset was sent to and the last to edit it. At the very least, it shows a lack of due diligence in not catching there were somehow many more datapoints added. Just conjecture of course, but this came at a time when governmental “nudges” were very en vogue. I could see where successful research could be thought of AAA a pathway to influence, prestige, and money through government grants and appointments. And there were some highly regarded behavioral psychologists who were substantiating its effectiveness.

robswc 3y ago

Oh for sure. It would be very easy to determine if the data is bad (in most cases) if someone tells you where to look. It is almost impossible to expect people to put their mental resources to trying to "crack" every paper that comes across their desk tho... so we have a bit of a problem, to say the least.