8 ms·

A paper’s central finding was based on polling that probably never happened

74 points by remarkEon 11y ago

caminante 11y ago
Here's the scary part: As Peter Winker, an economist at the University of Giessen in Germany who co-edited a book about poll fraud, put it in an email: “If the faker would have been a bit cleverer in this procedure, I doubt that the fraud would have been found this way.”
KurtMueller 11y ago
There should be a Github for Academia where scientists open source their data & formulas the use to draw conclusions - people could comment and annotate on specific parts, fork & create pull requests, and file any issues they might have.
- randomsearch 11y ago
  There is widespread provision of data and software on computing science papers in academia. It's currently a piecemeal approach, which is a problem, but here's an example of how things are changing: http://conf.researchr.org/track/pldi2015/PLDI+2015+Artifact+Evaluation http://conf.researchr.org/track/pldi2015/PLDI+2015+Artifact+... Universities all over the world are working on repositories to store electronic resources such as data and code for the long-term. This is non-trivial, as data may need to be available for decades or even hundreds of years. Github, by contrast, has been around for 7 years. Regarding the idea of organising research in a more change-control fashion, that's a more radical change. Academia will move slowly, and rightly so, as research practices evolve and persists for hundreds of years; to adopt an idea based on a (relatively, very) new method of software development is hugely risky.
- JupiterMoon 11y ago
  This is called peer reviewed publication... It has happened for a few centuries now. It's not perfect and it can certainly be improved but it does exist.
- NotOscarWilde 11y ago
  Indeed, but it is critical to me (and I think I'm not the only one in academia for which it is so) that it uses the arXiv funding model -- donations to be sustainable -- and not the Github/Authorea model, where you pay for each private paper/repository you set up. --- It is interesting to note that as a programmer, I do not have that many issues with releasing partial or incomplete source code on Github, whereas as a computer scientist, I do not like to release partial or incomplete proofs at all. It might be because the academic community cares about reputation: if you get a reputation as a researcher that releases incorrect proofs, your career is over. In programming, as the names are attached to project much more loosely, such negative reputation is rare and people care much less.
- cyphunk 11y ago
  Not at all. There should bit git4chan for Academia where you can submit articles but never obtain attribution.
haberman 11y ago
Even when data isn't outright fabricated, it often seems that studies like this are looking for a predetermined outcome that will reinforce the preconceived ideas of the authors. I feel so skeptical anytime a study is published that purports to show the truth of some idea that people would really want to believe ("if only they just met a gay person it would change their mind.")
- pedrocr 11y ago
  >it often seems that studies like this are looking for a predetermined outcome that will reinforce the preconceived ideas of the authors. Isn't this the definition of how science should work. 1) Formulate an hypothesis, 2) design an experiment to prove it 3) and then run it rigorously and accept whatever is the result. This case (and many others) failed 3) but everything else seems more than natural. Physicists smashing particles at the LHC were also trying to "reinforce their preconceived ideas" of how particles work. They have the standard model and they want to validate it where possible.
  - remarkEon 11y ago
    This is my sentiment exactly, and why I posted the original article and read the refuting paper [0]. There's been a flurry of posts on HN recently about hypothesis testing and people not understanding what p-values are and what these statistics really mean, and I feel like that sits within the same critique about this paper. People who manipulate their model spec to get a convenient p-value are just as guilty as those who completely fabricate data. [0] http://web.stanford.edu/~dbroock/broockman_kalla_aronow_lg_irregularities.pdf http://web.stanford.edu/~dbroock/broockman_kalla_aronow_lg_i...
  - datenwolf 11y ago
    > Isn't this the definition of how science should work. 1) Formulate an hypothesis, 2) design an experiment to prove it 3) and then run it rigorously and accept whatever is the result. No. The critical difference lies in steps 2 and 3. You can never prove something in science, because you'll never get all the corner cases. What you're really after is designing an experiment that is able to falsify (=disprove) your hypothesis and run that. Because if you find one single instance that contradicts the original hypothesis, you know that your hypothesis in its current form is wrong, because it can't be applied universally.
hudibras 11y ago
It's a little hard to tease out from this particular article, but the amazing thing is that the canvassing actually happened: dozens of volunteers talked with hundreds of people at 22 minutes for each conversation. Easily thousands of man-hours of work. But the grad student running the study didn't do the easy part, hiring a survey company to poll and measure the before-and-after attitudes of the people being studied. It's one thing to get data and then fraudulently change it to fit a story; at least then other people can go back and un-do the damage. It's a different thing to not do anything at all, completely make up data, and waste all that hard work.
thaumasiotes 11y ago
> LaCour said he couldn’t find files containing the survey data and indicated that he would write a retraction; when he didn’t, Green sent off his. Science hasn’t yet ruled on whether it will retract the original paper. Sure, the paper's author may want you to know that it was all faked... but Science stands by what they printed, no matter what.
- jacquesm 11y ago
  Retracting the paper would for Science be an admission that they have a less than stellar peer-review process and would cast a negative light on anything else Science has published. Never mind that not retracting the paper and publishing the retraction makes them look even worse.
  - caminante 11y ago
    You raise an overlooked issue. I'm curious what discussions Science (journal) is having about damage control.
- jellicle 11y ago
  If you read the report, the retraction letter was sent to Science, presumably by email, on May 19, 2015. I'm okay with Science giving some due process here. Contacting the primary author and other parties, the peer reviewers, discussing it with their editorial staff, and so on. The world isn't going to end because Science hasn't yet made a decision on May 21 about something sent to them on May 19. The odds are exceedingly high that a retraction will be issued. Your characterization of Science "standing by what they printed no matter what" is a gross distortion of the facts here, as Science has done no such thing.
  - thaumasiotes 11y ago
    > Your characterization of Science "standing by what they printed no matter what" is a gross distortion of the facts here, as Science has done no such thing. I have to disagree; I think jacquesm has the comment closest to the truth. My comment is a minor distortion of the facts -- later in the article we get this: > Science tweeted that it was assessing the retraction request and would in the meantime publish “an Editorial Expression of Concern.” Here's the thing. Science isn't doing any research itself. Once a paper's author has admitted the data was faked, there is no role for Science to play; they can't look into the past and determine that it was actually real. They've acknowledged receipt of the message, and they're willing to print a notice that the paper is a lie -- but they don't want to retract it, because that would damage their record. > The world isn't going to end because Science hasn't yet made a decision on May 21 about something sent to them on May 19. They have made a decision. They're going to print an unofficial warning. Without more pressure from the public, they will not print a retraction. Remember, Green is the primary author. He contacted them; for them to contact him is somewhat superfluous.
Fede_V 11y ago
The incentive system in science is immensely screwed up. Researchers are rewarded not for high quality work flows, reproducible results, and solid methodology but for neat, linear stories with a simple high impact result. Imagine that a scientist designs a very intelligent experiment to test a plausible hypothesis - she has an excellent plan to analyze the results (pre-registered, to avoid p-value hacking) - and after doing the experiment, she finds no evidence of the effect that she was looking for. Why should the result (which is the only thing the scientist is not in control of) determine the reward given to the scientist? Funding should be given to investigate a particular phenomena, not to find high impact results.
- jacquesm 11y ago
  The reason funding goes to those that find 'good' results is the same reason why funding goes to those that start successful start-ups. Even though both parties probably have less control over the outcome than they think the perception is that they do and that betting on them again is a better bet than betting on someone that did not succeed to score. That's how we're wired, we simply like success stories much more than that we like stories of failure.
  - digi_owl 11y ago
    Just world?
    - verbin217 11y ago
      False inherencies are the staple of those that wish to understand the world such that they might exploit it. If they weren't prepared to demonstrate that the pursuit of a solution was necessarily a fool's game then they'd have to give an actual shit.
  - pbhjpbhj 11y ago
    The analogy doesn't completely work. If the data shows your hypothesis is false, or doesn't support its truth then it doesn't matter if you're the best experimenter in your field nor if you worked most competently the world is what it is (barring poor experiment design!). An hypothesis isn't a product, falsifying your hypothesis is success - you've increased the pool of human knowledge (or increased the support for an element of it). In business good and bad products can both lead to profit and can both lead to losses. Profit and loss can both be considered success and failure. On your last sentence - I don't think it's as self-evident as it seems. We tend to like the successes that profit us but in general I'm not sure we humans are that congratulatory of other's success. Schadenfreude seems very popular.
deleted 11y ago
[deleted]
ylem 11y ago
The actual report showing the debunking is quite good! http://web.stanford.edu/~dbroock/broockman_kalla_aronow_lg_irregularities.pdf http://web.stanford.edu/~dbroock/broockman_kalla_aronow_lg_i... This was a relatively high profile paper. I wonder how much slips through in lower impact journals...

74 points by remarkEon 11y ago

Here's the scary part: As Peter Winker, an economist at the University of Giessen in Germany who co-edited a book about poll fraud, put it in an email: “If the faker would have been a bit cleverer in this procedure, I doubt that the fraud would have been found this way.”

KurtMueller 11y ago

There should be a Github for Academia where scientists open source their data & formulas the use to draw conclusions - people could comment and annotate on specific parts, fork & create pull requests, and file any issues they might have.

randomsearch 11y ago
There is widespread provision of data and software on computing science papers in academia. It's currently a piecemeal approach, which is a problem, but here's an example of how things are changing: http://conf.researchr.org/track/pldi2015/PLDI+2015+Artifact+Evaluation http://conf.researchr.org/track/pldi2015/PLDI+2015+Artifact+... Universities all over the world are working on repositories to store electronic resources such as data and code for the long-term. This is non-trivial, as data may need to be available for decades or even hundreds of years. Github, by contrast, has been around for 7 years. Regarding the idea of organising research in a more change-control fashion, that's a more radical change. Academia will move slowly, and rightly so, as research practices evolve and persists for hundreds of years; to adopt an idea based on a (relatively, very) new method of software development is hugely risky.
JupiterMoon 11y ago
This is called peer reviewed publication... It has happened for a few centuries now. It's not perfect and it can certainly be improved but it does exist.
NotOscarWilde 11y ago
Indeed, but it is critical to me (and I think I'm not the only one in academia for which it is so) that it uses the arXiv funding model -- donations to be sustainable -- and not the Github/Authorea model, where you pay for each private paper/repository you set up. --- It is interesting to note that as a programmer, I do not have that many issues with releasing partial or incomplete source code on Github, whereas as a computer scientist, I do not like to release partial or incomplete proofs at all. It might be because the academic community cares about reputation: if you get a reputation as a researcher that releases incorrect proofs, your career is over. In programming, as the names are attached to project much more loosely, such negative reputation is rare and people care much less.
cyphunk 11y ago
Not at all. There should bit git4chan for Academia where you can submit articles but never obtain attribution.

haberman 11y ago

Even when data isn't outright fabricated, it often seems that studies like this are looking for a predetermined outcome that will reinforce the preconceived ideas of the authors. I feel so skeptical anytime a study is published that purports to show the truth of some idea that people would really want to believe ("if only they just met a gay person it would change their mind.")

pedrocr 11y ago
>it often seems that studies like this are looking for a predetermined outcome that will reinforce the preconceived ideas of the authors. Isn't this the definition of how science should work. 1) Formulate an hypothesis, 2) design an experiment to prove it 3) and then run it rigorously and accept whatever is the result. This case (and many others) failed 3) but everything else seems more than natural. Physicists smashing particles at the LHC were also trying to "reinforce their preconceived ideas" of how particles work. They have the standard model and they want to validate it where possible.
- remarkEon 11y ago
  This is my sentiment exactly, and why I posted the original article and read the refuting paper [0]. There's been a flurry of posts on HN recently about hypothesis testing and people not understanding what p-values are and what these statistics really mean, and I feel like that sits within the same critique about this paper. People who manipulate their model spec to get a convenient p-value are just as guilty as those who completely fabricate data. [0] http://web.stanford.edu/~dbroock/broockman_kalla_aronow_lg_irregularities.pdf http://web.stanford.edu/~dbroock/broockman_kalla_aronow_lg_i...
- datenwolf 11y ago
  > Isn't this the definition of how science should work. 1) Formulate an hypothesis, 2) design an experiment to prove it 3) and then run it rigorously and accept whatever is the result. No. The critical difference lies in steps 2 and 3. You can never prove something in science, because you'll never get all the corner cases. What you're really after is designing an experiment that is able to falsify (=disprove) your hypothesis and run that. Because if you find one single instance that contradicts the original hypothesis, you know that your hypothesis in its current form is wrong, because it can't be applied universally.

hudibras 11y ago

It's a little hard to tease out from this particular article, but the amazing thing is that the canvassing actually happened: dozens of volunteers talked with hundreds of people at 22 minutes for each conversation. Easily thousands of man-hours of work. But the grad student running the study didn't do the easy part, hiring a survey company to poll and measure the before-and-after attitudes of the people being studied. It's one thing to get data and then fraudulently change it to fit a story; at least then other people can go back and un-do the damage. It's a different thing to not do anything at all, completely make up data, and waste all that hard work.

thaumasiotes 11y ago

> LaCour said he couldn’t find files containing the survey data and indicated that he would write a retraction; when he didn’t, Green sent off his. Science hasn’t yet ruled on whether it will retract the original paper. Sure, the paper's author may want you to know that it was all faked... but Science stands by what they printed, no matter what.

jacquesm 11y ago
Retracting the paper would for Science be an admission that they have a less than stellar peer-review process and would cast a negative light on anything else Science has published. Never mind that not retracting the paper and publishing the retraction makes them look even worse.
- caminante 11y ago
  You raise an overlooked issue. I'm curious what discussions Science (journal) is having about damage control.
jellicle 11y ago
If you read the report, the retraction letter was sent to Science, presumably by email, on May 19, 2015. I'm okay with Science giving some due process here. Contacting the primary author and other parties, the peer reviewers, discussing it with their editorial staff, and so on. The world isn't going to end because Science hasn't yet made a decision on May 21 about something sent to them on May 19. The odds are exceedingly high that a retraction will be issued. Your characterization of Science "standing by what they printed no matter what" is a gross distortion of the facts here, as Science has done no such thing.
- thaumasiotes 11y ago
  > Your characterization of Science "standing by what they printed no matter what" is a gross distortion of the facts here, as Science has done no such thing. I have to disagree; I think jacquesm has the comment closest to the truth. My comment is a minor distortion of the facts -- later in the article we get this: > Science tweeted that it was assessing the retraction request and would in the meantime publish “an Editorial Expression of Concern.” Here's the thing. Science isn't doing any research itself. Once a paper's author has admitted the data was faked, there is no role for Science to play; they can't look into the past and determine that it was actually real. They've acknowledged receipt of the message, and they're willing to print a notice that the paper is a lie -- but they don't want to retract it, because that would damage their record. > The world isn't going to end because Science hasn't yet made a decision on May 21 about something sent to them on May 19. They have made a decision. They're going to print an unofficial warning. Without more pressure from the public, they will not print a retraction. Remember, Green is the primary author. He contacted them; for them to contact him is somewhat superfluous.

Fede_V 11y ago

The incentive system in science is immensely screwed up. Researchers are rewarded not for high quality work flows, reproducible results, and solid methodology but for neat, linear stories with a simple high impact result. Imagine that a scientist designs a very intelligent experiment to test a plausible hypothesis - she has an excellent plan to analyze the results (pre-registered, to avoid p-value hacking) - and after doing the experiment, she finds no evidence of the effect that she was looking for. Why should the result (which is the only thing the scientist is not in control of) determine the reward given to the scientist? Funding should be given to investigate a particular phenomena, not to find high impact results.

jacquesm 11y ago
The reason funding goes to those that find 'good' results is the same reason why funding goes to those that start successful start-ups. Even though both parties probably have less control over the outcome than they think the perception is that they do and that betting on them again is a better bet than betting on someone that did not succeed to score. That's how we're wired, we simply like success stories much more than that we like stories of failure.
- digi_owl 11y ago
  Just world?
  - verbin217 11y ago
    False inherencies are the staple of those that wish to understand the world such that they might exploit it. If they weren't prepared to demonstrate that the pursuit of a solution was necessarily a fool's game then they'd have to give an actual shit.
- pbhjpbhj 11y ago
  The analogy doesn't completely work. If the data shows your hypothesis is false, or doesn't support its truth then it doesn't matter if you're the best experimenter in your field nor if you worked most competently the world is what it is (barring poor experiment design!). An hypothesis isn't a product, falsifying your hypothesis is success - you've increased the pool of human knowledge (or increased the support for an element of it). In business good and bad products can both lead to profit and can both lead to losses. Profit and loss can both be considered success and failure. On your last sentence - I don't think it's as self-evident as it seems. We tend to like the successes that profit us but in general I'm not sure we humans are that congratulatory of other's success. Schadenfreude seems very popular.

deleted 11y ago

[deleted]

ylem 11y ago

The actual report showing the debunking is quite good! http://web.stanford.edu/~dbroock/broockman_kalla_aronow_lg_irregularities.pdf http://web.stanford.edu/~dbroock/broockman_kalla_aronow_lg_i... This was a relatively high profile paper. I wonder how much slips through in lower impact journals...