18 ms·

A bot crawled thousands of studies looking for simple math errors

421 points by MollyR 10y ago

thampiman 10y ago
This coupled with the Automatic Statistician (https://www.automaticstatistician.com/index/ https://www.automaticstatistician.com/index/) will help fix a lot of biases and human errors that creep into scientific research.
alanfalcon 10y ago
I find it very disconcerting that people are trying to fend off criticism of previously published studies by calling it "bullying" or sometimes worse. What do feelings have to do with science?
- mratzloff 10y ago
  Right. If you can't handle critique (or embarrassment from invalidated conclusions), don't publish. Write in your diary instead.
  - nkrisc 10y ago
    Exactly. This isn't even subjective critique, it's math. Your math is right or it's wrong. Lastly from what I can tell there isn't even any value judgement attached, simply a notification of incorrect math.
    - hoopism 10y ago
      This was my thought exactly. The most disturbing aspect to this whole deal is the response of the administrators who I assume represent a good deal of the community. The researches who contributed this project ought to be thanked and rewarded for their participation... not called terrorists.
- merpnderp 10y ago
  Objectivity is listed as a problem with science curriculum. Welcome to the future. http://nsuworks.nova.edu/cgi/viewcontent.cgi?article=2467&context=tqr http://nsuworks.nova.edu/cgi/viewcontent.cgi?article=2467&co...
  - Practicality 10y ago
    While I am not a big fan of contemporary feminism, I am pretty sure taking issue with masculinity or "gendered" text is not the same thing as taking issue with objectivity. Edit: Found this quote in your source "Poststructuralism “rejects objectivity and the notions of an absolute truth and single reality,”" Hmm, this seems an awful lot like a religion.
    - hx87 10y ago
      It's pretty trivial to take the Ibn Warraq approach, though: just reject poststructuralism itself.
    - zeveb 10y ago
      > I am pretty sure taking issue with masculinity or "gendered" text is not the same thing as taking issue with objectivity. Well, since English is a gendered language, taking issue with 'gendered' text is taking issue with the objective reality that … English is a gendered language.
      - alphonsegaston 10y ago
        Aside from things like pronouns, English hasn't been gendered since the 14th century: https://en.m.wikipedia.org/wiki/Gender_in_English https://en.m.wikipedia.org/wiki/Gender_in_English
        zeveb 10y ago
        'Aside from that Mrs. Lincoln, how was the play?' Pronouns are precisely what folks who complain about gender in English are complaining about (well, that and words like 'mankind').
- WillAbides 10y ago
  > What do feelings have to do with science? When the science in question is psychology, feelings have everything to do with it.
  - lucb1e 10y ago
    Studying feelings does not make feelings a part of studying.
- deleted 10y ago
  [deleted]
- dahart 10y ago
  When I read that sentence, I immediately thought it sounded like some people were being childish and defensive. OTOH, a bunch of people got unsolicited error reports for already published papers, and I can understand being initially irritated. This was, by the admission of the statcheck authors, a way to get attention quickly. My guess is it'll all settle down and people won't complain about bullying once everyone just uses statcheck like they do spell-check. To answer your question though, I think feelings have everything to do with science. There are many reasons different people do science, and they all stem from emotions. Some people do science out of curiosity, to solve mysteries. Some do science as a means to an end, to gain knowledge required to further some other goal. Some do science to gain social standing & intellectual superiority. Some do science to help others understand how the world works. In all cases, the reasons people are doing science is because of a want, some kind of desire to achieve a goal. Nature and physics will continue to exist whether we explore it or not, we do so because we care, and caring is a feeling.
  - icantdrive55 10y ago
    And there's people/corporations/institutions who do science to further their agenda. This is a huge part of science. I wish it wasn't. I welcome any program that looks for errors. Now let's expose the bad studies, or bad science? I welcome any program that makes this job easier. I wish I wasen't so cynical towards science. I have just been duped too many times.
  - Mithaldu 10y ago
    > unsolicited error reports for already published papers, and I can understand being initially irritated Please don't see this as me attacking you, i am genuinely baffled by what you said and would like to understand your thinking. How exactly could you possibly have any sort of understanding for that? How would you feel about a software author being irritated by unsolicited error reports? If that irritation is less acceptable than the one in your quote, where would lie the difference?
    - dahart 10y ago
      > How exactly could you possibly have any sort of understanding for that? I'm also confused by your question, so I can relate to you. :) Despite your warning, the way you phrased that question does sound like an attack, it implies that you know my experience does not allow me to speak on this subject, which you do not know. Why is it hard to think that I can empathize? What do you think you know about me that makes it implausible for me to understand this situation? I can and do relate to it because I'm a published paper author. I imagine that it would be irritating to me as easily as I imagine it would be irritating to others. For me personally, I don't think it would make me angry, but it would give me anxiety to cast doubt on a paper I'd already published, even if the report about inaccuracies are true. It would mean that I didn't do as good a job as I thought, which of course I want to know, but a published paper is part of a record that cannot change. I'm sure many paper authors, for better or worse, have the same reaction, that something that casts doubt on their publications after the fact would cause some degree of mental anguish. That doesn't mean we shouldn't search for the truth, it means only what I said, that I can understand the reaction. Your analogy to software errors fails. (And I also have first-hand experience with this as the owner of a software business.) Software is an on-going work that can and should be improved to remove errors at all times, and errors that get fixed do not affect my personal reputation or career. Reports of software errors can also be irritating in their own way - I don't want to know my software is buggy - but they are always welcome. Published papers cannot be improved, they are fixed in the permanent record. There are some ways to recover from severe errors, but there are no ways to recover from minor errors, and public (academic) perception of the quality and level of the errors in a paper can change the way an author is viewed.
- hx87 10y ago
  Even if feelings have everything to do with science, isn't getting errors in your paper pointed out a good thing that you should be happy about?
- douche 10y ago
  I guess I can accuse my compiler of bullying me.
  - virgulino 10y ago
    I've been saying that for years!!!
  - mturmon 10y ago
    Bully it back with: $ cc test.c || sudo rm -f $(which cc) "Listen, cc, we can do this the easy way, or the hard way..."
    - marcosdumay 10y ago
      Just make sure you don't make a mistake on that command. You don't want to get into MAD with the shell.
- deleted 10y ago
  [deleted]
- trjordan 10y ago
  They weren't looking for errors in papers and trying to push for corrections or improve the science. They were using these errors as fodder for promoting their software. It might not be bullying, but it's not coming from a place of objectivity or honest debate.
  - Consultant32452 10y ago
    Yeah, this is an abusive marketing technique. If you don't want future embarrassment you'd better buy our service.
    - oldmanjay 10y ago
      Or just be more vigilant and less defensive, both of which improve the practice of science
      - Consultant32452 10y ago
        That's just not how polite human interaction works. If you become aware that someone has made a mistake you inform them privately to give them an opportunity to rectify that mistake. In the case of a scientific journal entry you might instead publish a response article. You don't put everyone on blast and practically blackmail them into purchasing your for-profit service. It was never about the science at all, it was about the profits for this product/service.
- cafebeen 10y ago
  Well, scientists are people too, so they have feelings and are not perfect. I think there are polite ways provide criticism and corrections, hopefully without humiliating people with good intentions. If it's a simple mistake, they can email the author and suggest an erratum. If it's more serious, then it's common to write a response in the journal. I think more public forms of criticism are perceived as bullying because a general audience can't typically judge the magnitude of the error, since they're not scientists working in the field, so it can be unjustly damaging to someone's reputation.
  - wyager 10y ago
    If someone objectively screwed up a published study, there's no room for feelings or social consideration. People might be wasting their precious time and money pursuing impossible goals based on the incorrect study. Let's say someone finds a mistake in a math paper. Do they politely ask the author to fix it? What happens if the author ignores them? The scientific community doesn't have time to coddle and pursue every mistaken author. Everyone makes mistakes, and the scientific community knows that, but those mistakes need to be brought into then open, not (maybe) resolved behind closed doors.
    - cortesoft 10y ago
      I think there is some nuance here; not all screw ups are equal, nor are all ways of correcting mistakes equal.
    - cafebeen 10y ago
      It's not coddling as much as professional courtesy. Mistakes are often brought up at conferences and in peer review, but most scientists within a specific research area try to be on good terms with one another and don't see value in publicly shaming colleagues for their mistakes.
      - wyager 10y ago
        There are two options here: Use a bot to find tons of mistakes automatically and risk coming across as rude. Or Let social trivialities dominate scientific discourse and let most of those mistakes go unchecked forever because there's no feasible way to "politely" address hundreds of authors who made mistakes and keep checking back to make sure they actually fixed the mistake. The former is clearly the preferable choice. Some individual scientists will suffer for it, but the scientific community as a whole will benefit greatly.
mratzloff 10y ago
Here's the GitHub page: https://github.com/MicheleNuijten/statcheck https://github.com/MicheleNuijten/statcheck And if you're curious how it works, as I was: Statcheck uses regular expressions to find statistical results in APA format. When a statistical result deviates from APA format, statcheck will not find it. The APA formats that statcheck uses are: t(df) = value, p = value; F(df1,df2) = value, p = value; r(df) = value, p = value; [chi]2 (df, N = value) = value, p = value (N is optional, delta G is also included); Z = value, p = value. All regular expressions take into account that test statistics and p values may be exactly (=) or inexactly (< or >) reported. Different spacing has also been taken into account.
- snerbles 10y ago
  > When a statistical result deviates from APA format, statcheck will not find it. Incentive for authors to obfuscate their math?
  - dahart 10y ago
    Pretty sure APA format is required for APA journals, but either way, fortunately, the desire on the part of authors to get papers published and on the part of reviewers to have standards and clear presentation outweighs anyone's desire to game statcheck. But you do make a great point in that statcheck could and should red-line papers that present no discernible stats, and provide links to the APA style guide!
  - schmichael 10y ago
    It'd be easier for them to not obfuscate and use the checker themselves prior to publishing.
    - snerbles 10y ago
      True, assuming no malfeasance on the part of the authors.
      - cortesoft 10y ago
michaelt 10y ago
I can certainly understand people being nervous about academic debate moving to social media. It would be a hassle for climate scientists if every paper got a brigade of climate change deniers criticising it and you had to respond to those criticisms. But this example - someone notifying you there's a mistake in your paper, when there really is a mistake? That seems like a strong argument /for/ academic debate via social media, not /against/ it.
deleted 10y ago
[deleted]
6stringmerc 10y ago
What a clever and, dare I say it, fantastically useful experiment! So much less harm than even "door knob twisting" type explorations - no, this was using published works and pretty much running them through a process to verify or not verify accuracy. Unsolicited? So what! As a practiced writer I make unsolicited judgments on language usage all the time. Are these people that completely write from their own minds and don't use a spell check or grammar check program of any sort before sending their material for editorial review? I'd strongly doubt it, because it's a tool to make communication more accurate. Math and formulas having a similar procedural check sounds quite constructive to me. It's not bullying to point out errors; it's bullying to use the existence of errors to belittle or insult a person. I don't see that happening here. Sure, it's a little sterile or "cold" in this fashion, but I think that's for the best if such a process / tool can gain acceptance. It just spits out results and I think that's all it should do. Neat to read about.
- SerLava 10y ago
  Certainly not bullying. It's actually shocking to me that these studies could have been published without running a similar check on all of the arithmetic. The technology to automate this has been possible for decades.
  - dclowd9901 10y ago
    Right. To my mind, this level of scrutiny is little different than a linter or static analyzer for a programming language.
- hueving 10y ago
  Putting grammar on the same level as this is wrong. Poor grammar and spelling doesn't frequently change the entire results of a paper. Math errors though can easily be the difference between significance and insignificance, changing the entire set of conclusions that can be drawn from the paper.
  - 6stringmerc 10y ago
    Uh, no, poor grammar or words or phrases used improperly can genuinely have cascade / domino effects. Brush up on "Eats, Shoots, and Leaves" if you wish to counter-balance your math-centric logic notions with how similar principles apply in written language. English is, without question, one of the more tricky (re: confusing) mutts of linguistics.
munificent 10y ago
> There’s a big, uncomfortable question of how to criticize past work, and whether online critiques of past work constitute “bullying” or shaming. Science is fundamentally reputation-driven. One of, if not the primary incentive that encourages scientists to do science work is the chance of raising their prestige. Citations are one very quantifiable yardstick for this. If positive social sanctions are a driving force for science, then it's entirely reasonable that negative sanctions should come into play too. If you can well-cited paper and attract fame, then a poor paper should likewise attract shame. Otherwise you have a positive feedback loop where once a scientist has attracted enough prestige, they are untouchable. We need negative feedback to balance that out.
- danharaj 10y ago
  Shaming becomes another power structure that allows the influential to bully their peers. It's not so simple as you suggest.
c3534l 10y ago
> and whether online critiques of past work constitute “bullying” or shaming. The PubPeer comments are just a tiny part of that debate. But it can hit a nerve. > Susan Fiske, a former president of the Association for Psychological Science, alluded to Statcheck in an interview with Business Insider, calling it a “gotcha algorithm.” > The draft of the article said these online critics were engaging in “methodological terrorism.” If these are attitudes typical of psychology, then I cannot say I consider psychology to be a proper social science. There is a fundamental misunderstanding of how knowledge is created through the scientific process if the verification step is considered to be offensive or taboo. That anyone in the field of psychology would even be comfortable publically espousing a non-scientific worldview like that means that psychologists are not being properly educated in the scientific method and should not be in the business of producing research since they do not have a mature understanding of what "scientific" implies.
- dmd 10y ago
  I took a class with Susan Fiske in grad school, and the following is a quote from one of her comments on a homework where I had called into question a paper that was clearly not just statistically incorrectly done but was based on urban-legend-level assumptions: "These results are from respected investigators, so it's inappropriate to question them."
  - ben_jones 10y ago
    While I trust your story, and I believe it's somewhat taboo on HN to ask for this, I feel that it would be much more valuable if evidence was provided. That is an absolutely egregious act for a professor to commit and I feel like it should be reported to someone if substantiated..
    - dmd 10y ago
      Oh, I made a huge stink about it (and this was just one example of many comments of that type). This was at Penn in 2007.
radarsat1 10y ago
The article sadly doesn't report on the false positive rate of statcheck. I assume the paper does? I mean, it just uses a basic regular expression, I can see it easily performing bad checks. I assume the authors take this into account.
RangerScience 10y ago
Haha! What if this is actually a marketing ploy for their web-app? Stir up some shit so everyone gets talking, and provide a service.
- EdHominem 10y ago
  And what if? Does that negate the results?
  - RangerScience 10y ago
    Oh no not at all! The results are still solid. So, "ploy" isn't the right word, not really. "Stunt"? "Technique"? I was just thinking - They could have found a way to email or otherwise contact the authors, instead of just posting comments, but posting comments (and getting some news on it) drives more traffic due to scandal effects. This rabbit hole can go deeper - what if private disclosure was expected to backfire? By surprising people publicly, they're more forced to admit there's an issue; and then by providing both the issue and a solution, there's an easy way at hand to fix it....
    - EdHominem 10y ago
      If I use software and there's a bug, I don't want to find out years later when the author decided to admit it, I want to find out right away so I can get it patched. Papers are equivalent to code, if I'm developing something based on one I want to know if the foundation is reliable or not. > what if private disclosure was expected to backfire? Well of course it could be expected to go badly. Look at all the "concern" spent over their methods already. But yes, it would have been interesting if they had contacted authors quietly and without telling them their methods, simply to see who updated their paper and who tried to blame the messenger... Then publish that list and let the field really correct itself.
deleted 10y ago
[deleted]
JBiserkov 10y ago
For more context, see http://andrewgelman.com/2016/09/21/what-has-happened-down-here-is-the-winds-have-changed/ http://andrewgelman.com/2016/09/21/what-has-happened-down-he...
vinchuco 10y ago
A good distinction between "peer reviewed" vs "computer verified" >“The literature is growing faster and faster, peer review is overstrained, and we need technology to help us out,” This is a problem in every field, not just Psychology. I want someone to tell me the distribution (or average ratio) of papers read to papers written. Every thesis written is supposed to add some delta to the state of the art. But there is no method for doing a diff between past and previous versions of human knowledge. How to make science less redundant and more efficient? I dream of aggregators for everything.
orionblastar 10y ago
I wanted to make a program like that, but considered the ethics of it.
serge2k 10y ago
> some found the emails annoying from PubPeer [since PubPeer notifies authors of comments] if Statcheck found no potential errors I would. > There’s a big, uncomfortable question of how to criticize past work, and whether online critiques of past work constitute “bullying” or shaming. It's facts about your work. Learn to handle it or quit pretending to be a scientist. > The gist of her article was that she feared too much of the criticism of past work had been ceded to social media, and that the criticism there is taking on an uncivil tone Valid enough point. Criticism and correction can be done in a civil manner, and in an accepted forum.
ChoHag 10y ago
Will I never guess what number 7 is? (DNR)
yakult 10y ago
While it is definitely to the benefit of all that the bot emails authors when it finds mistakes, emailing when it doesn't find anything is a dark pattern. Reminds me of those bots that spam me after scraping my linkedin.
l0b0 10y ago
Why does the article focus even for a paragraph on whether egos would be bruised? If the result is a general improvement to the readers' understanding that is, as far as I'm concerned, case closed. Good on them!
OliverJones 10y ago
Remember when writers could do spell-checking and grammar-checking by "running a program" on their text files? Here we have numbers-checking working the same way. I bet you this sort of feature gets built in to word processors eventually, and puts wavy red lines under the results it flags. We've had this sort of real-time "syntax" checking in software engineering for half a generation. It seems wise for other disciplines to consider adopting it too. It's obviously got to be discretionary, just like spell-check is discretionary in browsers. We will get a new genre of humor, thought "statcheck fail."

421 points by MollyR 10y ago

This coupled with the Automatic Statistician (https://www.automaticstatistician.com/index/ https://www.automaticstatistician.com/index/) will help fix a lot of biases and human errors that creep into scientific research.

alanfalcon 10y ago

I find it very disconcerting that people are trying to fend off criticism of previously published studies by calling it "bullying" or sometimes worse. What do feelings have to do with science?

mratzloff 10y ago
Right. If you can't handle critique (or embarrassment from invalidated conclusions), don't publish. Write in your diary instead.
- nkrisc 10y ago
  Exactly. This isn't even subjective critique, it's math. Your math is right or it's wrong. Lastly from what I can tell there isn't even any value judgement attached, simply a notification of incorrect math.
  - hoopism 10y ago
    This was my thought exactly. The most disturbing aspect to this whole deal is the response of the administrators who I assume represent a good deal of the community. The researches who contributed this project ought to be thanked and rewarded for their participation... not called terrorists.
merpnderp 10y ago
Objectivity is listed as a problem with science curriculum. Welcome to the future. http://nsuworks.nova.edu/cgi/viewcontent.cgi?article=2467&context=tqr http://nsuworks.nova.edu/cgi/viewcontent.cgi?article=2467&co...
- Practicality 10y ago
  While I am not a big fan of contemporary feminism, I am pretty sure taking issue with masculinity or "gendered" text is not the same thing as taking issue with objectivity. Edit: Found this quote in your source "Poststructuralism “rejects objectivity and the notions of an absolute truth and single reality,”" Hmm, this seems an awful lot like a religion.
  - hx87 10y ago
    It's pretty trivial to take the Ibn Warraq approach, though: just reject poststructuralism itself.
  - zeveb 10y ago
    > I am pretty sure taking issue with masculinity or "gendered" text is not the same thing as taking issue with objectivity. Well, since English is a gendered language, taking issue with 'gendered' text is taking issue with the objective reality that … English is a gendered language.
    - alphonsegaston 10y ago
      Aside from things like pronouns, English hasn't been gendered since the 14th century: https://en.m.wikipedia.org/wiki/Gender_in_English https://en.m.wikipedia.org/wiki/Gender_in_English
      - zeveb 10y ago
        'Aside from that Mrs. Lincoln, how was the play?' Pronouns are precisely what folks who complain about gender in English are complaining about (well, that and words like 'mankind').
WillAbides 10y ago
> What do feelings have to do with science? When the science in question is psychology, feelings have everything to do with it.
- lucb1e 10y ago
  Studying feelings does not make feelings a part of studying.
deleted 10y ago
[deleted]
dahart 10y ago
When I read that sentence, I immediately thought it sounded like some people were being childish and defensive. OTOH, a bunch of people got unsolicited error reports for already published papers, and I can understand being initially irritated. This was, by the admission of the statcheck authors, a way to get attention quickly. My guess is it'll all settle down and people won't complain about bullying once everyone just uses statcheck like they do spell-check. To answer your question though, I think feelings have everything to do with science. There are many reasons different people do science, and they all stem from emotions. Some people do science out of curiosity, to solve mysteries. Some do science as a means to an end, to gain knowledge required to further some other goal. Some do science to gain social standing & intellectual superiority. Some do science to help others understand how the world works. In all cases, the reasons people are doing science is because of a want, some kind of desire to achieve a goal. Nature and physics will continue to exist whether we explore it or not, we do so because we care, and caring is a feeling.
- icantdrive55 10y ago
  And there's people/corporations/institutions who do science to further their agenda. This is a huge part of science. I wish it wasn't. I welcome any program that looks for errors. Now let's expose the bad studies, or bad science? I welcome any program that makes this job easier. I wish I wasen't so cynical towards science. I have just been duped too many times.
- Mithaldu 10y ago
  > unsolicited error reports for already published papers, and I can understand being initially irritated Please don't see this as me attacking you, i am genuinely baffled by what you said and would like to understand your thinking. How exactly could you possibly have any sort of understanding for that? How would you feel about a software author being irritated by unsolicited error reports? If that irritation is less acceptable than the one in your quote, where would lie the difference?
  - dahart 10y ago
    > How exactly could you possibly have any sort of understanding for that? I'm also confused by your question, so I can relate to you. :) Despite your warning, the way you phrased that question does sound like an attack, it implies that you know my experience does not allow me to speak on this subject, which you do not know. Why is it hard to think that I can empathize? What do you think you know about me that makes it implausible for me to understand this situation? I can and do relate to it because I'm a published paper author. I imagine that it would be irritating to me as easily as I imagine it would be irritating to others. For me personally, I don't think it would make me angry, but it would give me anxiety to cast doubt on a paper I'd already published, even if the report about inaccuracies are true. It would mean that I didn't do as good a job as I thought, which of course I want to know, but a published paper is part of a record that cannot change. I'm sure many paper authors, for better or worse, have the same reaction, that something that casts doubt on their publications after the fact would cause some degree of mental anguish. That doesn't mean we shouldn't search for the truth, it means only what I said, that I can understand the reaction. Your analogy to software errors fails. (And I also have first-hand experience with this as the owner of a software business.) Software is an on-going work that can and should be improved to remove errors at all times, and errors that get fixed do not affect my personal reputation or career. Reports of software errors can also be irritating in their own way - I don't want to know my software is buggy - but they are always welcome. Published papers cannot be improved, they are fixed in the permanent record. There are some ways to recover from severe errors, but there are no ways to recover from minor errors, and public (academic) perception of the quality and level of the errors in a paper can change the way an author is viewed.
hx87 10y ago
Even if feelings have everything to do with science, isn't getting errors in your paper pointed out a good thing that you should be happy about?
douche 10y ago
I guess I can accuse my compiler of bullying me.
- virgulino 10y ago
  I've been saying that for years!!!
- mturmon 10y ago
  Bully it back with: $ cc test.c || sudo rm -f $(which cc) "Listen, cc, we can do this the easy way, or the hard way..."
  - marcosdumay 10y ago
    Just make sure you don't make a mistake on that command. You don't want to get into MAD with the shell.
deleted 10y ago
[deleted]
trjordan 10y ago
They weren't looking for errors in papers and trying to push for corrections or improve the science. They were using these errors as fodder for promoting their software. It might not be bullying, but it's not coming from a place of objectivity or honest debate.
- Consultant32452 10y ago
  Yeah, this is an abusive marketing technique. If you don't want future embarrassment you'd better buy our service.
  - oldmanjay 10y ago
    Or just be more vigilant and less defensive, both of which improve the practice of science
    - Consultant32452 10y ago
      That's just not how polite human interaction works. If you become aware that someone has made a mistake you inform them privately to give them an opportunity to rectify that mistake. In the case of a scientific journal entry you might instead publish a response article. You don't put everyone on blast and practically blackmail them into purchasing your for-profit service. It was never about the science at all, it was about the profits for this product/service.
cafebeen 10y ago
Well, scientists are people too, so they have feelings and are not perfect. I think there are polite ways provide criticism and corrections, hopefully without humiliating people with good intentions. If it's a simple mistake, they can email the author and suggest an erratum. If it's more serious, then it's common to write a response in the journal. I think more public forms of criticism are perceived as bullying because a general audience can't typically judge the magnitude of the error, since they're not scientists working in the field, so it can be unjustly damaging to someone's reputation.
- wyager 10y ago
  If someone objectively screwed up a published study, there's no room for feelings or social consideration. People might be wasting their precious time and money pursuing impossible goals based on the incorrect study. Let's say someone finds a mistake in a math paper. Do they politely ask the author to fix it? What happens if the author ignores them? The scientific community doesn't have time to coddle and pursue every mistaken author. Everyone makes mistakes, and the scientific community knows that, but those mistakes need to be brought into then open, not (maybe) resolved behind closed doors.
  - cortesoft 10y ago
    I think there is some nuance here; not all screw ups are equal, nor are all ways of correcting mistakes equal.
  - cafebeen 10y ago
    It's not coddling as much as professional courtesy. Mistakes are often brought up at conferences and in peer review, but most scientists within a specific research area try to be on good terms with one another and don't see value in publicly shaming colleagues for their mistakes.
    - wyager 10y ago
      There are two options here: Use a bot to find tons of mistakes automatically and risk coming across as rude. Or Let social trivialities dominate scientific discourse and let most of those mistakes go unchecked forever because there's no feasible way to "politely" address hundreds of authors who made mistakes and keep checking back to make sure they actually fixed the mistake. The former is clearly the preferable choice. Some individual scientists will suffer for it, but the scientific community as a whole will benefit greatly.

mratzloff 10y ago

Here's the GitHub page: https://github.com/MicheleNuijten/statcheck https://github.com/MicheleNuijten/statcheck And if you're curious how it works, as I was: Statcheck uses regular expressions to find statistical results in APA format. When a statistical result deviates from APA format, statcheck will not find it. The APA formats that statcheck uses are: t(df) = value, p = value; F(df1,df2) = value, p = value; r(df) = value, p = value; [chi]2 (df, N = value) = value, p = value (N is optional, delta G is also included); Z = value, p = value. All regular expressions take into account that test statistics and p values may be exactly (=) or inexactly (< or >) reported. Different spacing has also been taken into account.

snerbles 10y ago
> When a statistical result deviates from APA format, statcheck will not find it. Incentive for authors to obfuscate their math?
- dahart 10y ago
  Pretty sure APA format is required for APA journals, but either way, fortunately, the desire on the part of authors to get papers published and on the part of reviewers to have standards and clear presentation outweighs anyone's desire to game statcheck. But you do make a great point in that statcheck could and should red-line papers that present no discernible stats, and provide links to the APA style guide!
- schmichael 10y ago
  It'd be easier for them to not obfuscate and use the checker themselves prior to publishing.
  - snerbles 10y ago
    True, assuming no malfeasance on the part of the authors.
    - cortesoft 10y ago

michaelt 10y ago

I can certainly understand people being nervous about academic debate moving to social media. It would be a hassle for climate scientists if every paper got a brigade of climate change deniers criticising it and you had to respond to those criticisms. But this example - someone notifying you there's a mistake in your paper, when there really is a mistake? That seems like a strong argument /for/ academic debate via social media, not /against/ it.

deleted 10y ago

[deleted]

6stringmerc 10y ago

What a clever and, dare I say it, fantastically useful experiment! So much less harm than even "door knob twisting" type explorations - no, this was using published works and pretty much running them through a process to verify or not verify accuracy. Unsolicited? So what! As a practiced writer I make unsolicited judgments on language usage all the time. Are these people that completely write from their own minds and don't use a spell check or grammar check program of any sort before sending their material for editorial review? I'd strongly doubt it, because it's a tool to make communication more accurate. Math and formulas having a similar procedural check sounds quite constructive to me. It's not bullying to point out errors; it's bullying to use the existence of errors to belittle or insult a person. I don't see that happening here. Sure, it's a little sterile or "cold" in this fashion, but I think that's for the best if such a process / tool can gain acceptance. It just spits out results and I think that's all it should do. Neat to read about.

SerLava 10y ago
Certainly not bullying. It's actually shocking to me that these studies could have been published without running a similar check on all of the arithmetic. The technology to automate this has been possible for decades.
- dclowd9901 10y ago
  Right. To my mind, this level of scrutiny is little different than a linter or static analyzer for a programming language.
hueving 10y ago
Putting grammar on the same level as this is wrong. Poor grammar and spelling doesn't frequently change the entire results of a paper. Math errors though can easily be the difference between significance and insignificance, changing the entire set of conclusions that can be drawn from the paper.
- 6stringmerc 10y ago
  Uh, no, poor grammar or words or phrases used improperly can genuinely have cascade / domino effects. Brush up on "Eats, Shoots, and Leaves" if you wish to counter-balance your math-centric logic notions with how similar principles apply in written language. English is, without question, one of the more tricky (re: confusing) mutts of linguistics.

munificent 10y ago

> There’s a big, uncomfortable question of how to criticize past work, and whether online critiques of past work constitute “bullying” or shaming. Science is fundamentally reputation-driven. One of, if not the primary incentive that encourages scientists to do science work is the chance of raising their prestige. Citations are one very quantifiable yardstick for this. If positive social sanctions are a driving force for science, then it's entirely reasonable that negative sanctions should come into play too. If you can well-cited paper and attract fame, then a poor paper should likewise attract shame. Otherwise you have a positive feedback loop where once a scientist has attracted enough prestige, they are untouchable. We need negative feedback to balance that out.

danharaj 10y ago
Shaming becomes another power structure that allows the influential to bully their peers. It's not so simple as you suggest.

c3534l 10y ago

> and whether online critiques of past work constitute “bullying” or shaming. The PubPeer comments are just a tiny part of that debate. But it can hit a nerve. > Susan Fiske, a former president of the Association for Psychological Science, alluded to Statcheck in an interview with Business Insider, calling it a “gotcha algorithm.” > The draft of the article said these online critics were engaging in “methodological terrorism.” If these are attitudes typical of psychology, then I cannot say I consider psychology to be a proper social science. There is a fundamental misunderstanding of how knowledge is created through the scientific process if the verification step is considered to be offensive or taboo. That anyone in the field of psychology would even be comfortable publically espousing a non-scientific worldview like that means that psychologists are not being properly educated in the scientific method and should not be in the business of producing research since they do not have a mature understanding of what "scientific" implies.

dmd 10y ago
I took a class with Susan Fiske in grad school, and the following is a quote from one of her comments on a homework where I had called into question a paper that was clearly not just statistically incorrectly done but was based on urban-legend-level assumptions: "These results are from respected investigators, so it's inappropriate to question them."
- ben_jones 10y ago
  While I trust your story, and I believe it's somewhat taboo on HN to ask for this, I feel that it would be much more valuable if evidence was provided. That is an absolutely egregious act for a professor to commit and I feel like it should be reported to someone if substantiated..
  - dmd 10y ago
    Oh, I made a huge stink about it (and this was just one example of many comments of that type). This was at Penn in 2007.

radarsat1 10y ago

The article sadly doesn't report on the false positive rate of statcheck. I assume the paper does? I mean, it just uses a basic regular expression, I can see it easily performing bad checks. I assume the authors take this into account.

RangerScience 10y ago

Haha! What if this is actually a marketing ploy for their web-app? Stir up some shit so everyone gets talking, and provide a service.

EdHominem 10y ago
And what if? Does that negate the results?
- RangerScience 10y ago
  Oh no not at all! The results are still solid. So, "ploy" isn't the right word, not really. "Stunt"? "Technique"? I was just thinking - They could have found a way to email or otherwise contact the authors, instead of just posting comments, but posting comments (and getting some news on it) drives more traffic due to scandal effects. This rabbit hole can go deeper - what if private disclosure was expected to backfire? By surprising people publicly, they're more forced to admit there's an issue; and then by providing both the issue and a solution, there's an easy way at hand to fix it....
  - EdHominem 10y ago
    If I use software and there's a bug, I don't want to find out years later when the author decided to admit it, I want to find out right away so I can get it patched. Papers are equivalent to code, if I'm developing something based on one I want to know if the foundation is reliable or not. > what if private disclosure was expected to backfire? Well of course it could be expected to go badly. Look at all the "concern" spent over their methods already. But yes, it would have been interesting if they had contacted authors quietly and without telling them their methods, simply to see who updated their paper and who tried to blame the messenger... Then publish that list and let the field really correct itself.

deleted 10y ago

[deleted]

JBiserkov 10y ago

For more context, see http://andrewgelman.com/2016/09/21/what-has-happened-down-here-is-the-winds-have-changed/ http://andrewgelman.com/2016/09/21/what-has-happened-down-he...

vinchuco 10y ago

A good distinction between "peer reviewed" vs "computer verified" >“The literature is growing faster and faster, peer review is overstrained, and we need technology to help us out,” This is a problem in every field, not just Psychology. I want someone to tell me the distribution (or average ratio) of papers read to papers written. Every thesis written is supposed to add some delta to the state of the art. But there is no method for doing a diff between past and previous versions of human knowledge. How to make science less redundant and more efficient? I dream of aggregators for everything.

orionblastar 10y ago

I wanted to make a program like that, but considered the ethics of it.

serge2k 10y ago

> some found the emails annoying from PubPeer [since PubPeer notifies authors of comments] if Statcheck found no potential errors I would. > There’s a big, uncomfortable question of how to criticize past work, and whether online critiques of past work constitute “bullying” or shaming. It's facts about your work. Learn to handle it or quit pretending to be a scientist. > The gist of her article was that she feared too much of the criticism of past work had been ceded to social media, and that the criticism there is taking on an uncivil tone Valid enough point. Criticism and correction can be done in a civil manner, and in an accepted forum.

ChoHag 10y ago

Will I never guess what number 7 is? (DNR)

yakult 10y ago

While it is definitely to the benefit of all that the bot emails authors when it finds mistakes, emailing when it doesn't find anything is a dark pattern. Reminds me of those bots that spam me after scraping my linkedin.

l0b0 10y ago

Why does the article focus even for a paragraph on whether egos would be bruised? If the result is a general improvement to the readers' understanding that is, as far as I'm concerned, case closed. Good on them!

OliverJones 10y ago

Remember when writers could do spell-checking and grammar-checking by "running a program" on their text files? Here we have numbers-checking working the same way. I bet you this sort of feature gets built in to word processors eventually, and puts wavy red lines under the results it flags. We've had this sort of real-time "syntax" checking in software engineering for half a generation. It seems wise for other disciplines to consider adopting it too. It's obviously got to be discretionary, just like spell-check is discretionary in browsers. We will get a new genre of humor, thought "statcheck fail."