18 ms·

1 in 4 Statisticians Say They Were Asked to Commit Scientific Fraud

527 points by brahmwg 8y ago

deleted 8y ago
[deleted]
jboggan 8y ago
I was going to give a lecture across several departments about my PhD research in bioinformatics. The night before the talk I was generating some new figures and saw something weird, which led to me digging through source code and discovering a bug which invalidated the last 18 months of research and all my conclusions. I went to my advisor with this problem and he told me to present it anyways. I refused. I suffered consequences for that, including getting stonewalled by my advisor whenever I tried to publish a paper. I wish I had actually been in a position to refuse.
- anticensor 8y ago
  Why did you not try to publish those problematic results without your adviser's name on? There are venues focused just on negative/non-reproducible/problematic results.
- Xcelerate 8y ago
  This is par for the course with computational research. I discovered a bug in my code the last week before submitting my PhD dissertation. Luckily, all of my code was organized in a pipeline that could automatically regenerate everything on the fly, but I needed a supercomputer. The queue was too long for Titan, so I set up an impromptu cluster on Azure (it was the only cloud provider with Infiniband at the time) and paid $200 out of pocket to regenerate the correct figures. I wouldn’t be surprised if a significant portion of published computational research has bugs that totally invalidate the conclusions. I think we need to push hard to require all taxpayer-funded research to make any code that results in a journal article publicly available.
  - thrmsforbfast 8y ago
    Open source doesn't ensure quality code. Ideally the code would be part of the peer review process, but code review is really expensive, so who knows how that would play out.
    - shkkmo 8y ago
      > Open source doesn't ensure quality code. Yes, but closed source helps ensure that low quality code is hidden from sight. It also means that people who distrust or doubt the conclusions have no chance to identify any bug(s) and disprove the results or conclusions.
      - deleted 8y ago
        [deleted]
    - fifnir 8y ago
      It's simple: We stop publishing in papers, and instead adopt smaller chunks of our work as the core publishing units. Each figure should be an individually published entity which contains the entire computational pipeline. Figures are our observations on which we apply logic/philosophy/whatyouwannacallit. Publishing them alongside their relevant code makes the process transparent, reproducible and individually reviewable, as it should be. We can then "publish" comments, observations, conclusions etc on those Figures as a separate thing. Now the logic of the conclusions can be reviewed separately from the statistics and code of the figure.
      - no_identd 8y ago
        I have two words for you: Ted. Nelson.
        j88439h84 8y ago
        Can you expand on this?
- montecarl 8y ago
  I spent a couple months implementing a promising optimization technique that was published in Nature. I assumed that I was missing something or had a bug in my code because I could not reproduce any of the results. It turns out, one guy in my research group knew the author. He requested the source code for me to the program used to generate the data for publication. That program did not even come close to doing what the paper claimed due to some very serious bugs. When I brought up these issues, nobody really seemed to care much. It wouldn't be worth anyone's time to try to correct the issue or write to the journal.
  - BugsJustFindMe 8y ago
    > it wouldn't be worth anyone's time to...write to the journal. I'm sorry, what? It almost sounds like everyone in your lab is scum. Hopefully YOU at least spent the 60 seconds to write an email?
    - aluren 8y ago
      It's not at all uncommon that implementations provided with papers do not actually do what the paper says it does. You're often lucky when there is an implementation at all. But most of the time it's just that running the exact same implementation under the same conditions requires setting up a very specific environment, installing specific versions of libraries, using some niche software and converting data from one byzantine format to another. Each deviation from the original paper is liable to subtly affect the results in nondeterministic ways. That's why no one really gets surprised or even cares, life is too short to call out all the bad software written in academia.
      - RidingPegasus 8y ago
        > requires setting up a very specific environment, installing specific versions of libraries, using some niche software and converting data from one byzantine format to another Containerisation is fairly mature and simple to use. Many in other fields struggle with these exact same issues and are able to create reproducible environments just fine. I find it amazing that those publishing don't include their implementation, all that work locked away on a rusty hdd.
- afarrell 8y ago
  Was it an option to present the paper as normal, but then have a sudden surprise twist at the end which left the audience both surprised and feeling the pathos of imagining how they would feel about discovering the last 18 months of their research was invalid? Or am I mistaken in reading "present" as "stand up in front of a crowd of people"?
  - naasking 8y ago
    Just thought the same. There was a way to "follow orders" and satisfy your principles in a way that serves science. And if anything happened to him after that, it would be clear that it was a political firing.
- astockwell 8y ago
  Good for you, +1 to the greater good for your sacrifice.
- throwaway1982x 8y ago
  Also in bioinformatics: I worked up some statistics that turned out to cleanly and robustly contradict my supervisor's previous research. I was told to reverse my conclusions - refused - and, hey, I no longer work in bioinformatics.
- alexnewman 8y ago
  It took me less than a minute to find your adviser https://www.linkedin.com/in/jake-boggan-6528944a/ https://www.linkedin.com/in/jake-boggan-6528944a/ -> http://www.biomath.gatech.edu/people/ http://www.biomath.gatech.edu/people/ -> Jake Boggan (Advisor: Bunimovich)
  - danieltillett 8y ago
    Probably not the best idea for the OP to post under their own name in a public forum. Jake ask the mods if you can remove your post.
- drb91 8y ago
  Do grad students have any leverage over their advisors? Surely there must be some incentive to root out professors that degrade the reputation of the school. Public shame? Accusations of fraud? There are more options here than students admit, and some other school would have loved to have you.
  - selimthegrim 8y ago
    Transferring grad schools is never easy, and even less so if the adviser decides to start a whispering campaign
  - contact_fusion 8y ago
    No, graduate students in general have no leverage over their advisors. A single word from your advisor and you're out of the field. There is little incentive to root out professors for any reason. The process of becoming a professor (grad school -> postdoc (N times) -> tenure track faculty -> tenured professor) is generally believed (by tenured professors, of course) to root out anyone unworthy of the position. You can believe what you want about the efficiency of such a process. Public shame requires public understanding of scientific (mal)practice, so, good luck communicating that. Most of the time, the bad actors in question have already gotten papers past referees; what makes you think the public is capable of more thorough review? Fraud is considered a serious allegation and as a result accusing someone of it requires going through a thorough process involving a host of university administrators, whose incentives are aligned with the profit motives of the university system. Transferring graduate schools is essentially impossible, and even in the exceptionally rare circumstances that it happens, it always involves burnt bridges and often has to do with bigger fish (i.e. your advisor being offered a position elsewhere, and you're lucky enough they take you with them.) Without external funding to support you, you are usually replaceable. All graduate departments receive applications far in excess of the number of students they can support. They certainly will not consider taking on another from a school at which you've proven to be a problem. Academia has already established a quite successful leaky pipeline; the beginning (graduate school) is no different. In academia, hierarchy is the rule, flat organizations the exception. You must purchase your influence, usually at significant cost (and luck is a significant component). As an undergraduate, the system is designed to cater to your interests; as a graduate student, you cater to the university's interests. Scientific integrity is a noble notion, and in some corners of academia, it survives, but it does so in spite of bad actors who thrive in a system designed to produce ten times the number of qualified applicants for each job, all of whom are judged according to easily gamed metrics. It would be nice if things weren't this way. But the problem is, typically... if you decide to get a PhD in science, it is possible that you're already too obsessed with the subject to ever, truly, give it up, especially if it's "just" over working conditions. I can't speak for everyone, but most people leave because they were forced to.
- tomcam 8y ago
  You rock. Thanks for your integrity. Sorry you had to go through that. In the bigger picture, it drives me crazy that these results aren't welcomed. We learn from failures too. Funding sources should recognize this.
  - edoo 8y ago
    You could almost say we mostly learn from failures. If you had blind success with a complex method the only way to test the limits would be tweaking it to failure.
    - tomcam 8y ago
      That's an incredibly eloquent way to put it. Even though it's how I live most of my life I was never smart enough to express it like you did. Thank you.
- Fomite 8y ago
  Based on my experience on Academia Stack Exchange, in a parallel universe there is a student complaining that their advisor made them rework everything the night before a presentation. Candidly, I probably would have told you to present it as is, and add a caveat to the last slide that this was work in progress (most internal presentations are assumed to be so) and you're still chasing down some problems in your code. The reason for this is two-fold: 1) It's the night before. Many students I have known and mentored aren't at the point in their career that they can "wing" a major presentation. It would be setting them up to fail in a way I couldn't shield them from (I can deal with changing results, but a bad presentation is largely on the student). 2) The quality of your checking is likely to be poor the night before. There's a number of times I've found an error as something was being prepped for presentation/poster printing/etc. and been convinced it changed everything, only to discover after 48 hours of thought and more checking that the difference in results was pretty negligible - especially in the sense of the qualitative take aways from a presentation. This, of course, may not have been your exchange with your PI. But I thought it was worthwhile that there are reasons not to have you change everything the night before that aren't the result of villainous fraud.
  - hnmonkey 8y ago
    This seems like a great neutral path forward that shouldn't upset anyone and doesn't paint the research as 100% factual and in-stone, yet. It leaves the door open to continue the work while rectifying the issues. However, it doesn't excuse the way the advisor asked them to just go with it anyway unless there was some nuance left out in that part of the conversation (akin to you what you've stated) we're not hearing.
- raverbashing 8y ago
  Oh, scientific/research code is hard First of all, a lot of people writing it are inexperienced in SW best practices. Then, turning formulas into code is hard, not to mention some article "handwavings" like showing pseudocode that's almost ok except for "do hard thing in this line" and that line expands into a lot of code. I've also had some weird bugs in code like that (but nothing that would invalidate 18mo of results - btw did you know RAND_MAX is as low as 64k in different compilers, like some older versions of VS?) (And computation times, though the cloud helps a lot with this)
- blensor 8y ago
  This might sound harsh, but that is something you have to deal with in life (academia or non-academia). What you were tasked to do is present your work so other people made room im their calendars to listen to your talk. Finding an error is no reason/excuse not to present your work. BUT you have to make it clear upfront that there is a problem which might invalidate the results. And maybe already give an estimate how far reaching this error is. If you have been working 18 months on a topic you will have substantial knowledge you can communicate to your peers and often some work inspires other work even if they are not using your findings.
- Waterluvian 8y ago
  This has me really curious about the state of software engineering principles in academia if you can find bugs that invalidate 18 months of work. I'm not saying it should have been preventable. It just looks like there may be opportunity to improve practices. Admittedly I'm completely naive to the domain. Are there no forms of validation checkpoints you can reach where your foundations are rock solid and well backed with tests and such? I had a Prof in grad school who lost years of cryospheric research data because an external hard drive was stolen. This was in 2010. It was a head scratcher, especially being faculty-adjacent to some of the best CS and engineering faculties in Canada.
  - hiker512 8y ago
    I've seldomly seen a PhD student in CS here in Germany with a proper software engineering background. I know several that use Dropbox for version control on their code and don't know how to use git. Some at least know and use svn. That stuff just isn't thought at universities and it's assumed, like oh so many things, that you take it up along the way.
- hnmonkey 8y ago
  There really needs to be repercussions for this type of thing. Naming and shaming is a good first step towards that utopia. If those responsible for these types of unethical actions never get punished they'll just keep doing it to the detriment of society.
BenoitEssiambre 8y ago
This didn't even cover the most frequent type of fraud which is to redo the experiment, or reanalyze the data until a sample of noise looks like a signal and then pretend you found something (as illustrated here: https://www.xkcd.com/882/ https://www.xkcd.com/882/). I got out of academia and stopped trusting most university research because I observed too much of this culture of fraud.
truantbuick 8y ago
Beyond overt fraud, I feel like there's so much incentive for imprudent analysis. Several times when I've worked on a project at work that involves data analysis, I get really impatient responses when I don't come to a firm conclusion. We're not even talking about highly charged subjects. I'm not aware if my stakeholders are biased toward one conclusion or another -- I think they're just very upset that my analysis contains uncertainty. They expect me to be able to massage any kind of data to derive clear and obvious facts. Thankfully, I can refuse this without much consequence, but it's really opened my eyes to the potential pressures to corrupt the integrity of fact-finding in even mundane circumstances.
- haskellandchill 8y ago
  I saw some truly horrific things done at the last place I worked, and the kind of environment fostered was such that I wouldn't even hold myself to any standards if push came to shove. Now I work at a different company on a team of one and could do whatever handwaving I want but due to our culture I'm allowed to maintain a high standard of analysis, which is my default as a scientific and mathematically minded researcher.
- lucb1e 8y ago
  > They expect me to be able to massage any kind of data to derive clear and obvious facts. Not to be rude, but if that's in your job description, then I can understand where they are coming from. As someone in IT, it's always my fault that the computer is not working even if the ten year old device simply broke. That doesn't mean they are trying to get me to do an imprudent analysis. Sure, there is a difference between operations and trying to find correct answers to questions (i.e. science), but they do have money to make, and if they're not (even slightly) asking you to bias your work, I can understand if they put a bit of pressure on getting the results clear. Edit: As u/jeremyjh points out, I completely misread this post. If it's about uncertainty due to not enough data, then please completely disregard what I said. (I can't delete the post anymore because there exists a reply to it.)
  - jeremyjh 8y ago
    That isn't how statistics works. You can't remove uncertainty from a given set of data.
aluren 8y ago
To be honest I'm surprised it's not higher. Relevant discussion: https://news.ycombinator.com/item?id=17789308 https://news.ycombinator.com/item?id=17789308 The bottom line is that there's not much incentive for doing the boring statistical validations (I can tell you that no one likes doing statistics apart from statisticians, and not even all of them) and verifying that everything is reproductible, and a huge incentive for, let's say, 'arrange' a thing or two so that the paper looks better. So many people just kinda do it, and it slips through the cracks because: -In many fields, the reviewers are not statisticians themselves -No one really bothers to download the data, setup the environment and libraries, and reproduce the exact steps taken to obtain the very same figures in a paper. Which is understandable given how doing all of that can be such a chore. Anyway, most of the time the exact steps aren't even described. No, jupyter notebooks aren't a solution either (it would take too long to explain why and the post is already long). -In many cases, the results turn out to be true anyway so people don't notice they were initially put forward with fraudulent validations -When they turn out to be false, people just shrug and move on with what's actually true. Bogus results often fail to stand the test of time and get forgotten quickly despite initial hype. No one bothers to say: "Hey, that paper from 8 years ago is bullshit and their authors are hacks!" because nobody cares. -There's a huge psychological barrier to actually call out one of your peers and affirm that they're an impostor and their work is bogus. Especially when said impostor happens to be a big name in the field, that part of you still doesn't believe they would commit fraud, not to mention the social repercussions and backlash of doing such an accusation. We scientists aren't a very confrontational bunch. So most of the time it kinda works and we're all bumbling along hoping to find some modicum of truth at the end of the road with minimal harm done. But of course sometimes you get these guys who kick off their entire career on a high profile, much hyped fabricated result (Woo-Suk) or even a series of bogus papers (Sato), and that may lead to long-term harm. The good news is that hyped papers or very prominent figures quickly attract scrutiny from their peers, and sooner or later reality catches up with them as labs around the world fail to replicate their 'breakthroughs'. All in all, I'd say we're doing fine. We're just not the ethereal source of truth that some people hold us to be, the very same people who, after claming that 'God is dead', are very quick to replace Him with His sillicon-based equivalent around which we would act like priests, except with lab coats in lieu of clerical garments.
phaemon 8y ago
...in the USA, in medical research...
- adtac 8y ago
  That title is too long for HN. Sometimes I wish there was a limit longer than 80 characters so that no one has to resort to slightly more sensational titles.
  - cwyers 8y ago
    I don't know, it's actually more upsetting if it's just medical research? Otherwise I could've told myself it happens less when it really matters.
orthros 8y ago
And the other 3 lied
- deleted 8y ago
  [deleted]
coldtea 8y ago
The other 3 in 4 just lie about it.
- munk-a 8y ago
  Alternatively just one statistician lied, the one conducting this study.
  - emanuelsaringan 8y ago
    ...or it could be all of them
DoreenMichele 8y ago
Of the less serious offenses, 55% of biostatisticians said that they received requests to underreport non-significant results. Sigh.
weberc2 8y ago
I wonder how prevalent this sort of thing is in other fields. I also wonder how prevalent fraud is generally, and the extent to which it is driving our decreasing faith in the academy. Also, how much (if any) of this fraud is motivated by political (instead of strictly personal) outcomes.
TwoBit 8y ago
I wonder what the stats are in China.
- brahmwg 8y ago
  80% of China’s clinical trial data are fraudulent, investigation finds https://www.bmj.com/content/355/bmj.i5396 https://www.bmj.com/content/355/bmj.i5396
  - deleted 8y ago
    [deleted]
tomlock 8y ago
What's odd to me is that some people seem to, even in the face of this, think it's fair to criticize liberal arts journals for sometimes publishing bad papers. People are just people, in every field.
- deleted 8y ago
  [deleted]
ms013 8y ago
Reminds me of a project (not in medicine) I had a couple years ago where my group was in the data science role for another group. We helped out with experimental design, data acquisition, and analysis. After all of our work, when we crunched the numbers, we found that the hypothesis being tested didn’t hold. The people funding the work were not at all happy, and went so far as to claim that “the plots must be wrong”. We showed them everything we had done, and nobody could find any issues with the analysis. But, they were irritated because we didn’t want to play the dishonest game of farting around with plots and numbers to make it look like the experiment worked. So, they angrily broke up with us. On the plus side, when I looked a year or two after, it looks like the grumpy people haven’t made any progress and are either out of business or something.
- overkalix 8y ago
  At least you had the data... I've been a number of times in a situation where I was supposed to conduct extensive, thorough analyses of (expensively acquired) data that was evidently useless. I still had to go through the motions, knowing full well that if somehow the analysis yielded the "desired" results, any caveats would be ignored.
raincom 8y ago
That's why one should big consulting companies like McKinsey, Bain, BCG, etc, who don't need to use hardcore statistics to prove whatever the board/CEO wanted in order to execute some policy.
- casefields 8y ago
  You mean like good ole Arthur Andersen? Even the big boys will play dirty for the right price.
- amrx431 8y ago
  They have been playing dirty for a long time.
jlg23 8y ago
"Prove our new train signaling system is as secure as the old one. If you can, you'll get your degree, if you can't, you find someone else to sponsor your research/degree and we find someone else to do the proof." -- Deutsche Bahn (government held national train company in Germany) in around 2001 to a friend of mine (they and him found "someone else")
- WrtCdEvrydy 8y ago
  "Noone has died from the new train signaling system that has not been put into effect yet. Therefore, we can conclude that the new train signaling system is safer than the old existing system."
- mr_toad 8y ago
  In government this is so common as to have a name. https://en.m.wikipedia.org/wiki/Policy-based_evidence_making https://en.m.wikipedia.org/wiki/Policy-based_evidence_making
danieltillett 8y ago
The pressure to get positive results is just too high to be compatible with good science. When you have people's whole life and career on the line all the time (publish or perish) then people are going to do what they have to stay employed. Like Gresham's law bad science drives out good, because it is much faster to do bad science than good. Those that try to maintain quality can't pump out as many papers as the bad and so lose on the grant treadmill. Any solution that does not address the incentives is doomed to fail. Not fixing this problem will kill science.
- 2spicy_thrwaway 8y ago
  And that pressure to get positive results is extra high right at the beginning of scientists careers, when they should be learning best practices! They do a big project for their doctorate and if it turns out poorly, they can be screwed. There's their chance to move on with their careers flying out the window. Maybe the already had their next gig lined up, but it's time to put that on hold :( Or, maybe if we just make an extra assumption, or twist and turn the data just this way, using this model and these covariates... Oh! Here's something significant! Here's your PhD welcome to science.
  - danieltillett 8y ago
    The pressure doesn't stop at any stage in your scientific career. It is bad at the PhD stage (get results or no PhD), at the postdoc stage (get results or no tenure track position), at the tenure track stage (get results or no tenure), tenured (get results or no grants). About the only point you can actually slow down enough to care about quality is at the emeritus stage, but even then if you don't produce your lab will be moved into a broom cupboard.
    - Fomite 8y ago
      One of the things I make my students do is to try and design projects so that a myriad of answers are all potentially interesting, such that the likelihood of getting a result is high, but the particulars of the research are responsible for what that result is.
DrNuke 8y ago
Sensitivity / ablation studies plus datasets opensourcing should be made mandatory for every computational proposition and research asap, so that results can be checked by peers and by the community at large. With publish-or-perish pressure, democratisation of tools, rogue parties and hyperspecialisation, is it getting to the point almost nothing would sustain any stringent scrutiny? Science cannot afford opacity any more, more than that in these blurred times.
flagfeedingusa 8y ago
But remember, if you are skeptical about global warming and the conclusions from these datasets, you are a "denier" of science. Hmmm I wonder if given the choice of "come to the conclusion global warming is happening and we will fund you" or "come to the conclusion global warming is not happening and find funding elsewhere" I wonder what choice the average person would make. How hard would it be to massage the data and "proove" what the organization funding you wants you to believe? How about doing research into dna and the correlation between that and iq. No way would that information be suppressed right? Science is rife with fraud. If you don't tow the party line, watch as you are black listed and your career is ruined. Academia, the university and modern scientific journals needs to be completely dismantled.
- tomrod 8y ago
  These are broadly addressed critiques with potentially large rewards for discovering/verifying notions contrary to the prevailing understanding. In other words, the incentive to just go-along if one has evidence to the contrary is low relative to the opportunity cost. In even less words: money can be made if strong counter evidence to these topics can be verified. Conclusion: its unlikely that these examples are really areas of hidden wisdom that runs counter to prevailing understanding.
- the_duke 8y ago
  Not surprising that you created a throwaway for this comment...
- WrtCdEvrydy 8y ago
  On the one hand, interesting, but I don't think there's a worldwide conspiracy to prove global warming. On the other hand, 1/4 is a large number, we'd have to break it down by discipline.
wwhuang 8y ago
Very interesting study. I'd love to see a follow up on 2 things: 1) Did the statisticians refuse or comply with the request? 2) When they refused, how did the requester react? Were the requesters actually malicious, or just bad at statistics? If everything was fine once the statisticians explained that removing "just an outlier" wasn't a valid option, then this report isn't quite as concerning, and is maybe just an indication that more researchers need to hire statisticians to help them out.
postit 8y ago
That happens, a lot. Not only statisticians. My friend was half way over her 5 years vesting with a startup as when the CFO asked her to help then improve their numbers due a foreseen investment round. The idea was basically bump revenue basing it solely on the GMV masking returns and not calculating discounts and shipping. They also wanted to hide running costs by forcing vendors to agree on a 90+ days billing cycle, they also pushed the CTO on turning off parts of the system during weekends and holidays and forbid PTO until the deal was closed. She refused doing the number masking and was asked to leave.
- HillaryBriss 8y ago
  The story I heard related to an actuarial consultant who, after analyzing data and suggesting a defensible year-over-year rate hike to their insurance company client, was asked to change their analysis to support the rate hike the insurance company wanted. And, of course, if the consultant didn't play ball, the insurance company could always consult with a different actuarial firm.
  - sk5t 8y ago
    That's the name of the game for certain types of consulting--support the already-made decision. Mostly, but not exclusively, seen in management consulting to support staff reduction, outsourcing, and the like.
IronWolve 8y ago
I was working on the Cingular and ATT merger, my upper management asked me to fudge the numbers on roaming. I would gave the numbers in blue and orange, and blue (att) had better data handoffs on 3g. It made orange look so bad, (shitty network), they made me combine them, to which tanked ATT's network stats to the c-level. One of many times, stats made someone look bad, and they made me change them. Most the time, support metrics are altered to make it look like support contracts are hitting everything contractual. This is daily business everywhere in tech. Mostly fudging to downright lying, just depends on the importance of the data if some mone is tied to it. I don't approve of this shit, but lucky, I never been asked to commit fraud, like reporting sales that don't exist...
- WrtCdEvrydy 8y ago
  We've had this issue before with people scraping our websites, as our software gets better at detecting scraping, our number of page impressions went down. Luckily our board was pretty understanding the reduction in infrastructure strain took a nice percentage out of operating costs.
LeonB 8y ago
It’s “nearly 1 in 4” in the text, but I guess they altered the statistics slightly to make their case look better.
tomahunt 8y ago
"Can you make the error bars a bit smaller please?".... "um no, I'm affraid they are as I've calculated "
dev_dull 8y ago
The troubling part of this article (and especially the comments) is how easy it is to be labeled “anti science” today when questioning certain studies. I’d hate to think our culture is causing us to lose the required academic rigor to make data-based decisions.
- dwaltrip 8y ago
  It's a tricky thing. There is a lot of high quality science. There is some lower quality science -- outright fraud even, unfortunately. And then there is science reporting, which is all over the map, and can easily be misinterpreted. Humanity has built up an enormous amount of legitimate scientific knowledge and understanding. There has been much difficulty, confusion, and dead-ends along the way -- and it seems this continues to be the case. But it really is an incredible thing, how much we know at this point. It took me a long time and a lot of self-learning to develop a rich appreciation for this. Of course, there is still much for us to figure out. And we should keep doing so, difficulties be damned (and hopefully mitigated over time).
- amanaplanacanal 8y ago
  Yes. Of course the other side of the coin is questioning the science because it doesn't match someone's political philosophy. Sailing between the Scylla and Charybdis is hard.
sjg007 8y ago
Is this result statistically significant?
agentofoblivion 8y ago
How do I know this study isn’t part of the 1/4?
rwilson4 8y ago
Most statisticians work for people who are trying to sell something. Statistics is not always the best way of selling something.
2spicy_thrwaway 8y ago
Even within the statistics community, there's a spectrum of quick-and-dirty vs fully rigorous. People with the ability and inclination to be fully rigorous often get treated as pedants and perfectionists (in the bad sense). I often get that with business partners. "The data says <likely X, but with caveats/nuance/uncertainty/under certain assumptions we can't justify>" to which they respond "Can we just say X?" Or "can we get numbers on Y to support a presentation on Z?" when Y seems to support Z, but actually you can't draw that connection, so it's misleading. Stuff like this happens because people treat extra rigor as pedantry and are comfortable making supporting assumptions that aren't supported by data. The people making fraudulent requests aren't aware that they're fraudulent (usually). In my experience, they just think they're being practical.
- tomrod 8y ago
  I agree, and see this regularly. "Picking battles" is one way to describe my counterpoint: caveat exactly as much as needed so that a proper decision can be made with the risks involved. If you want to query your whole team for a joint lunch location but coworker X is out, it is still appropriate to say (assuming more than you and X on team) that you asked the team and you decided on lunch place Y. It's not rigorous (X is left out) but it's still accurate. This is very different from, say, regulatory or securities reporting where ambiguity is not appropriate.
octocode 8y ago
You mean 0 out of 4. wink
deleted 8y ago
[deleted]
jiveturkey 8y ago
'25%' would have been a more appealing headline ... :D
hotprogrammer 8y ago
Can confirm the corruption that exists. I was in academia for some time and I had an advisor who pushed for the publication of a paper I was coauthoring, even when we found out that the methodology didn't work at all. They pressed on, deciding to misrepresent the results by leaving out certain findings - and I chose to have my name removed from the paper. I knew it would cause drama to reveal the real reason I wanted to be taken off the paper, so I made up something else. It was strange because they couldn't possibly understand why I would not want to be associated with that. Meanwhile my other advisor would also choose to leave out relevant statistical findings which provided evidence directly against their hypotheses. I didn't stay there much longer :)
anarchop 8y ago
I think that this should prompt scrutiny in fields that are highly dependent upon statistical analysis, such as climate change. However, this very suggestion is intolerable to many and in a way equivalent to blasphemy against the church science. Scrutiny will not be tolerated by academe with regard to ideas held as sacrosanct. I find this culture in academia pretty disgusting.
ACow_Adonis 8y ago
... and 1 in 4 not prepared to answer truthfully, 1 in 4 not having worked in the field long enough, and 1 in 4 not competent enough to know the difference. /haha, only serious.
DoreenMichele 8y ago
I have serious health issues. I've gradually gotten healthier, in part because I am skeptical of a lot of studies. I know how borked so many of them are, not a thing most people want to hear me assert. I get a metric fuck ton of flack over my skepticism. I kind of have a mental box where I squirrel away little tidbits that meet two criteria: 1. They seem to come from rigorous studies and 2. They also fit with my general understanding of how life works. Over time, I mentally group things -- a la A and B seem related -- without assuming that I know how they relate. I seem to have an inordinately high tolerance for ambiguity. Most people seem to need An Answer even if it's wrong. They have two categories -- black and white -- and when presented with purple or pink or blue, they force fit it to one of their existing categories and don't confuse them with the facts. I wasn't trying to prove anything to anyone. I was just trying to deal with my life. But having gotten substantially healthier, I now wonder how to talk about such. It seems a wasted opportunity for the world for me to not share, but the world has treated me pretty horribly and done all in its power to tell me to STFU, I'm just crazy and spouting nonsense. So I sometimes think I should write what I understand to be true and then carefully back it up with citations to try to support it. Then I read articles like this, throw my hands in the air and go "Why bother?" The way I have been treated seems particularly unfair when you learn how much "real scientists" cook the books. Like why? It feels like pure prejudice when I read things like this.
- deleted 8y ago
  [deleted]
Proven 8y ago
And 3 in 4 were told to do it. They weren't even asked because they work for the government.
edoo 8y ago
I can't help but wonder what the rate is that didn't admit to being asked to commit scientific fraud because they did.
usgroup 8y ago
Another statistic just in: 3/4 statisticians lie about not committing scientific fraud
Dowwie 8y ago
What is the "American Council on Science and Health"? See here: https://www.sourcewatch.org/index.php/American_Council_on_Science_and_Health https://www.sourcewatch.org/index.php/American_Council_on_Sc... 1 in 4 Statisticians weren't asked to commit scientific fraud. Within the article: "Researchers often make "inappropriate requests" to statisticians. And by "inappropriate," the authors aren't referring to accidental requests for incorrect statistical analyses; instead, they're referring to requests for unscrupulous data manipulation or even fraud." This isn't even remotely close to what the title of the article claims.
jimnotgym 8y ago
1 in 4? How do we know that number is not made up? In fact there is a 25% chance that the person involved in working that out has been asked to commit fraud at some point.
- ubitaco 8y ago
  >In fact there is a 25% chance that the person involved in working that out has been asked to commit fraud at some point. But there's only a 10% chance of that.
jamesb93 8y ago
Is this fraudulent stats?
nathan_long 8y ago
"1 in 4 Statisticians Say They Were Asked to Commit Scientific Fraud" It would be hilarious if this report were debunked on the grounds that they inflated the number of statisticians surveyed

527 points by brahmwg 8y ago

deleted 8y ago

[deleted]

I was going to give a lecture across several departments about my PhD research in bioinformatics. The night before the talk I was generating some new figures and saw something weird, which led to me digging through source code and discovering a bug which invalidated the last 18 months of research and all my conclusions. I went to my advisor with this problem and he told me to present it anyways. I refused. I suffered consequences for that, including getting stonewalled by my advisor whenever I tried to publish a paper. I wish I had actually been in a position to refuse.

anticensor 8y ago
Why did you not try to publish those problematic results without your adviser's name on? There are venues focused just on negative/non-reproducible/problematic results.
Xcelerate 8y ago
This is par for the course with computational research. I discovered a bug in my code the last week before submitting my PhD dissertation. Luckily, all of my code was organized in a pipeline that could automatically regenerate everything on the fly, but I needed a supercomputer. The queue was too long for Titan, so I set up an impromptu cluster on Azure (it was the only cloud provider with Infiniband at the time) and paid $200 out of pocket to regenerate the correct figures. I wouldn’t be surprised if a significant portion of published computational research has bugs that totally invalidate the conclusions. I think we need to push hard to require all taxpayer-funded research to make any code that results in a journal article publicly available.
- thrmsforbfast 8y ago
  Open source doesn't ensure quality code. Ideally the code would be part of the peer review process, but code review is really expensive, so who knows how that would play out.
  - shkkmo 8y ago
    > Open source doesn't ensure quality code. Yes, but closed source helps ensure that low quality code is hidden from sight. It also means that people who distrust or doubt the conclusions have no chance to identify any bug(s) and disprove the results or conclusions.
    - deleted 8y ago
      [deleted]
  - fifnir 8y ago
    It's simple: We stop publishing in papers, and instead adopt smaller chunks of our work as the core publishing units. Each figure should be an individually published entity which contains the entire computational pipeline. Figures are our observations on which we apply logic/philosophy/whatyouwannacallit. Publishing them alongside their relevant code makes the process transparent, reproducible and individually reviewable, as it should be. We can then "publish" comments, observations, conclusions etc on those Figures as a separate thing. Now the logic of the conclusions can be reviewed separately from the statistics and code of the figure.
    - no_identd 8y ago
      I have two words for you: Ted. Nelson.
      - j88439h84 8y ago
        Can you expand on this?
montecarl 8y ago
I spent a couple months implementing a promising optimization technique that was published in Nature. I assumed that I was missing something or had a bug in my code because I could not reproduce any of the results. It turns out, one guy in my research group knew the author. He requested the source code for me to the program used to generate the data for publication. That program did not even come close to doing what the paper claimed due to some very serious bugs. When I brought up these issues, nobody really seemed to care much. It wouldn't be worth anyone's time to try to correct the issue or write to the journal.
- BugsJustFindMe 8y ago
  > it wouldn't be worth anyone's time to...write to the journal. I'm sorry, what? It almost sounds like everyone in your lab is scum. Hopefully YOU at least spent the 60 seconds to write an email?
  - aluren 8y ago
    It's not at all uncommon that implementations provided with papers do not actually do what the paper says it does. You're often lucky when there is an implementation at all. But most of the time it's just that running the exact same implementation under the same conditions requires setting up a very specific environment, installing specific versions of libraries, using some niche software and converting data from one byzantine format to another. Each deviation from the original paper is liable to subtly affect the results in nondeterministic ways. That's why no one really gets surprised or even cares, life is too short to call out all the bad software written in academia.
    - RidingPegasus 8y ago
      > requires setting up a very specific environment, installing specific versions of libraries, using some niche software and converting data from one byzantine format to another Containerisation is fairly mature and simple to use. Many in other fields struggle with these exact same issues and are able to create reproducible environments just fine. I find it amazing that those publishing don't include their implementation, all that work locked away on a rusty hdd.
afarrell 8y ago
Was it an option to present the paper as normal, but then have a sudden surprise twist at the end which left the audience both surprised and feeling the pathos of imagining how they would feel about discovering the last 18 months of their research was invalid? Or am I mistaken in reading "present" as "stand up in front of a crowd of people"?
- naasking 8y ago
  Just thought the same. There was a way to "follow orders" and satisfy your principles in a way that serves science. And if anything happened to him after that, it would be clear that it was a political firing.
astockwell 8y ago
Good for you, +1 to the greater good for your sacrifice.
throwaway1982x 8y ago
Also in bioinformatics: I worked up some statistics that turned out to cleanly and robustly contradict my supervisor's previous research. I was told to reverse my conclusions - refused - and, hey, I no longer work in bioinformatics.
alexnewman 8y ago
It took me less than a minute to find your adviser https://www.linkedin.com/in/jake-boggan-6528944a/ https://www.linkedin.com/in/jake-boggan-6528944a/ -> http://www.biomath.gatech.edu/people/ http://www.biomath.gatech.edu/people/ -> Jake Boggan (Advisor: Bunimovich)
- danieltillett 8y ago
  Probably not the best idea for the OP to post under their own name in a public forum. Jake ask the mods if you can remove your post.
drb91 8y ago
Do grad students have any leverage over their advisors? Surely there must be some incentive to root out professors that degrade the reputation of the school. Public shame? Accusations of fraud? There are more options here than students admit, and some other school would have loved to have you.
- selimthegrim 8y ago
  Transferring grad schools is never easy, and even less so if the adviser decides to start a whispering campaign
- contact_fusion 8y ago
  No, graduate students in general have no leverage over their advisors. A single word from your advisor and you're out of the field. There is little incentive to root out professors for any reason. The process of becoming a professor (grad school -> postdoc (N times) -> tenure track faculty -> tenured professor) is generally believed (by tenured professors, of course) to root out anyone unworthy of the position. You can believe what you want about the efficiency of such a process. Public shame requires public understanding of scientific (mal)practice, so, good luck communicating that. Most of the time, the bad actors in question have already gotten papers past referees; what makes you think the public is capable of more thorough review? Fraud is considered a serious allegation and as a result accusing someone of it requires going through a thorough process involving a host of university administrators, whose incentives are aligned with the profit motives of the university system. Transferring graduate schools is essentially impossible, and even in the exceptionally rare circumstances that it happens, it always involves burnt bridges and often has to do with bigger fish (i.e. your advisor being offered a position elsewhere, and you're lucky enough they take you with them.) Without external funding to support you, you are usually replaceable. All graduate departments receive applications far in excess of the number of students they can support. They certainly will not consider taking on another from a school at which you've proven to be a problem. Academia has already established a quite successful leaky pipeline; the beginning (graduate school) is no different. In academia, hierarchy is the rule, flat organizations the exception. You must purchase your influence, usually at significant cost (and luck is a significant component). As an undergraduate, the system is designed to cater to your interests; as a graduate student, you cater to the university's interests. Scientific integrity is a noble notion, and in some corners of academia, it survives, but it does so in spite of bad actors who thrive in a system designed to produce ten times the number of qualified applicants for each job, all of whom are judged according to easily gamed metrics. It would be nice if things weren't this way. But the problem is, typically... if you decide to get a PhD in science, it is possible that you're already too obsessed with the subject to ever, truly, give it up, especially if it's "just" over working conditions. I can't speak for everyone, but most people leave because they were forced to.
tomcam 8y ago
You rock. Thanks for your integrity. Sorry you had to go through that. In the bigger picture, it drives me crazy that these results aren't welcomed. We learn from failures too. Funding sources should recognize this.
- edoo 8y ago
  You could almost say we mostly learn from failures. If you had blind success with a complex method the only way to test the limits would be tweaking it to failure.
  - tomcam 8y ago
    That's an incredibly eloquent way to put it. Even though it's how I live most of my life I was never smart enough to express it like you did. Thank you.
Fomite 8y ago
Based on my experience on Academia Stack Exchange, in a parallel universe there is a student complaining that their advisor made them rework everything the night before a presentation. Candidly, I probably would have told you to present it as is, and add a caveat to the last slide that this was work in progress (most internal presentations are assumed to be so) and you're still chasing down some problems in your code. The reason for this is two-fold: 1) It's the night before. Many students I have known and mentored aren't at the point in their career that they can "wing" a major presentation. It would be setting them up to fail in a way I couldn't shield them from (I can deal with changing results, but a bad presentation is largely on the student). 2) The quality of your checking is likely to be poor the night before. There's a number of times I've found an error as something was being prepped for presentation/poster printing/etc. and been convinced it changed everything, only to discover after 48 hours of thought and more checking that the difference in results was pretty negligible - especially in the sense of the qualitative take aways from a presentation. This, of course, may not have been your exchange with your PI. But I thought it was worthwhile that there are reasons not to have you change everything the night before that aren't the result of villainous fraud.
- hnmonkey 8y ago
  This seems like a great neutral path forward that shouldn't upset anyone and doesn't paint the research as 100% factual and in-stone, yet. It leaves the door open to continue the work while rectifying the issues. However, it doesn't excuse the way the advisor asked them to just go with it anyway unless there was some nuance left out in that part of the conversation (akin to you what you've stated) we're not hearing.
raverbashing 8y ago
Oh, scientific/research code is hard First of all, a lot of people writing it are inexperienced in SW best practices. Then, turning formulas into code is hard, not to mention some article "handwavings" like showing pseudocode that's almost ok except for "do hard thing in this line" and that line expands into a lot of code. I've also had some weird bugs in code like that (but nothing that would invalidate 18mo of results - btw did you know RAND_MAX is as low as 64k in different compilers, like some older versions of VS?) (And computation times, though the cloud helps a lot with this)
blensor 8y ago
This might sound harsh, but that is something you have to deal with in life (academia or non-academia). What you were tasked to do is present your work so other people made room im their calendars to listen to your talk. Finding an error is no reason/excuse not to present your work. BUT you have to make it clear upfront that there is a problem which might invalidate the results. And maybe already give an estimate how far reaching this error is. If you have been working 18 months on a topic you will have substantial knowledge you can communicate to your peers and often some work inspires other work even if they are not using your findings.
Waterluvian 8y ago
This has me really curious about the state of software engineering principles in academia if you can find bugs that invalidate 18 months of work. I'm not saying it should have been preventable. It just looks like there may be opportunity to improve practices. Admittedly I'm completely naive to the domain. Are there no forms of validation checkpoints you can reach where your foundations are rock solid and well backed with tests and such? I had a Prof in grad school who lost years of cryospheric research data because an external hard drive was stolen. This was in 2010. It was a head scratcher, especially being faculty-adjacent to some of the best CS and engineering faculties in Canada.
- hiker512 8y ago
  I've seldomly seen a PhD student in CS here in Germany with a proper software engineering background. I know several that use Dropbox for version control on their code and don't know how to use git. Some at least know and use svn. That stuff just isn't thought at universities and it's assumed, like oh so many things, that you take it up along the way.
hnmonkey 8y ago
There really needs to be repercussions for this type of thing. Naming and shaming is a good first step towards that utopia. If those responsible for these types of unethical actions never get punished they'll just keep doing it to the detriment of society.

BenoitEssiambre 8y ago

This didn't even cover the most frequent type of fraud which is to redo the experiment, or reanalyze the data until a sample of noise looks like a signal and then pretend you found something (as illustrated here: https://www.xkcd.com/882/ https://www.xkcd.com/882/). I got out of academia and stopped trusting most university research because I observed too much of this culture of fraud.

truantbuick 8y ago

Beyond overt fraud, I feel like there's so much incentive for imprudent analysis. Several times when I've worked on a project at work that involves data analysis, I get really impatient responses when I don't come to a firm conclusion. We're not even talking about highly charged subjects. I'm not aware if my stakeholders are biased toward one conclusion or another -- I think they're just very upset that my analysis contains uncertainty. They expect me to be able to massage any kind of data to derive clear and obvious facts. Thankfully, I can refuse this without much consequence, but it's really opened my eyes to the potential pressures to corrupt the integrity of fact-finding in even mundane circumstances.

haskellandchill 8y ago
I saw some truly horrific things done at the last place I worked, and the kind of environment fostered was such that I wouldn't even hold myself to any standards if push came to shove. Now I work at a different company on a team of one and could do whatever handwaving I want but due to our culture I'm allowed to maintain a high standard of analysis, which is my default as a scientific and mathematically minded researcher.
lucb1e 8y ago
> They expect me to be able to massage any kind of data to derive clear and obvious facts. Not to be rude, but if that's in your job description, then I can understand where they are coming from. As someone in IT, it's always my fault that the computer is not working even if the ten year old device simply broke. That doesn't mean they are trying to get me to do an imprudent analysis. Sure, there is a difference between operations and trying to find correct answers to questions (i.e. science), but they do have money to make, and if they're not (even slightly) asking you to bias your work, I can understand if they put a bit of pressure on getting the results clear. Edit: As u/jeremyjh points out, I completely misread this post. If it's about uncertainty due to not enough data, then please completely disregard what I said. (I can't delete the post anymore because there exists a reply to it.)
- jeremyjh 8y ago
  That isn't how statistics works. You can't remove uncertainty from a given set of data.

aluren 8y ago

To be honest I'm surprised it's not higher. Relevant discussion: https://news.ycombinator.com/item?id=17789308 https://news.ycombinator.com/item?id=17789308 The bottom line is that there's not much incentive for doing the boring statistical validations (I can tell you that no one likes doing statistics apart from statisticians, and not even all of them) and verifying that everything is reproductible, and a huge incentive for, let's say, 'arrange' a thing or two so that the paper looks better. So many people just kinda do it, and it slips through the cracks because: -In many fields, the reviewers are not statisticians themselves -No one really bothers to download the data, setup the environment and libraries, and reproduce the exact steps taken to obtain the very same figures in a paper. Which is understandable given how doing all of that can be such a chore. Anyway, most of the time the exact steps aren't even described. No, jupyter notebooks aren't a solution either (it would take too long to explain why and the post is already long). -In many cases, the results turn out to be true anyway so people don't notice they were initially put forward with fraudulent validations -When they turn out to be false, people just shrug and move on with what's actually true. Bogus results often fail to stand the test of time and get forgotten quickly despite initial hype. No one bothers to say: "Hey, that paper from 8 years ago is bullshit and their authors are hacks!" because nobody cares. -There's a huge psychological barrier to actually call out one of your peers and affirm that they're an impostor and their work is bogus. Especially when said impostor happens to be a big name in the field, that part of you still doesn't believe they would commit fraud, not to mention the social repercussions and backlash of doing such an accusation. We scientists aren't a very confrontational bunch. So most of the time it kinda works and we're all bumbling along hoping to find some modicum of truth at the end of the road with minimal harm done. But of course sometimes you get these guys who kick off their entire career on a high profile, much hyped fabricated result (Woo-Suk) or even a series of bogus papers (Sato), and that may lead to long-term harm. The good news is that hyped papers or very prominent figures quickly attract scrutiny from their peers, and sooner or later reality catches up with them as labs around the world fail to replicate their 'breakthroughs'. All in all, I'd say we're doing fine. We're just not the ethereal source of truth that some people hold us to be, the very same people who, after claming that 'God is dead', are very quick to replace Him with His sillicon-based equivalent around which we would act like priests, except with lab coats in lieu of clerical garments.

phaemon 8y ago

...in the USA, in medical research...

adtac 8y ago
That title is too long for HN. Sometimes I wish there was a limit longer than 80 characters so that no one has to resort to slightly more sensational titles.
- cwyers 8y ago
  I don't know, it's actually more upsetting if it's just medical research? Otherwise I could've told myself it happens less when it really matters.

orthros 8y ago

And the other 3 lied

deleted 8y ago
[deleted]

coldtea 8y ago

The other 3 in 4 just lie about it.

munk-a 8y ago
Alternatively just one statistician lied, the one conducting this study.
- emanuelsaringan 8y ago
  ...or it could be all of them

DoreenMichele 8y ago

Of the less serious offenses, 55% of biostatisticians said that they received requests to underreport non-significant results. Sigh.

weberc2 8y ago

I wonder how prevalent this sort of thing is in other fields. I also wonder how prevalent fraud is generally, and the extent to which it is driving our decreasing faith in the academy. Also, how much (if any) of this fraud is motivated by political (instead of strictly personal) outcomes.

TwoBit 8y ago

I wonder what the stats are in China.

brahmwg 8y ago
80% of China’s clinical trial data are fraudulent, investigation finds https://www.bmj.com/content/355/bmj.i5396 https://www.bmj.com/content/355/bmj.i5396
- deleted 8y ago
  [deleted]

tomlock 8y ago

What's odd to me is that some people seem to, even in the face of this, think it's fair to criticize liberal arts journals for sometimes publishing bad papers. People are just people, in every field.

deleted 8y ago
[deleted]

ms013 8y ago

Reminds me of a project (not in medicine) I had a couple years ago where my group was in the data science role for another group. We helped out with experimental design, data acquisition, and analysis. After all of our work, when we crunched the numbers, we found that the hypothesis being tested didn’t hold. The people funding the work were not at all happy, and went so far as to claim that “the plots must be wrong”. We showed them everything we had done, and nobody could find any issues with the analysis. But, they were irritated because we didn’t want to play the dishonest game of farting around with plots and numbers to make it look like the experiment worked. So, they angrily broke up with us. On the plus side, when I looked a year or two after, it looks like the grumpy people haven’t made any progress and are either out of business or something.

overkalix 8y ago
At least you had the data... I've been a number of times in a situation where I was supposed to conduct extensive, thorough analyses of (expensively acquired) data that was evidently useless. I still had to go through the motions, knowing full well that if somehow the analysis yielded the "desired" results, any caveats would be ignored.

raincom 8y ago

That's why one should big consulting companies like McKinsey, Bain, BCG, etc, who don't need to use hardcore statistics to prove whatever the board/CEO wanted in order to execute some policy.

casefields 8y ago
You mean like good ole Arthur Andersen? Even the big boys will play dirty for the right price.
amrx431 8y ago
They have been playing dirty for a long time.

jlg23 8y ago

"Prove our new train signaling system is as secure as the old one. If you can, you'll get your degree, if you can't, you find someone else to sponsor your research/degree and we find someone else to do the proof." -- Deutsche Bahn (government held national train company in Germany) in around 2001 to a friend of mine (they and him found "someone else")

WrtCdEvrydy 8y ago
"Noone has died from the new train signaling system that has not been put into effect yet. Therefore, we can conclude that the new train signaling system is safer than the old existing system."
mr_toad 8y ago
In government this is so common as to have a name. https://en.m.wikipedia.org/wiki/Policy-based_evidence_making https://en.m.wikipedia.org/wiki/Policy-based_evidence_making

danieltillett 8y ago

The pressure to get positive results is just too high to be compatible with good science. When you have people's whole life and career on the line all the time (publish or perish) then people are going to do what they have to stay employed. Like Gresham's law bad science drives out good, because it is much faster to do bad science than good. Those that try to maintain quality can't pump out as many papers as the bad and so lose on the grant treadmill. Any solution that does not address the incentives is doomed to fail. Not fixing this problem will kill science.

2spicy_thrwaway 8y ago
And that pressure to get positive results is extra high right at the beginning of scientists careers, when they should be learning best practices! They do a big project for their doctorate and if it turns out poorly, they can be screwed. There's their chance to move on with their careers flying out the window. Maybe the already had their next gig lined up, but it's time to put that on hold :( Or, maybe if we just make an extra assumption, or twist and turn the data just this way, using this model and these covariates... Oh! Here's something significant! Here's your PhD welcome to science.
- danieltillett 8y ago
  The pressure doesn't stop at any stage in your scientific career. It is bad at the PhD stage (get results or no PhD), at the postdoc stage (get results or no tenure track position), at the tenure track stage (get results or no tenure), tenured (get results or no grants). About the only point you can actually slow down enough to care about quality is at the emeritus stage, but even then if you don't produce your lab will be moved into a broom cupboard.
  - Fomite 8y ago
    One of the things I make my students do is to try and design projects so that a myriad of answers are all potentially interesting, such that the likelihood of getting a result is high, but the particulars of the research are responsible for what that result is.

DrNuke 8y ago

Sensitivity / ablation studies plus datasets opensourcing should be made mandatory for every computational proposition and research asap, so that results can be checked by peers and by the community at large. With publish-or-perish pressure, democratisation of tools, rogue parties and hyperspecialisation, is it getting to the point almost nothing would sustain any stringent scrutiny? Science cannot afford opacity any more, more than that in these blurred times.

flagfeedingusa 8y ago

But remember, if you are skeptical about global warming and the conclusions from these datasets, you are a "denier" of science. Hmmm I wonder if given the choice of "come to the conclusion global warming is happening and we will fund you" or "come to the conclusion global warming is not happening and find funding elsewhere" I wonder what choice the average person would make. How hard would it be to massage the data and "proove" what the organization funding you wants you to believe? How about doing research into dna and the correlation between that and iq. No way would that information be suppressed right? Science is rife with fraud. If you don't tow the party line, watch as you are black listed and your career is ruined. Academia, the university and modern scientific journals needs to be completely dismantled.

tomrod 8y ago
These are broadly addressed critiques with potentially large rewards for discovering/verifying notions contrary to the prevailing understanding. In other words, the incentive to just go-along if one has evidence to the contrary is low relative to the opportunity cost. In even less words: money can be made if strong counter evidence to these topics can be verified. Conclusion: its unlikely that these examples are really areas of hidden wisdom that runs counter to prevailing understanding.
the_duke 8y ago
Not surprising that you created a throwaway for this comment...
WrtCdEvrydy 8y ago
On the one hand, interesting, but I don't think there's a worldwide conspiracy to prove global warming. On the other hand, 1/4 is a large number, we'd have to break it down by discipline.

wwhuang 8y ago

Very interesting study. I'd love to see a follow up on 2 things: 1) Did the statisticians refuse or comply with the request? 2) When they refused, how did the requester react? Were the requesters actually malicious, or just bad at statistics? If everything was fine once the statisticians explained that removing "just an outlier" wasn't a valid option, then this report isn't quite as concerning, and is maybe just an indication that more researchers need to hire statisticians to help them out.

postit 8y ago

That happens, a lot. Not only statisticians. My friend was half way over her 5 years vesting with a startup as when the CFO asked her to help then improve their numbers due a foreseen investment round. The idea was basically bump revenue basing it solely on the GMV masking returns and not calculating discounts and shipping. They also wanted to hide running costs by forcing vendors to agree on a 90+ days billing cycle, they also pushed the CTO on turning off parts of the system during weekends and holidays and forbid PTO until the deal was closed. She refused doing the number masking and was asked to leave.

HillaryBriss 8y ago
The story I heard related to an actuarial consultant who, after analyzing data and suggesting a defensible year-over-year rate hike to their insurance company client, was asked to change their analysis to support the rate hike the insurance company wanted. And, of course, if the consultant didn't play ball, the insurance company could always consult with a different actuarial firm.
- sk5t 8y ago
  That's the name of the game for certain types of consulting--support the already-made decision. Mostly, but not exclusively, seen in management consulting to support staff reduction, outsourcing, and the like.

IronWolve 8y ago

I was working on the Cingular and ATT merger, my upper management asked me to fudge the numbers on roaming. I would gave the numbers in blue and orange, and blue (att) had better data handoffs on 3g. It made orange look so bad, (shitty network), they made me combine them, to which tanked ATT's network stats to the c-level. One of many times, stats made someone look bad, and they made me change them. Most the time, support metrics are altered to make it look like support contracts are hitting everything contractual. This is daily business everywhere in tech. Mostly fudging to downright lying, just depends on the importance of the data if some mone is tied to it. I don't approve of this shit, but lucky, I never been asked to commit fraud, like reporting sales that don't exist...

WrtCdEvrydy 8y ago
We've had this issue before with people scraping our websites, as our software gets better at detecting scraping, our number of page impressions went down. Luckily our board was pretty understanding the reduction in infrastructure strain took a nice percentage out of operating costs.

LeonB 8y ago

It’s “nearly 1 in 4” in the text, but I guess they altered the statistics slightly to make their case look better.

tomahunt 8y ago

"Can you make the error bars a bit smaller please?".... "um no, I'm affraid they are as I've calculated "

dev_dull 8y ago

The troubling part of this article (and especially the comments) is how easy it is to be labeled “anti science” today when questioning certain studies. I’d hate to think our culture is causing us to lose the required academic rigor to make data-based decisions.

dwaltrip 8y ago
It's a tricky thing. There is a lot of high quality science. There is some lower quality science -- outright fraud even, unfortunately. And then there is science reporting, which is all over the map, and can easily be misinterpreted. Humanity has built up an enormous amount of legitimate scientific knowledge and understanding. There has been much difficulty, confusion, and dead-ends along the way -- and it seems this continues to be the case. But it really is an incredible thing, how much we know at this point. It took me a long time and a lot of self-learning to develop a rich appreciation for this. Of course, there is still much for us to figure out. And we should keep doing so, difficulties be damned (and hopefully mitigated over time).
amanaplanacanal 8y ago
Yes. Of course the other side of the coin is questioning the science because it doesn't match someone's political philosophy. Sailing between the Scylla and Charybdis is hard.

sjg007 8y ago

Is this result statistically significant?

agentofoblivion 8y ago

How do I know this study isn’t part of the 1/4?

rwilson4 8y ago

Most statisticians work for people who are trying to sell something. Statistics is not always the best way of selling something.

2spicy_thrwaway 8y ago

Even within the statistics community, there's a spectrum of quick-and-dirty vs fully rigorous. People with the ability and inclination to be fully rigorous often get treated as pedants and perfectionists (in the bad sense). I often get that with business partners. "The data says <likely X, but with caveats/nuance/uncertainty/under certain assumptions we can't justify>" to which they respond "Can we just say X?" Or "can we get numbers on Y to support a presentation on Z?" when Y seems to support Z, but actually you can't draw that connection, so it's misleading. Stuff like this happens because people treat extra rigor as pedantry and are comfortable making supporting assumptions that aren't supported by data. The people making fraudulent requests aren't aware that they're fraudulent (usually). In my experience, they just think they're being practical.

tomrod 8y ago
I agree, and see this regularly. "Picking battles" is one way to describe my counterpoint: caveat exactly as much as needed so that a proper decision can be made with the risks involved. If you want to query your whole team for a joint lunch location but coworker X is out, it is still appropriate to say (assuming more than you and X on team) that you asked the team and you decided on lunch place Y. It's not rigorous (X is left out) but it's still accurate. This is very different from, say, regulatory or securities reporting where ambiguity is not appropriate.

octocode 8y ago

You mean 0 out of 4. wink

deleted 8y ago

[deleted]

jiveturkey 8y ago

'25%' would have been a more appealing headline ... :D

hotprogrammer 8y ago

Can confirm the corruption that exists. I was in academia for some time and I had an advisor who pushed for the publication of a paper I was coauthoring, even when we found out that the methodology didn't work at all. They pressed on, deciding to misrepresent the results by leaving out certain findings - and I chose to have my name removed from the paper. I knew it would cause drama to reveal the real reason I wanted to be taken off the paper, so I made up something else. It was strange because they couldn't possibly understand why I would not want to be associated with that. Meanwhile my other advisor would also choose to leave out relevant statistical findings which provided evidence directly against their hypotheses. I didn't stay there much longer :)

anarchop 8y ago

I think that this should prompt scrutiny in fields that are highly dependent upon statistical analysis, such as climate change. However, this very suggestion is intolerable to many and in a way equivalent to blasphemy against the church science. Scrutiny will not be tolerated by academe with regard to ideas held as sacrosanct. I find this culture in academia pretty disgusting.

ACow_Adonis 8y ago

... and 1 in 4 not prepared to answer truthfully, 1 in 4 not having worked in the field long enough, and 1 in 4 not competent enough to know the difference. /haha, only serious.

DoreenMichele 8y ago

I have serious health issues. I've gradually gotten healthier, in part because I am skeptical of a lot of studies. I know how borked so many of them are, not a thing most people want to hear me assert. I get a metric fuck ton of flack over my skepticism. I kind of have a mental box where I squirrel away little tidbits that meet two criteria: 1. They seem to come from rigorous studies and 2. They also fit with my general understanding of how life works. Over time, I mentally group things -- a la A and B seem related -- without assuming that I know how they relate. I seem to have an inordinately high tolerance for ambiguity. Most people seem to need An Answer even if it's wrong. They have two categories -- black and white -- and when presented with purple or pink or blue, they force fit it to one of their existing categories and don't confuse them with the facts. I wasn't trying to prove anything to anyone. I was just trying to deal with my life. But having gotten substantially healthier, I now wonder how to talk about such. It seems a wasted opportunity for the world for me to not share, but the world has treated me pretty horribly and done all in its power to tell me to STFU, I'm just crazy and spouting nonsense. So I sometimes think I should write what I understand to be true and then carefully back it up with citations to try to support it. Then I read articles like this, throw my hands in the air and go "Why bother?" The way I have been treated seems particularly unfair when you learn how much "real scientists" cook the books. Like why? It feels like pure prejudice when I read things like this.

deleted 8y ago
[deleted]

Proven 8y ago

And 3 in 4 were told to do it. They weren't even asked because they work for the government.

edoo 8y ago

I can't help but wonder what the rate is that didn't admit to being asked to commit scientific fraud because they did.

usgroup 8y ago

Another statistic just in: 3/4 statisticians lie about not committing scientific fraud

Dowwie 8y ago

What is the "American Council on Science and Health"? See here: https://www.sourcewatch.org/index.php/American_Council_on_Science_and_Health https://www.sourcewatch.org/index.php/American_Council_on_Sc... 1 in 4 Statisticians weren't asked to commit scientific fraud. Within the article: "Researchers often make "inappropriate requests" to statisticians. And by "inappropriate," the authors aren't referring to accidental requests for incorrect statistical analyses; instead, they're referring to requests for unscrupulous data manipulation or even fraud." This isn't even remotely close to what the title of the article claims.

jimnotgym 8y ago

1 in 4? How do we know that number is not made up? In fact there is a 25% chance that the person involved in working that out has been asked to commit fraud at some point.

ubitaco 8y ago
>In fact there is a 25% chance that the person involved in working that out has been asked to commit fraud at some point. But there's only a 10% chance of that.

jamesb93 8y ago

Is this fraudulent stats?

nathan_long 8y ago

"1 in 4 Statisticians Say They Were Asked to Commit Scientific Fraud" It would be hilarious if this report were debunked on the grounds that they inflated the number of statisticians surveyed