16 ms·

I was going to give a lecture across several departments about my PhD research in bioinformatics. The night before the talk I was generating some new figures an

I was going to give a lecture across several departments about my PhD research in bioinformatics. The night before the talk I was generating some new figures and saw something weird, which led to me digging through source code and discovering a bug which invalidated the last 18 months of research and all my conclusions. I went to my advisor with this problem and he told me to present it anyways. I refused. I suffered consequences for that, including getting stonewalled by my advisor whenever I tried to publish a paper. I wish I had actually been in a position to refuse.

anticensor 8y ago
Why did you not try to publish those problematic results without your adviser's name on? There are venues focused just on negative/non-reproducible/problematic results.
Xcelerate 8y ago
This is par for the course with computational research. I discovered a bug in my code the last week before submitting my PhD dissertation. Luckily, all of my code was organized in a pipeline that could automatically regenerate everything on the fly, but I needed a supercomputer. The queue was too long for Titan, so I set up an impromptu cluster on Azure (it was the only cloud provider with Infiniband at the time) and paid $200 out of pocket to regenerate the correct figures. I wouldn’t be surprised if a significant portion of published computational research has bugs that totally invalidate the conclusions. I think we need to push hard to require all taxpayer-funded research to make any code that results in a journal article publicly available.
- thrmsforbfast 8y ago
  Open source doesn't ensure quality code. Ideally the code would be part of the peer review process, but code review is really expensive, so who knows how that would play out.
  - shkkmo 8y ago
    > Open source doesn't ensure quality code. Yes, but closed source helps ensure that low quality code is hidden from sight. It also means that people who distrust or doubt the conclusions have no chance to identify any bug(s) and disprove the results or conclusions.
    - deleted 8y ago
      [deleted]
  - fifnir 8y ago
    It's simple: We stop publishing in papers, and instead adopt smaller chunks of our work as the core publishing units. Each figure should be an individually published entity which contains the entire computational pipeline. Figures are our observations on which we apply logic/philosophy/whatyouwannacallit. Publishing them alongside their relevant code makes the process transparent, reproducible and individually reviewable, as it should be. We can then "publish" comments, observations, conclusions etc on those Figures as a separate thing. Now the logic of the conclusions can be reviewed separately from the statistics and code of the figure.
    - no_identd 8y ago
      I have two words for you: Ted. Nelson.
      - j88439h84 8y ago
        Can you expand on this?
        lvh 8y ago
        I can’t speak for GP, but Nelson invented hypermedia/hyperlinks and had a vision for the future that included documents including other documents. All of that seems pretty compatible.
    - 8y ago
- andyv 8y ago
  Computational research is incredibly difficult because it's usually hard to see the effect of a bug. A triangle drawn on the wrong place of a screen can be easy to see, but a typo in your integration subroutine? Hard to spot if you don't catch it when it is born. I also had a few do-overs at the end of my thesis, but fortunately had a cluster standing by...
  - BeetleB 8y ago
    >A triangle drawn on the wrong place of a screen can be easy to see, but a typo in your integration subroutine? Hard to spot if you don't catch it when it is born. Well, there's also this notion of testing and regression. As I said in another comment a few days ago: >A few weeks ago I had a conversation with a friend of mine who is wrapping up his PhD. He pointed out that not one of his colleagues is concerned whether anyone can reproduce their work. They use a home grown simulation suite which only they have access to, and is constantly being updated with the worst software practices you can think of. No one in their team believes that the tool will give the same results they did 4 years ago. The troubling part is, no one sees that as being a problem. They got their papers published, and so the SW did its job.
    - fouc 8y ago
      Wow. If it's not reproducible, can it even be called science?
      - BurningFrog 8y ago
        Perhaps it's a reproducible way to build careers?
      - slavik81 8y ago
        It's not really much different from a physical experiment, where the expectation is that you'll have to rebuild the experimental apparatus yourself to reproduce the experiment. An independent implementation of the experiment is neccessary for a full reproduction anyways. If you just run their code again, you'll end up with all their bugs again. (But, don't get me wrong. I like when researchers release their code. It's still very useful.)
- nestorD 8y ago
  A few days ago in the description of a paper found on hackernews (https://news.ycombinator.com/item?id=18346943 https://news.ycombinator.com/item?id=18346943) : > We also noticed significant improvements in performance of RND every time we discovered and fixed a bug [...]. Getting such details right was a significant part of achieving high performance even with algorithms conceptually similar to prior work. They call bugs 'details' which, I find, is a frightening state of mind for someone publishing an algorithm.
  - geezerjay 8y ago
    > They call bugs 'details' which, I find, is a frightening state of mind for someone publishing an algorithm. It really depends on the algorithm. For example, a bug in the random number generator of a stochastic search algorithm that affects, say, the variance of a distribution won't have a relevant impact on the outcome.
    - mlthoughts2018 8y ago
      That could have a huge impact. For example if it affected random draws of a hyperparameter in a Bayesian model, leading to incorrect credible intervals ultimately in the posterior distribution. Or worse, if the RNG bug was affecting the variance of randon samples in some deep component of an MCMC algorithm like NUTS or even simple Metropolis. Depending on the exact nature of the bug, it could even cause the model to violate Detailed Balance, and entirely invalidate sampling based inferences or conclusions.
      - riskneutral 8y ago
        Well that is why you would always run statistical validity tests on the RNG and other intermediate values whenever using a Monte Carlo model in production. Ideally the tests should run as part of a Continuous Integration workflow.
        mlthoughts2018 8y ago
        If you (dubiously) write your own RNG, sure. But you should never do that. And for library RNGs, you should execute the unit tests of the library. Frankly, running it as part of CI is at best overkill and at worst adds complexity that costs you. If you pin versions of your dependency and isolate the artifact into an in-house artifact repository, so that the library code is literally never changing unless you explicitly modify the version, then you should test it up front, then again only occasionally if you actually have evidence of a bug. And as part of any code review of the code change that introduces a version change.
- badpun 8y ago
  This is why I'm a bit skeptical about the global warming predictions. AFAIK they're all based on multi-million lines of code models. Not only it's monumentally hard to have such large code base without bugs, but also, in case of a software implementing scientific models, finding bugs is extra hard (compared to software which works in a visible way and has millions of users, such as Linux kernel, games, car's embedded software etc.). An effort to have such model verified (let's say to a NASA standard) to be bug-free would probably cost billions of dollars. And that's only bugs, let's not forget that all the models are approximations, plus the numerical methods used all have their quirks and limitations etc. All in all, the problem seems too hard to tackle for humanity right now.
  - nafey 8y ago
    The counterpoint is that we have already seen a steady temperature rise. So even though the specifics of the various simulations might not play out as predicted we can expect temperatures to rise.
    - badpun 8y ago
      Knowing that the temperature will rise is not enough to make policy decisions though. You need to predict the increase's magnitude, as well as practical consequences, such as climate change, how much the sea level will rise etc. And for that you need resonable and bug-free models.
      - leeoniya 8y ago
        > how much the sea level will rise etc this will be the least of our problems. we understand very little about nature to have any reasonable prediction model. we don't know the inflection point which will cause massive collapse of major ecosystems we depend on. all we know is that things are changing fast. faster than many non-human organisms are able to adapt.
    - femidav 8y ago
      Steady? We have seen 20-years rise after a colder period. Nothing out of order.
- bad_news_bears 8y ago
  Was there a large, I guess “significant” difference between your results pre- and post-bug?
- deleted 8y ago
  [deleted]
- fghtr 8y ago
  > I think we need to push hard to require all taxpayer-funded research to make any code that results in a journal article publicly available. "Already 18824 SIGNATURES – sign the open letter now!" https://publiccode.eu https://publiccode.eu
montecarl 8y ago
I spent a couple months implementing a promising optimization technique that was published in Nature. I assumed that I was missing something or had a bug in my code because I could not reproduce any of the results. It turns out, one guy in my research group knew the author. He requested the source code for me to the program used to generate the data for publication. That program did not even come close to doing what the paper claimed due to some very serious bugs. When I brought up these issues, nobody really seemed to care much. It wouldn't be worth anyone's time to try to correct the issue or write to the journal.
- BugsJustFindMe 8y ago
  > it wouldn't be worth anyone's time to...write to the journal. I'm sorry, what? It almost sounds like everyone in your lab is scum. Hopefully YOU at least spent the 60 seconds to write an email?
  - aluren 8y ago
    It's not at all uncommon that implementations provided with papers do not actually do what the paper says it does. You're often lucky when there is an implementation at all. But most of the time it's just that running the exact same implementation under the same conditions requires setting up a very specific environment, installing specific versions of libraries, using some niche software and converting data from one byzantine format to another. Each deviation from the original paper is liable to subtly affect the results in nondeterministic ways. That's why no one really gets surprised or even cares, life is too short to call out all the bad software written in academia.
    - RidingPegasus 8y ago
      > requires setting up a very specific environment, installing specific versions of libraries, using some niche software and converting data from one byzantine format to another Containerisation is fairly mature and simple to use. Many in other fields struggle with these exact same issues and are able to create reproducible environments just fine. I find it amazing that those publishing don't include their implementation, all that work locked away on a rusty hdd.
afarrell 8y ago
Was it an option to present the paper as normal, but then have a sudden surprise twist at the end which left the audience both surprised and feeling the pathos of imagining how they would feel about discovering the last 18 months of their research was invalid? Or am I mistaken in reading "present" as "stand up in front of a crowd of people"?
- naasking 8y ago
  Just thought the same. There was a way to "follow orders" and satisfy your principles in a way that serves science. And if anything happened to him after that, it would be clear that it was a political firing.
astockwell 8y ago
Good for you, +1 to the greater good for your sacrifice.
throwaway1982x 8y ago
Also in bioinformatics: I worked up some statistics that turned out to cleanly and robustly contradict my supervisor's previous research. I was told to reverse my conclusions - refused - and, hey, I no longer work in bioinformatics.
alexnewman 8y ago
It took me less than a minute to find your adviser https://www.linkedin.com/in/jake-boggan-6528944a/ https://www.linkedin.com/in/jake-boggan-6528944a/ -> http://www.biomath.gatech.edu/people/ http://www.biomath.gatech.edu/people/ -> Jake Boggan (Advisor: Bunimovich)
- danieltillett 8y ago
  Probably not the best idea for the OP to post under their own name in a public forum. Jake ask the mods if you can remove your post.
drb91 8y ago
Do grad students have any leverage over their advisors? Surely there must be some incentive to root out professors that degrade the reputation of the school. Public shame? Accusations of fraud? There are more options here than students admit, and some other school would have loved to have you.
- selimthegrim 8y ago
  Transferring grad schools is never easy, and even less so if the adviser decides to start a whispering campaign
- contact_fusion 8y ago
  No, graduate students in general have no leverage over their advisors. A single word from your advisor and you're out of the field. There is little incentive to root out professors for any reason. The process of becoming a professor (grad school -> postdoc (N times) -> tenure track faculty -> tenured professor) is generally believed (by tenured professors, of course) to root out anyone unworthy of the position. You can believe what you want about the efficiency of such a process. Public shame requires public understanding of scientific (mal)practice, so, good luck communicating that. Most of the time, the bad actors in question have already gotten papers past referees; what makes you think the public is capable of more thorough review? Fraud is considered a serious allegation and as a result accusing someone of it requires going through a thorough process involving a host of university administrators, whose incentives are aligned with the profit motives of the university system. Transferring graduate schools is essentially impossible, and even in the exceptionally rare circumstances that it happens, it always involves burnt bridges and often has to do with bigger fish (i.e. your advisor being offered a position elsewhere, and you're lucky enough they take you with them.) Without external funding to support you, you are usually replaceable. All graduate departments receive applications far in excess of the number of students they can support. They certainly will not consider taking on another from a school at which you've proven to be a problem. Academia has already established a quite successful leaky pipeline; the beginning (graduate school) is no different. In academia, hierarchy is the rule, flat organizations the exception. You must purchase your influence, usually at significant cost (and luck is a significant component). As an undergraduate, the system is designed to cater to your interests; as a graduate student, you cater to the university's interests. Scientific integrity is a noble notion, and in some corners of academia, it survives, but it does so in spite of bad actors who thrive in a system designed to produce ten times the number of qualified applicants for each job, all of whom are judged according to easily gamed metrics. It would be nice if things weren't this way. But the problem is, typically... if you decide to get a PhD in science, it is possible that you're already too obsessed with the subject to ever, truly, give it up, especially if it's "just" over working conditions. I can't speak for everyone, but most people leave because they were forced to.
tomcam 8y ago
You rock. Thanks for your integrity. Sorry you had to go through that. In the bigger picture, it drives me crazy that these results aren't welcomed. We learn from failures too. Funding sources should recognize this.
- edoo 8y ago
  You could almost say we mostly learn from failures. If you had blind success with a complex method the only way to test the limits would be tweaking it to failure.
  - tomcam 8y ago
    That's an incredibly eloquent way to put it. Even though it's how I live most of my life I was never smart enough to express it like you did. Thank you.
Fomite 8y ago
Based on my experience on Academia Stack Exchange, in a parallel universe there is a student complaining that their advisor made them rework everything the night before a presentation. Candidly, I probably would have told you to present it as is, and add a caveat to the last slide that this was work in progress (most internal presentations are assumed to be so) and you're still chasing down some problems in your code. The reason for this is two-fold: 1) It's the night before. Many students I have known and mentored aren't at the point in their career that they can "wing" a major presentation. It would be setting them up to fail in a way I couldn't shield them from (I can deal with changing results, but a bad presentation is largely on the student). 2) The quality of your checking is likely to be poor the night before. There's a number of times I've found an error as something was being prepped for presentation/poster printing/etc. and been convinced it changed everything, only to discover after 48 hours of thought and more checking that the difference in results was pretty negligible - especially in the sense of the qualitative take aways from a presentation. This, of course, may not have been your exchange with your PI. But I thought it was worthwhile that there are reasons not to have you change everything the night before that aren't the result of villainous fraud.
- hnmonkey 8y ago
  This seems like a great neutral path forward that shouldn't upset anyone and doesn't paint the research as 100% factual and in-stone, yet. It leaves the door open to continue the work while rectifying the issues. However, it doesn't excuse the way the advisor asked them to just go with it anyway unless there was some nuance left out in that part of the conversation (akin to you what you've stated) we're not hearing.
raverbashing 8y ago
Oh, scientific/research code is hard First of all, a lot of people writing it are inexperienced in SW best practices. Then, turning formulas into code is hard, not to mention some article "handwavings" like showing pseudocode that's almost ok except for "do hard thing in this line" and that line expands into a lot of code. I've also had some weird bugs in code like that (but nothing that would invalidate 18mo of results - btw did you know RAND_MAX is as low as 64k in different compilers, like some older versions of VS?) (And computation times, though the cloud helps a lot with this)
blensor 8y ago
This might sound harsh, but that is something you have to deal with in life (academia or non-academia). What you were tasked to do is present your work so other people made room im their calendars to listen to your talk. Finding an error is no reason/excuse not to present your work. BUT you have to make it clear upfront that there is a problem which might invalidate the results. And maybe already give an estimate how far reaching this error is. If you have been working 18 months on a topic you will have substantial knowledge you can communicate to your peers and often some work inspires other work even if they are not using your findings.
Waterluvian 8y ago
This has me really curious about the state of software engineering principles in academia if you can find bugs that invalidate 18 months of work. I'm not saying it should have been preventable. It just looks like there may be opportunity to improve practices. Admittedly I'm completely naive to the domain. Are there no forms of validation checkpoints you can reach where your foundations are rock solid and well backed with tests and such? I had a Prof in grad school who lost years of cryospheric research data because an external hard drive was stolen. This was in 2010. It was a head scratcher, especially being faculty-adjacent to some of the best CS and engineering faculties in Canada.
- hiker512 8y ago
  I've seldomly seen a PhD student in CS here in Germany with a proper software engineering background. I know several that use Dropbox for version control on their code and don't know how to use git. Some at least know and use svn. That stuff just isn't thought at universities and it's assumed, like oh so many things, that you take it up along the way.
hnmonkey 8y ago
There really needs to be repercussions for this type of thing. Naming and shaming is a good first step towards that utopia. If those responsible for these types of unethical actions never get punished they'll just keep doing it to the detriment of society.

anticensor 8y ago

Why did you not try to publish those problematic results without your adviser's name on? There are venues focused just on negative/non-reproducible/problematic results.

Xcelerate 8y ago

This is par for the course with computational research. I discovered a bug in my code the last week before submitting my PhD dissertation. Luckily, all of my code was organized in a pipeline that could automatically regenerate everything on the fly, but I needed a supercomputer. The queue was too long for Titan, so I set up an impromptu cluster on Azure (it was the only cloud provider with Infiniband at the time) and paid $200 out of pocket to regenerate the correct figures. I wouldn’t be surprised if a significant portion of published computational research has bugs that totally invalidate the conclusions. I think we need to push hard to require all taxpayer-funded research to make any code that results in a journal article publicly available.

thrmsforbfast 8y ago
Open source doesn't ensure quality code. Ideally the code would be part of the peer review process, but code review is really expensive, so who knows how that would play out.
- shkkmo 8y ago
  > Open source doesn't ensure quality code. Yes, but closed source helps ensure that low quality code is hidden from sight. It also means that people who distrust or doubt the conclusions have no chance to identify any bug(s) and disprove the results or conclusions.
  - deleted 8y ago
    [deleted]
- fifnir 8y ago
  It's simple: We stop publishing in papers, and instead adopt smaller chunks of our work as the core publishing units. Each figure should be an individually published entity which contains the entire computational pipeline. Figures are our observations on which we apply logic/philosophy/whatyouwannacallit. Publishing them alongside their relevant code makes the process transparent, reproducible and individually reviewable, as it should be. We can then "publish" comments, observations, conclusions etc on those Figures as a separate thing. Now the logic of the conclusions can be reviewed separately from the statistics and code of the figure.
  - no_identd 8y ago
    I have two words for you: Ted. Nelson.
    - j88439h84 8y ago
      Can you expand on this?
      - lvh 8y ago
        I can’t speak for GP, but Nelson invented hypermedia/hyperlinks and had a vision for the future that included documents including other documents. All of that seems pretty compatible.
  - 8y ago
andyv 8y ago
Computational research is incredibly difficult because it's usually hard to see the effect of a bug. A triangle drawn on the wrong place of a screen can be easy to see, but a typo in your integration subroutine? Hard to spot if you don't catch it when it is born. I also had a few do-overs at the end of my thesis, but fortunately had a cluster standing by...
- BeetleB 8y ago
  >A triangle drawn on the wrong place of a screen can be easy to see, but a typo in your integration subroutine? Hard to spot if you don't catch it when it is born. Well, there's also this notion of testing and regression. As I said in another comment a few days ago: >A few weeks ago I had a conversation with a friend of mine who is wrapping up his PhD. He pointed out that not one of his colleagues is concerned whether anyone can reproduce their work. They use a home grown simulation suite which only they have access to, and is constantly being updated with the worst software practices you can think of. No one in their team believes that the tool will give the same results they did 4 years ago. The troubling part is, no one sees that as being a problem. They got their papers published, and so the SW did its job.
  - fouc 8y ago
    Wow. If it's not reproducible, can it even be called science?
    - BurningFrog 8y ago
      Perhaps it's a reproducible way to build careers?
    - slavik81 8y ago
      It's not really much different from a physical experiment, where the expectation is that you'll have to rebuild the experimental apparatus yourself to reproduce the experiment. An independent implementation of the experiment is neccessary for a full reproduction anyways. If you just run their code again, you'll end up with all their bugs again. (But, don't get me wrong. I like when researchers release their code. It's still very useful.)
nestorD 8y ago
A few days ago in the description of a paper found on hackernews (https://news.ycombinator.com/item?id=18346943 https://news.ycombinator.com/item?id=18346943) : > We also noticed significant improvements in performance of RND every time we discovered and fixed a bug [...]. Getting such details right was a significant part of achieving high performance even with algorithms conceptually similar to prior work. They call bugs 'details' which, I find, is a frightening state of mind for someone publishing an algorithm.
- geezerjay 8y ago
  > They call bugs 'details' which, I find, is a frightening state of mind for someone publishing an algorithm. It really depends on the algorithm. For example, a bug in the random number generator of a stochastic search algorithm that affects, say, the variance of a distribution won't have a relevant impact on the outcome.
  - mlthoughts2018 8y ago
    That could have a huge impact. For example if it affected random draws of a hyperparameter in a Bayesian model, leading to incorrect credible intervals ultimately in the posterior distribution. Or worse, if the RNG bug was affecting the variance of randon samples in some deep component of an MCMC algorithm like NUTS or even simple Metropolis. Depending on the exact nature of the bug, it could even cause the model to violate Detailed Balance, and entirely invalidate sampling based inferences or conclusions.
    - riskneutral 8y ago
      Well that is why you would always run statistical validity tests on the RNG and other intermediate values whenever using a Monte Carlo model in production. Ideally the tests should run as part of a Continuous Integration workflow.
      - mlthoughts2018 8y ago
        If you (dubiously) write your own RNG, sure. But you should never do that. And for library RNGs, you should execute the unit tests of the library. Frankly, running it as part of CI is at best overkill and at worst adds complexity that costs you. If you pin versions of your dependency and isolate the artifact into an in-house artifact repository, so that the library code is literally never changing unless you explicitly modify the version, then you should test it up front, then again only occasionally if you actually have evidence of a bug. And as part of any code review of the code change that introduces a version change.
badpun 8y ago
This is why I'm a bit skeptical about the global warming predictions. AFAIK they're all based on multi-million lines of code models. Not only it's monumentally hard to have such large code base without bugs, but also, in case of a software implementing scientific models, finding bugs is extra hard (compared to software which works in a visible way and has millions of users, such as Linux kernel, games, car's embedded software etc.). An effort to have such model verified (let's say to a NASA standard) to be bug-free would probably cost billions of dollars. And that's only bugs, let's not forget that all the models are approximations, plus the numerical methods used all have their quirks and limitations etc. All in all, the problem seems too hard to tackle for humanity right now.
- nafey 8y ago
  The counterpoint is that we have already seen a steady temperature rise. So even though the specifics of the various simulations might not play out as predicted we can expect temperatures to rise.
  - badpun 8y ago
    Knowing that the temperature will rise is not enough to make policy decisions though. You need to predict the increase's magnitude, as well as practical consequences, such as climate change, how much the sea level will rise etc. And for that you need resonable and bug-free models.
    - leeoniya 8y ago
      > how much the sea level will rise etc this will be the least of our problems. we understand very little about nature to have any reasonable prediction model. we don't know the inflection point which will cause massive collapse of major ecosystems we depend on. all we know is that things are changing fast. faster than many non-human organisms are able to adapt.
  - femidav 8y ago
    Steady? We have seen 20-years rise after a colder period. Nothing out of order.
bad_news_bears 8y ago
Was there a large, I guess “significant” difference between your results pre- and post-bug?
deleted 8y ago
[deleted]
fghtr 8y ago
> I think we need to push hard to require all taxpayer-funded research to make any code that results in a journal article publicly available. "Already 18824 SIGNATURES – sign the open letter now!" https://publiccode.eu https://publiccode.eu

montecarl 8y ago

I spent a couple months implementing a promising optimization technique that was published in Nature. I assumed that I was missing something or had a bug in my code because I could not reproduce any of the results. It turns out, one guy in my research group knew the author. He requested the source code for me to the program used to generate the data for publication. That program did not even come close to doing what the paper claimed due to some very serious bugs. When I brought up these issues, nobody really seemed to care much. It wouldn't be worth anyone's time to try to correct the issue or write to the journal.

BugsJustFindMe 8y ago
> it wouldn't be worth anyone's time to...write to the journal. I'm sorry, what? It almost sounds like everyone in your lab is scum. Hopefully YOU at least spent the 60 seconds to write an email?
- aluren 8y ago
  It's not at all uncommon that implementations provided with papers do not actually do what the paper says it does. You're often lucky when there is an implementation at all. But most of the time it's just that running the exact same implementation under the same conditions requires setting up a very specific environment, installing specific versions of libraries, using some niche software and converting data from one byzantine format to another. Each deviation from the original paper is liable to subtly affect the results in nondeterministic ways. That's why no one really gets surprised or even cares, life is too short to call out all the bad software written in academia.
  - RidingPegasus 8y ago
    > requires setting up a very specific environment, installing specific versions of libraries, using some niche software and converting data from one byzantine format to another Containerisation is fairly mature and simple to use. Many in other fields struggle with these exact same issues and are able to create reproducible environments just fine. I find it amazing that those publishing don't include their implementation, all that work locked away on a rusty hdd.

afarrell 8y ago

Was it an option to present the paper as normal, but then have a sudden surprise twist at the end which left the audience both surprised and feeling the pathos of imagining how they would feel about discovering the last 18 months of their research was invalid? Or am I mistaken in reading "present" as "stand up in front of a crowd of people"?

naasking 8y ago
Just thought the same. There was a way to "follow orders" and satisfy your principles in a way that serves science. And if anything happened to him after that, it would be clear that it was a political firing.

astockwell 8y ago

Good for you, +1 to the greater good for your sacrifice.

throwaway1982x 8y ago

Also in bioinformatics: I worked up some statistics that turned out to cleanly and robustly contradict my supervisor's previous research. I was told to reverse my conclusions - refused - and, hey, I no longer work in bioinformatics.

alexnewman 8y ago

It took me less than a minute to find your adviser https://www.linkedin.com/in/jake-boggan-6528944a/ https://www.linkedin.com/in/jake-boggan-6528944a/ -> http://www.biomath.gatech.edu/people/ http://www.biomath.gatech.edu/people/ -> Jake Boggan (Advisor: Bunimovich)

danieltillett 8y ago
Probably not the best idea for the OP to post under their own name in a public forum. Jake ask the mods if you can remove your post.

drb91 8y ago

Do grad students have any leverage over their advisors? Surely there must be some incentive to root out professors that degrade the reputation of the school. Public shame? Accusations of fraud? There are more options here than students admit, and some other school would have loved to have you.

selimthegrim 8y ago
Transferring grad schools is never easy, and even less so if the adviser decides to start a whispering campaign
contact_fusion 8y ago
No, graduate students in general have no leverage over their advisors. A single word from your advisor and you're out of the field. There is little incentive to root out professors for any reason. The process of becoming a professor (grad school -> postdoc (N times) -> tenure track faculty -> tenured professor) is generally believed (by tenured professors, of course) to root out anyone unworthy of the position. You can believe what you want about the efficiency of such a process. Public shame requires public understanding of scientific (mal)practice, so, good luck communicating that. Most of the time, the bad actors in question have already gotten papers past referees; what makes you think the public is capable of more thorough review? Fraud is considered a serious allegation and as a result accusing someone of it requires going through a thorough process involving a host of university administrators, whose incentives are aligned with the profit motives of the university system. Transferring graduate schools is essentially impossible, and even in the exceptionally rare circumstances that it happens, it always involves burnt bridges and often has to do with bigger fish (i.e. your advisor being offered a position elsewhere, and you're lucky enough they take you with them.) Without external funding to support you, you are usually replaceable. All graduate departments receive applications far in excess of the number of students they can support. They certainly will not consider taking on another from a school at which you've proven to be a problem. Academia has already established a quite successful leaky pipeline; the beginning (graduate school) is no different. In academia, hierarchy is the rule, flat organizations the exception. You must purchase your influence, usually at significant cost (and luck is a significant component). As an undergraduate, the system is designed to cater to your interests; as a graduate student, you cater to the university's interests. Scientific integrity is a noble notion, and in some corners of academia, it survives, but it does so in spite of bad actors who thrive in a system designed to produce ten times the number of qualified applicants for each job, all of whom are judged according to easily gamed metrics. It would be nice if things weren't this way. But the problem is, typically... if you decide to get a PhD in science, it is possible that you're already too obsessed with the subject to ever, truly, give it up, especially if it's "just" over working conditions. I can't speak for everyone, but most people leave because they were forced to.

tomcam 8y ago

You rock. Thanks for your integrity. Sorry you had to go through that. In the bigger picture, it drives me crazy that these results aren't welcomed. We learn from failures too. Funding sources should recognize this.

edoo 8y ago
You could almost say we mostly learn from failures. If you had blind success with a complex method the only way to test the limits would be tweaking it to failure.
- tomcam 8y ago
  That's an incredibly eloquent way to put it. Even though it's how I live most of my life I was never smart enough to express it like you did. Thank you.

Fomite 8y ago

Based on my experience on Academia Stack Exchange, in a parallel universe there is a student complaining that their advisor made them rework everything the night before a presentation. Candidly, I probably would have told you to present it as is, and add a caveat to the last slide that this was work in progress (most internal presentations are assumed to be so) and you're still chasing down some problems in your code. The reason for this is two-fold: 1) It's the night before. Many students I have known and mentored aren't at the point in their career that they can "wing" a major presentation. It would be setting them up to fail in a way I couldn't shield them from (I can deal with changing results, but a bad presentation is largely on the student). 2) The quality of your checking is likely to be poor the night before. There's a number of times I've found an error as something was being prepped for presentation/poster printing/etc. and been convinced it changed everything, only to discover after 48 hours of thought and more checking that the difference in results was pretty negligible - especially in the sense of the qualitative take aways from a presentation. This, of course, may not have been your exchange with your PI. But I thought it was worthwhile that there are reasons not to have you change everything the night before that aren't the result of villainous fraud.

hnmonkey 8y ago
This seems like a great neutral path forward that shouldn't upset anyone and doesn't paint the research as 100% factual and in-stone, yet. It leaves the door open to continue the work while rectifying the issues. However, it doesn't excuse the way the advisor asked them to just go with it anyway unless there was some nuance left out in that part of the conversation (akin to you what you've stated) we're not hearing.

raverbashing 8y ago

Oh, scientific/research code is hard First of all, a lot of people writing it are inexperienced in SW best practices. Then, turning formulas into code is hard, not to mention some article "handwavings" like showing pseudocode that's almost ok except for "do hard thing in this line" and that line expands into a lot of code. I've also had some weird bugs in code like that (but nothing that would invalidate 18mo of results - btw did you know RAND_MAX is as low as 64k in different compilers, like some older versions of VS?) (And computation times, though the cloud helps a lot with this)

blensor 8y ago

This might sound harsh, but that is something you have to deal with in life (academia or non-academia). What you were tasked to do is present your work so other people made room im their calendars to listen to your talk. Finding an error is no reason/excuse not to present your work. BUT you have to make it clear upfront that there is a problem which might invalidate the results. And maybe already give an estimate how far reaching this error is. If you have been working 18 months on a topic you will have substantial knowledge you can communicate to your peers and often some work inspires other work even if they are not using your findings.

Waterluvian 8y ago

This has me really curious about the state of software engineering principles in academia if you can find bugs that invalidate 18 months of work. I'm not saying it should have been preventable. It just looks like there may be opportunity to improve practices. Admittedly I'm completely naive to the domain. Are there no forms of validation checkpoints you can reach where your foundations are rock solid and well backed with tests and such? I had a Prof in grad school who lost years of cryospheric research data because an external hard drive was stolen. This was in 2010. It was a head scratcher, especially being faculty-adjacent to some of the best CS and engineering faculties in Canada.

hiker512 8y ago
I've seldomly seen a PhD student in CS here in Germany with a proper software engineering background. I know several that use Dropbox for version control on their code and don't know how to use git. Some at least know and use svn. That stuff just isn't thought at universities and it's assumed, like oh so many things, that you take it up along the way.

hnmonkey 8y ago

There really needs to be repercussions for this type of thing. Naming and shaming is a good first step towards that utopia. If those responsible for these types of unethical actions never get punished they'll just keep doing it to the detriment of society.