4 ms·

To be honest I'm surprised it's not higher. Relevant discussion: https://news.ycombinator.com/item?id=17789308 https://news.ycombinator.com/item?id=17789308 Th

by aluren 8y ago

To be honest I'm surprised it's not higher. Relevant discussion: https://news.ycombinator.com/item?id=17789308 https://news.ycombinator.com/item?id=17789308 The bottom line is that there's not much incentive for doing the boring statistical validations (I can tell you that no one likes doing statistics apart from statisticians, and not even all of them) and verifying that everything is reproductible, and a huge incentive for, let's say, 'arrange' a thing or two so that the paper looks better. So many people just kinda do it, and it slips through the cracks because: -In many fields, the reviewers are not statisticians themselves -No one really bothers to download the data, setup the environment and libraries, and reproduce the exact steps taken to obtain the very same figures in a paper. Which is understandable given how doing all of that can be such a chore. Anyway, most of the time the exact steps aren't even described. No, jupyter notebooks aren't a solution either (it would take too long to explain why and the post is already long). -In many cases, the results turn out to be true anyway so people don't notice they were initially put forward with fraudulent validations -When they turn out to be false, people just shrug and move on with what's actually true. Bogus results often fail to stand the test of time and get forgotten quickly despite initial hype. No one bothers to say: "Hey, that paper from 8 years ago is bullshit and their authors are hacks!" because nobody cares. -There's a huge psychological barrier to actually call out one of your peers and affirm that they're an impostor and their work is bogus. Especially when said impostor happens to be a big name in the field, that part of you still doesn't believe they would commit fraud, not to mention the social repercussions and backlash of doing such an accusation. We scientists aren't a very confrontational bunch. So most of the time it kinda works and we're all bumbling along hoping to find some modicum of truth at the end of the road with minimal harm done. But of course sometimes you get these guys who kick off their entire career on a high profile, much hyped fabricated result (Woo-Suk) or even a series of bogus papers (Sato), and that may lead to long-term harm. The good news is that hyped papers or very prominent figures quickly attract scrutiny from their peers, and sooner or later reality catches up with them as labs around the world fail to replicate their 'breakthroughs'. All in all, I'd say we're doing fine. We're just not the ethereal source of truth that some people hold us to be, the very same people who, after claming that 'God is dead', are very quick to replace Him with His sillicon-based equivalent around which we would act like priests, except with lab coats in lieu of clerical garments.

yesenadam 8y ago
Ok, so why aren't jupyter notebooks a solution?! I was gonna suggest that also in response to another comment above suggesting figures should be published 'alongside their relevant code'.
- aluren 8y ago
  They are so unwieldy. JSON formatting means the ipynb format is a huge mess to handle by simple means, so you have to open a browser window to do anything. Loading, editing, formatting, saving is just... ugh. You can't do proper diffs (though some tools are trying to alleviate that). Loading, stopping and restarting kernels is excruciatingly slow. Because cell execution isn't necessarily done in order it can become very, very easy to lose yourself and feel like you're back in the 70's experiencing GOTOs and the joys of spaghetti code. And that's from my point of view with a technical background. Imagine explaining notebooks to a biologist to whom writing any line of code is still something new and daunting. Imagine their reaction when they happen to mess up one cell in their own execution order and alter all the subsequent workflow, try to grok some of the magic %commands, or get some cryptic error message due to some config file not having the proper rights because they installed a library with 'pip --user' and not 'sudo pip'. It's not realistic to think that an entire community of people who aren't technically minded, many of which actually loathe or fear anything looking like code, is going to adopt a tool that even technically minded people struggle to use they way it's intended. On top of that, many steps necessary to reproduce a pipeline typically need to load enormous datasets. Terabytes of simulated protein structures, hundreds of gigabytes of sequencing reads, phylogenetic trees, alignment files, what have you. Once you somehow acquire that dataset, you need the appropriate tools, many of which need to be specifically compiled for your platform, then run them onto a powerful machine if you don't want the pipeline to take months to complete, etc. (And that's if you didn't use any proprietary software or any GUI based application with no command line interface.) You can't exactly load all of that into Github+Mybinder and call it a day. You can't ask a community of people that doesn't like coding to learn about Docker containers either. Nevertheless, we (at our lab) do use notebooks when we can because we know they're fashionable. We can only present parts of our results, though, due to the aforementioned constraints, but it's still pretty looking and people like them, so we write short demos using them.

by aluren 8y ago

yesenadam 8y ago

Ok, so why aren't jupyter notebooks a solution?! I was gonna suggest that also in response to another comment above suggesting figures should be published 'alongside their relevant code'.

aluren 8y ago
They are so unwieldy. JSON formatting means the ipynb format is a huge mess to handle by simple means, so you have to open a browser window to do anything. Loading, editing, formatting, saving is just... ugh. You can't do proper diffs (though some tools are trying to alleviate that). Loading, stopping and restarting kernels is excruciatingly slow. Because cell execution isn't necessarily done in order it can become very, very easy to lose yourself and feel like you're back in the 70's experiencing GOTOs and the joys of spaghetti code. And that's from my point of view with a technical background. Imagine explaining notebooks to a biologist to whom writing any line of code is still something new and daunting. Imagine their reaction when they happen to mess up one cell in their own execution order and alter all the subsequent workflow, try to grok some of the magic %commands, or get some cryptic error message due to some config file not having the proper rights because they installed a library with 'pip --user' and not 'sudo pip'. It's not realistic to think that an entire community of people who aren't technically minded, many of which actually loathe or fear anything looking like code, is going to adopt a tool that even technically minded people struggle to use they way it's intended. On top of that, many steps necessary to reproduce a pipeline typically need to load enormous datasets. Terabytes of simulated protein structures, hundreds of gigabytes of sequencing reads, phylogenetic trees, alignment files, what have you. Once you somehow acquire that dataset, you need the appropriate tools, many of which need to be specifically compiled for your platform, then run them onto a powerful machine if you don't want the pipeline to take months to complete, etc. (And that's if you didn't use any proprietary software or any GUI based application with no command line interface.) You can't exactly load all of that into Github+Mybinder and call it a day. You can't ask a community of people that doesn't like coding to learn about Docker containers either. Nevertheless, we (at our lab) do use notebooks when we can because we know they're fashionable. We can only present parts of our results, though, due to the aforementioned constraints, but it's still pretty looking and people like them, so we write short demos using them.