3 ms·

No worries, it's fine. I was specifically talking about peer-reviewed papers and how bad some of them are. I'm working in the space of parallelization and opti

No worries, it's fine. I was specifically talking about peer-reviewed papers and how bad some of them are. I'm working in the space of parallelization and optimization and a lot of papers out there just implement any given algorithm on any hardware of their choice. They describe vaguely what they have done, claim to be faster than any other papers and call it a day. That's worthless in my opinion. For all we know they could have a bug in their code and that's why it's fast. Some may even lie, who knows without code or data. But oh well, I guess it's a step towards graduation for them. So at least they got something for their time (and presumably they have that cool, fast implementation they need for a project and some useful experience, but the paper: worthless).

__MatrixMan__ 2y ago
I took a bioinformatics class recently. The final project was taking wet lab data from a published paper and using it to do our own analysis and then comparing that analysis to the paper. I'm pretty sure I found an error in the paper that we were partially replicating: Their charts are swapped: mutant A's chart is labeled as mutant B and visa versa. While the resemblance of my A chart with their B chart might be a bit subjective, I have a reproducible build of my chart and its labeling that I can trace all the way back to the source data (via nix). It might not be perfect, but at least there's an artifact available for scrutiny... at all. The published version though, it's just a PDF. I can't see their code so that I can point them at the bug that I suspect exists in it. Their data is available on NCBI, but not their code. So yeah, I hear you re: being a bit disappointed with the reproducibility of the computational aspects of these papers. It's somewhat understandable for replication to be difficult in places like psychology or chemistry... but if the work is being done in silico, it should be a matter of two or three commands to re-do it in silico. And if that's not true, we need to fix the tooling until it is. I'd prefer to have the raw data, intermediate artifacts, and their computational relationships, all made available as soon as the researcher has an inkling that they might be worth presenting to someone. Then, when it's time to celebrate those results, they can just reference things that have been publicly searchable and verifiable for a while now. That way computational verification can happen before the paper is even written, and we don't even have to bother writing prose around what turned out to be bogus results.

__MatrixMan__ 2y ago

I took a bioinformatics class recently. The final project was taking wet lab data from a published paper and using it to do our own analysis and then comparing that analysis to the paper. I'm pretty sure I found an error in the paper that we were partially replicating: Their charts are swapped: mutant A's chart is labeled as mutant B and visa versa. While the resemblance of my A chart with their B chart might be a bit subjective, I have a reproducible build of my chart and its labeling that I can trace all the way back to the source data (via nix). It might not be perfect, but at least there's an artifact available for scrutiny... at all. The published version though, it's just a PDF. I can't see their code so that I can point them at the bug that I suspect exists in it. Their data is available on NCBI, but not their code. So yeah, I hear you re: being a bit disappointed with the reproducibility of the computational aspects of these papers. It's somewhat understandable for replication to be difficult in places like psychology or chemistry... but if the work is being done in silico, it should be a matter of two or three commands to re-do it in silico. And if that's not true, we need to fix the tooling until it is. I'd prefer to have the raw data, intermediate artifacts, and their computational relationships, all made available as soon as the researcher has an inkling that they might be worth presenting to someone. Then, when it's time to celebrate those results, they can just reference things that have been publicly searchable and verifiable for a while now. That way computational verification can happen before the paper is even written, and we don't even have to bother writing prose around what turned out to be bogus results.