7 ms·

I've implemented several things from computer science papers in my career now, mostly related to database stuff. They are mostly terribly wrong or show the exac

I've implemented several things from computer science papers in my career now, mostly related to database stuff. They are mostly terribly wrong or show the exact OPPOSITE as to what they claim in the paper. It's so frustrating. Even occasionally, they offer their code used to write the paper and it is missing entire features they claim are integral for it to function properly; to the point that I wonder how they even came up with the results they came up with. My favorite example was a huge paper that was almost entirely mathematics-based. It wasn't until you implemented everything that you would realize it just didn't even make any sense. Then, when you read between the lines, you even saw their acknowledgement of that fact in the conclusion. Clever dude. Anyway, I have very little faith in academic papers; at least when it comes to computer science. Of all the things out there, it is just code. It isn't hard to write and verify what you purport (usually takes less than a week to write the code), so I have no idea what the peer reviews actually do. As a peer in the industry, I would reject so many papers by this point. And don't even get me started on when I send the (now professor) questions via email to see if I just implemented it wrong, or whatever, that just never fucking reply.

reasonableklout 2y ago
Wow, sounds awful. Help the rest of us out - what was the huge paper that didn't work or was actively misleading?
- withinboredom 2y ago
  I'd rather not, for obvious reasons. The less obvious reason is that I don't remember the title/author of the paper. It was back in 2016/17 when I was working on a temporal database project at work and was searching literature for temporal query syntax though.
Lerc 2y ago
For papers with code, I have a seen a tendency to consider the code, not the paper to be the ground truth. If the code works, then it doesn't matter what the paper says, the information is there. If the code doesn't work, it seems like a red flag. It's not an advantage that can be applied to biology or physics, but at least computer science catches a break here.
- withinboredom 2y ago
  > For papers with code, I have a[sic] seen a tendency to consider the code My favorite for those is to search the code for "todo" and look up that part of the paper. Usually, these are the most complicated parts of the paper. Not always though, sometimes they are just trivial things.
DaoVeles 2y ago
It is also frustrating when a papers summary says one thing, you pay for the full thing only to see it is a complete opposite of the claims. Waste of time and money, bleh!
- geysersam 2y ago
  Scihub. Use scihub. What use is it that you pay the publishing company for the paper? The researchers doing the work won't see any of that money.
jltsiren 2y ago
This is a common failure mode when people outside academic CS read CS papers. They take the papers too literally. Computer science studies computation as an abstract concept. The work may be motivated by what happens in the industry, but it's not supposed to produce anything immediately applicable. Papers may include fake justifications and fake applications, because populist politicians decided long ago that all publicly funded research must have practical real-world applications. But you should not take them at face value. Academic CS values abstract results over concrete results, because real-world systems change too rapidly. Real-world results tend to become obsolete too quickly to be relevant in the time scales the academia is supposed to operate. If you are not in academic CS, you should be careful when reading the papers that you understand the context. Most of the time, you are not in the target audience. Even when there is something relevant in the paper, it's probably not the main result, but an idea related to it. And if you start investigating where that idea came from, it probably builds on many earlier results that seemed obscure and practically irrelevant on their own. Peer reviewers usually spend a few hours on a single review (though there is a lot of variation between fields). A week would be so expensive that most established academics would have to stop teaching and doing research and become full-time reviewers.
- deleted 2y ago
  [deleted]
- withinboredom 2y ago
  > Academic CS values abstract results over concrete results, because real-world systems change too rapidly. Real-world results tend to become obsolete too quickly to be relevant in the time scales the academia is supposed to operate. This isn't true. When I'm implementing a paper, I usually go for JUST implementing what they describe, usually by hand. Like if it is a new SQL syntax, I will write a custom recursive descent parser, and hand-roll the query planner, for just the new stuff and hard code some other parts, just as a demonstration. I'm not interested in the industry application part, I'm interested in replicating their work. Once I can replicate it, assuming it is correct, then I will factor the work into a production system. It's this first part that I am frustrated with, not the full implementation in production software.
- 2y ago
meling 2y ago
If it is as bad as you claim, it would be interesting if you could back this up with a falsification report for the papers in question.
- withinboredom 2y ago
  I am working on implementing a paper right now (surprise, surprise, it has issues). Maybe I will do blog posts with a peer review of the paper, and do that for all future papers as well.

reasonableklout 2y ago

Wow, sounds awful. Help the rest of us out - what was the huge paper that didn't work or was actively misleading?

withinboredom 2y ago
I'd rather not, for obvious reasons. The less obvious reason is that I don't remember the title/author of the paper. It was back in 2016/17 when I was working on a temporal database project at work and was searching literature for temporal query syntax though.

Lerc 2y ago

For papers with code, I have a seen a tendency to consider the code, not the paper to be the ground truth. If the code works, then it doesn't matter what the paper says, the information is there. If the code doesn't work, it seems like a red flag. It's not an advantage that can be applied to biology or physics, but at least computer science catches a break here.

withinboredom 2y ago
> For papers with code, I have a[sic] seen a tendency to consider the code My favorite for those is to search the code for "todo" and look up that part of the paper. Usually, these are the most complicated parts of the paper. Not always though, sometimes they are just trivial things.

DaoVeles 2y ago

It is also frustrating when a papers summary says one thing, you pay for the full thing only to see it is a complete opposite of the claims. Waste of time and money, bleh!

geysersam 2y ago
Scihub. Use scihub. What use is it that you pay the publishing company for the paper? The researchers doing the work won't see any of that money.

jltsiren 2y ago

This is a common failure mode when people outside academic CS read CS papers. They take the papers too literally. Computer science studies computation as an abstract concept. The work may be motivated by what happens in the industry, but it's not supposed to produce anything immediately applicable. Papers may include fake justifications and fake applications, because populist politicians decided long ago that all publicly funded research must have practical real-world applications. But you should not take them at face value. Academic CS values abstract results over concrete results, because real-world systems change too rapidly. Real-world results tend to become obsolete too quickly to be relevant in the time scales the academia is supposed to operate. If you are not in academic CS, you should be careful when reading the papers that you understand the context. Most of the time, you are not in the target audience. Even when there is something relevant in the paper, it's probably not the main result, but an idea related to it. And if you start investigating where that idea came from, it probably builds on many earlier results that seemed obscure and practically irrelevant on their own. Peer reviewers usually spend a few hours on a single review (though there is a lot of variation between fields). A week would be so expensive that most established academics would have to stop teaching and doing research and become full-time reviewers.

deleted 2y ago
[deleted]
withinboredom 2y ago
> Academic CS values abstract results over concrete results, because real-world systems change too rapidly. Real-world results tend to become obsolete too quickly to be relevant in the time scales the academia is supposed to operate. This isn't true. When I'm implementing a paper, I usually go for JUST implementing what they describe, usually by hand. Like if it is a new SQL syntax, I will write a custom recursive descent parser, and hand-roll the query planner, for just the new stuff and hard code some other parts, just as a demonstration. I'm not interested in the industry application part, I'm interested in replicating their work. Once I can replicate it, assuming it is correct, then I will factor the work into a production system. It's this first part that I am frustrated with, not the full implementation in production software.
2y ago

meling 2y ago

If it is as bad as you claim, it would be interesting if you could back this up with a falsification report for the papers in question.

withinboredom 2y ago
I am working on implementing a paper right now (surprise, surprise, it has issues). Maybe I will do blog posts with a peer review of the paper, and do that for all future papers as well.