5 ms·

I mean you're totally 100% right, but I don't think he wrote that with the intent of harm. I think in his post is a more serious issue. Academic papers are incr

by jsharf 9y ago

I mean you're totally 100% right, but I don't think he wrote that with the intent of harm. I think in his post is a more serious issue. Academic papers are incredibly hard to understand. I received an undergrad degree in CS and spend a lot of time teaching myself theoretical CS. I often struggle to read CS papers, and find it hard to imagine many of them being useful to anyone without a PhD. For instance I was reading a paper on computer vision which used a Bayesian Network to predict a depth map from a regular 2D color image. After spending several weeks, I felt like I understood the article almost well enough to implement it. Thing is, there were some steps in the learning algorithm which relied on a specific kind of optimization (I forget which), but they didn't specify which parameters were used, so it wasn't obvious how to implement it to me. My friend has a PhD in EE (on the theoretical side, his education focused heavily on statistics and optimization) and he only looked at it for 30m-1hr, but he wasn't sure what they meant either. I can always just fall back to gradient descent instead of the optimization parts of the algorithm I don't understand, but that impacts performance, and I got the feeling I'd spend hours implementing this only to have something which can't perform in real-time so I gave up. It was just a side project. It's frustrating after devoting weeks to this. Reading and attempting to implement a research paper in my own field is incredibly hard. I feel like if research papers were written in a more accessible, less dry manner and specified details important to implementation, the world could be a better place. If it were easier to implement these things as side projects, you might have people experimenting with these things at home and then starting businesses out of them.

SubiculumCode 9y ago
Failure to fully specify the methods is a problem, but it is an institutionally wide issue related to a number of complex issues. In honest open areas, this lack is due to issues with space in journal formats. In other fields, it may be to protect IP...ugh
spaceseaman 9y ago
I suppose I sensed more malice than might have been there, and I agree with you. I also find many papers difficult to read and especially lacking important implementation details. I've found that many researchers are very open to questions via email, however. Most of the time I'll get a response that can clarify a thing or two. I personally feel that researchers just need to be better writers though. Most of the time, the writing is just plain bad. Perhaps there needs to be a way to enforce a bare minimum in writing quality. Personally I wouldn't be surprised to see more researchers employ editors for themselves.
bocklund 9y ago
> After spending several weeks, I felt like I understood the article almost well enough to implement it. How long do you think it took the researcher to develop the idea? Probably longer than a few weeks. Even if they could have communicated better, it's still the case that you, a non-expert (my apologies if this is untrue, I am assuming based on the context), can decipher and implement the method in a few weeks!
Denzel 9y ago
If I was you, I'd definitely reach out to the author(s). They're usually surprisingly responsive. And they love to hear from people interested in and using their work.
jampekka 9y ago
As a PhD student doing computational stuff, I agree with this fully. I'd say that in many, probably majority, of cases it is not possible implement the algorithm based on just the article. Many corner cases needed are omitted from the paper and if a pseudocode is given, it may have huge steps handwaved just giving a line like "optimize this functional". It's one of the most mindboggling things (the publishing racket is perhaps worse) about academia that CS papers introducing an algorithm aren't required to publish their implementation, even though there necessarily seems to be one doing eg. simulation studies. I don't really understand the rationale behind omitting the implementation. Maybe people write such crappy code that they're ashamed of publishing it. Or it's the more sinister scenario that the algorithm is actually crappier than the paper claims.
- AstralStorm 9y ago
  Latter has happened quite a few times in experience. Algorithm performed as designed on the training or example set just to fail terribly on real life data. Or someone handwaved important things like having an information function be available (impossible, used only in proofs of correctness, way of estimating it is critical). Or a key assumption on input was just mentioned somewhere in the depths of the paper. Or a very specific way of measuring the result hides the deficiencies. (Similar to p-hacking or misusing stats in medicine.)
sevensor 9y ago
About 50% of the journal articles I read in grad school (engineering) were straight-up BS. Unfortunately, it took me about three years to figure out which 50%. It's sounds like the paper you struggled with was designed to obfuscate the particular conditions under which its results were produced. This may be beause one of the authors thinks there's money to be made from the secret sauce, or because they're misrepresenting how broadly applicable the result is. Or because they're sloppy. Or all three. At a certain point, you have to stop blaming yourself for not understanding the literature. Sometimes it wasn't meant to be understood.
brudgers 9y ago
I think academic CS papers are hard to understand because computer science is hard. I mean Knuth is an excellent writer and doesn't use a lot of jargon and yet, The Art of Computer Programming is extremely difficult for most people due to the subject matter. This despite much of the subject matter not being at the cutting edge of computer science.

by jsharf 9y ago

SubiculumCode 9y ago

Failure to fully specify the methods is a problem, but it is an institutionally wide issue related to a number of complex issues. In honest open areas, this lack is due to issues with space in journal formats. In other fields, it may be to protect IP...ugh

spaceseaman 9y ago

I suppose I sensed more malice than might have been there, and I agree with you. I also find many papers difficult to read and especially lacking important implementation details. I've found that many researchers are very open to questions via email, however. Most of the time I'll get a response that can clarify a thing or two. I personally feel that researchers just need to be better writers though. Most of the time, the writing is just plain bad. Perhaps there needs to be a way to enforce a bare minimum in writing quality. Personally I wouldn't be surprised to see more researchers employ editors for themselves.

bocklund 9y ago

> After spending several weeks, I felt like I understood the article almost well enough to implement it. How long do you think it took the researcher to develop the idea? Probably longer than a few weeks. Even if they could have communicated better, it's still the case that you, a non-expert (my apologies if this is untrue, I am assuming based on the context), can decipher and implement the method in a few weeks!

Denzel 9y ago

If I was you, I'd definitely reach out to the author(s). They're usually surprisingly responsive. And they love to hear from people interested in and using their work.

jampekka 9y ago

As a PhD student doing computational stuff, I agree with this fully. I'd say that in many, probably majority, of cases it is not possible implement the algorithm based on just the article. Many corner cases needed are omitted from the paper and if a pseudocode is given, it may have huge steps handwaved just giving a line like "optimize this functional". It's one of the most mindboggling things (the publishing racket is perhaps worse) about academia that CS papers introducing an algorithm aren't required to publish their implementation, even though there necessarily seems to be one doing eg. simulation studies. I don't really understand the rationale behind omitting the implementation. Maybe people write such crappy code that they're ashamed of publishing it. Or it's the more sinister scenario that the algorithm is actually crappier than the paper claims.

AstralStorm 9y ago
Latter has happened quite a few times in experience. Algorithm performed as designed on the training or example set just to fail terribly on real life data. Or someone handwaved important things like having an information function be available (impossible, used only in proofs of correctness, way of estimating it is critical). Or a key assumption on input was just mentioned somewhere in the depths of the paper. Or a very specific way of measuring the result hides the deficiencies. (Similar to p-hacking or misusing stats in medicine.)

sevensor 9y ago

About 50% of the journal articles I read in grad school (engineering) were straight-up BS. Unfortunately, it took me about three years to figure out which 50%. It's sounds like the paper you struggled with was designed to obfuscate the particular conditions under which its results were produced. This may be beause one of the authors thinks there's money to be made from the secret sauce, or because they're misrepresenting how broadly applicable the result is. Or because they're sloppy. Or all three. At a certain point, you have to stop blaming yourself for not understanding the literature. Sometimes it wasn't meant to be understood.

brudgers 9y ago

I think academic CS papers are hard to understand because computer science is hard. I mean Knuth is an excellent writer and doesn't use a lot of jargon and yet, The Art of Computer Programming is extremely difficult for most people due to the subject matter. This despite much of the subject matter not being at the cutting edge of computer science.