6 ms·

There are also some unfortunate features of academic research. A big one is that the product the researcher cares about is papers and citations, not working sof

There are also some unfortunate features of academic research. A big one is that the product the researcher cares about is papers and citations, not working software per se. Papers you only write once. You can cherry-pick examples, work around bugs, do some facile analysis, publish the paper and then move on never to look at the code again. This means that code and data is typically hacky, unpolished, poorly documented, unmaintained, and unsuitable for public use (or often not publicly inspectable at all). Often the folks doing the grunt work are graduate students, who leave after a few years, leaving a new batch of graduate students to deal with the code without much guidance, so it tends to accrete features but seldom get any refactoring love. Nobody is paying for bug fixes or software maintenance or documentation. Someone coming from industry (or even just a tinkerer) who wants to use the same idea in a production environment often realizes halfway through reimplementation that it isn’t actually better than existing alternatives, and that the analysis/benchmarks in the paper were limited or misleading for a general context. Sorting the wheat from the chaff can be pretty demanding, ideas can sometimes be obfuscated by abstruse mathematical formalism, and academic researchers don’t typically go out of their way to make their output usable for non-academics.

3pt14159 9y ago
True, but sometimes you commercialize something with some insights you bring from industry that never would have occurred to the original academic that made the piece in the first place. Basically what I and another guy did in 2012. We were stupid though, took the first real acquisition offer. That's the problem with being in your twenties you want money ASAP. "$1m today or $10m three years from now? Take the $1m, we'll figure out how to get the other millions later." I think it's another reason why the rich keep getting richer. Muuuuuch easier to turn down $1m when daddy is worth $50m.
joe_the_user 9y ago
Well, the thing with academic research is that it involves a "community of researcher" whose activities involve "evolving an idea". And in such a situation, the result is primarily expressed by its text/human-language representation while code is secondary to this. While this is problematic to verifying the results, it should be noted that code as a result has significant problems. Code doesn't have a long history of being read, interrogated and modified while human-language does. As the article effectively says, don't underestimate the significant value of putting a result in historical context. A human-language description can give a clue whether X has been done before, perfectly functioning code might not. This question touches on "programming literacy", which in the extreme asks, "how can computer programs also search as _documents_ in the fashion scientific papers have historically served as documents." So far, I don't think any answer has really appeared. Perhaps that problem itself is "AI complete", present either a terrible or green pastures for research.
gHosts 9y ago
Indeed, Tinkerer vs Researcher is a false dichotomy. Far Better would be to compare Maker vs Researcher. How "Production Ready" is the result of both activities is the interesting question. How well covered with tests? How maintainable? How re-usable in broader contexts? How readable and understandable? I used to live on the margins of the Physics community and blamed myself for being a really really Bad Researcher for having no patience with the Physics papers I read. Then one day I had an epiphany. If I were to look at the physics journals the same way as I look documentation for software or hardware.... I would be really really Angry. The average research paper is extraordinarily badly written and unreadable. They have adopted as best practices, styles and habits, that you'd never forgive in a datasheet. If you picked up a datasheet written like a physics paper.... You'd throw it against the wall and spec a different part from a semi-decent manufacturer. (Hint: Datasheets are notorious for being terse and indigestible and badly written.) If you are not from a first world university with a first world library at hand with paid for subscriptions to the premium journals and a fat budget to cover pagefees... Scholarship be damned. It's a rort. It's a racket supporting a closed shop guild. It's not Good Manners except in the sense of Manners defining a "Gentlemens Club".
dpwm 9y ago
A few years ago I found myself reading pretty much every paper on methods that could give a better looking solution to "deblurring" an image than the more established methods. This is really a field which has overlap with many different areas of research. You've got traditional iterative methods through to bayesian methods and machine learning methods. Each of these approaches seemed to be relatively isolated from each other and largely cited from their own subfield, despite all solving variants of the same problem. When they did cite outside the field, they tended to cite less relevant or far from state-of-the-art papers. What was amazing was the number of papers where you would have a plot showing how amazing the new deconvolution method was in the presence of noise compared to other methods, often using a vaguely defined metric. This plot would take up about half a page, usually following a few example images that were set very very small, often ~2cm and usually in a grid. That was always deeply suspicious. The other thing I remember were models with tons of unexplained parameters, which were carefully tuned to give good results for that image. But really it was guesswork. Sometimes the authors would carefully avoid well-cited work in another field that they could easily have found. As a tinkerer with a scarcity of time you learn where not to waste your time and move on.
- joe_the_user 9y ago
  Having tried to find image-processing research/scholarship around 2009, before the neural net explosion really set-in, I have to say the field was a complete rag-bag of fragile, vague methods. And efforts to extend these didn't help them. Which is to say that no all incremental, scholarly research is necessarily what's needed. In fact, I think what the OP describes is effectively what Thomas Kuhn calls "normal science" [1], which can be great until it becomes terrible - ie, "normal science" periodically reaches crises that require a different approach (inspiration, insight, iconoclasm, etc, whatever). But that shouldn't detract from the point that anyone should know what traditional research, that if someone is going to engage in something other than normal science, they would benefit from being really aware of why and how they are doing this. [1] https://en.wikipedia.org/wiki/Normal_science https://en.wikipedia.org/wiki/Normal_science
  - bigiain 9y ago
bigiain 9y ago
At least some of academia recognise and acknowledge the state of their "hacky, unpolished, poorly documented, unmaintained, and unsuitable for public use" code: http://matt.might.net/articles/crapl/ http://matt.might.net/articles/crapl/
MikkoFinell 9y ago
> ideas can sometimes be obfuscated by abstruse mathematical formalism They are trying to communicate with other researchers who actually care that the proofs are correct. Maybe you don't, if you're simply trying to reimplement some algorithm, but criticizing research papers for using mathematical formalism is a bit ridiculous.
- Firadeoclus 9y ago
  You might be dealing with mathematical informalism, inconsistent and idiosyncratic notation which is never truly explained anywhere. Having both mathematical notation and prose can help uncover gaps in understanding and works much better as a system of organising human knowledge.
- deleted 9y ago
  [deleted]
fsloth 9y ago
Sometimes the alternative to published research is no material at all. In this instance you need to pay the price of working in an obscure domain, research literature, and experiment what works and how. As a concrete example, boolean operations on polygon meshes still have even not one "de facto" implementation. You can find papers from early 70:s, but each method has one catch or another. Compare this to, for example to sort alogrithms, where just quoting the method name gives sufficient information - "quicksort or mergesort"? The only source are few obscure books,a ton of research papers, and some open source libraries with very little documentation on the algorithmic specifics. In this instance I'm very happy the research papers exist - they give lots of information and ideas, on what has been tried and so on. If not for the papers, the occasional tinkerer would have all of that ... decades of academic research ... to fumble through, all alone.

3pt14159 9y ago

True, but sometimes you commercialize something with some insights you bring from industry that never would have occurred to the original academic that made the piece in the first place. Basically what I and another guy did in 2012. We were stupid though, took the first real acquisition offer. That's the problem with being in your twenties you want money ASAP. "$1m today or $10m three years from now? Take the $1m, we'll figure out how to get the other millions later." I think it's another reason why the rich keep getting richer. Muuuuuch easier to turn down $1m when daddy is worth $50m.

joe_the_user 9y ago

Well, the thing with academic research is that it involves a "community of researcher" whose activities involve "evolving an idea". And in such a situation, the result is primarily expressed by its text/human-language representation while code is secondary to this. While this is problematic to verifying the results, it should be noted that code as a result has significant problems. Code doesn't have a long history of being read, interrogated and modified while human-language does. As the article effectively says, don't underestimate the significant value of putting a result in historical context. A human-language description can give a clue whether X has been done before, perfectly functioning code might not. This question touches on "programming literacy", which in the extreme asks, "how can computer programs also search as _documents_ in the fashion scientific papers have historically served as documents." So far, I don't think any answer has really appeared. Perhaps that problem itself is "AI complete", present either a terrible or green pastures for research.

gHosts 9y ago

Indeed, Tinkerer vs Researcher is a false dichotomy. Far Better would be to compare Maker vs Researcher. How "Production Ready" is the result of both activities is the interesting question. How well covered with tests? How maintainable? How re-usable in broader contexts? How readable and understandable? I used to live on the margins of the Physics community and blamed myself for being a really really Bad Researcher for having no patience with the Physics papers I read. Then one day I had an epiphany. If I were to look at the physics journals the same way as I look documentation for software or hardware.... I would be really really Angry. The average research paper is extraordinarily badly written and unreadable. They have adopted as best practices, styles and habits, that you'd never forgive in a datasheet. If you picked up a datasheet written like a physics paper.... You'd throw it against the wall and spec a different part from a semi-decent manufacturer. (Hint: Datasheets are notorious for being terse and indigestible and badly written.) If you are not from a first world university with a first world library at hand with paid for subscriptions to the premium journals and a fat budget to cover pagefees... Scholarship be damned. It's a rort. It's a racket supporting a closed shop guild. It's not Good Manners except in the sense of Manners defining a "Gentlemens Club".

dpwm 9y ago

A few years ago I found myself reading pretty much every paper on methods that could give a better looking solution to "deblurring" an image than the more established methods. This is really a field which has overlap with many different areas of research. You've got traditional iterative methods through to bayesian methods and machine learning methods. Each of these approaches seemed to be relatively isolated from each other and largely cited from their own subfield, despite all solving variants of the same problem. When they did cite outside the field, they tended to cite less relevant or far from state-of-the-art papers. What was amazing was the number of papers where you would have a plot showing how amazing the new deconvolution method was in the presence of noise compared to other methods, often using a vaguely defined metric. This plot would take up about half a page, usually following a few example images that were set very very small, often ~2cm and usually in a grid. That was always deeply suspicious. The other thing I remember were models with tons of unexplained parameters, which were carefully tuned to give good results for that image. But really it was guesswork. Sometimes the authors would carefully avoid well-cited work in another field that they could easily have found. As a tinkerer with a scarcity of time you learn where not to waste your time and move on.

joe_the_user 9y ago
Having tried to find image-processing research/scholarship around 2009, before the neural net explosion really set-in, I have to say the field was a complete rag-bag of fragile, vague methods. And efforts to extend these didn't help them. Which is to say that no all incremental, scholarly research is necessarily what's needed. In fact, I think what the OP describes is effectively what Thomas Kuhn calls "normal science" [1], which can be great until it becomes terrible - ie, "normal science" periodically reaches crises that require a different approach (inspiration, insight, iconoclasm, etc, whatever). But that shouldn't detract from the point that anyone should know what traditional research, that if someone is going to engage in something other than normal science, they would benefit from being really aware of why and how they are doing this. [1] https://en.wikipedia.org/wiki/Normal_science https://en.wikipedia.org/wiki/Normal_science
- bigiain 9y ago

bigiain 9y ago

At least some of academia recognise and acknowledge the state of their "hacky, unpolished, poorly documented, unmaintained, and unsuitable for public use" code: http://matt.might.net/articles/crapl/ http://matt.might.net/articles/crapl/

MikkoFinell 9y ago

> ideas can sometimes be obfuscated by abstruse mathematical formalism They are trying to communicate with other researchers who actually care that the proofs are correct. Maybe you don't, if you're simply trying to reimplement some algorithm, but criticizing research papers for using mathematical formalism is a bit ridiculous.

Firadeoclus 9y ago
You might be dealing with mathematical informalism, inconsistent and idiosyncratic notation which is never truly explained anywhere. Having both mathematical notation and prose can help uncover gaps in understanding and works much better as a system of organising human knowledge.
deleted 9y ago
[deleted]

fsloth 9y ago

Sometimes the alternative to published research is no material at all. In this instance you need to pay the price of working in an obscure domain, research literature, and experiment what works and how. As a concrete example, boolean operations on polygon meshes still have even not one "de facto" implementation. You can find papers from early 70:s, but each method has one catch or another. Compare this to, for example to sort alogrithms, where just quoting the method name gives sufficient information - "quicksort or mergesort"? The only source are few obscure books,a ton of research papers, and some open source libraries with very little documentation on the algorithmic specifics. In this instance I'm very happy the research papers exist - they give lots of information and ideas, on what has been tried and so on. If not for the papers, the occasional tinkerer would have all of that ... decades of academic research ... to fumble through, all alone.