3 ms·

> so the code is almost never published Even worse is that a friend of mine who is a researcher in neuroscience/BCI has found that many of the papers couldn't

> so the code is almost never published Even worse is that a friend of mine who is a researcher in neuroscience/BCI has found that many of the papers couldn't be reproduced at all with flawed or incorrect conclusions. Since they typically don't publish their code, its very difficult for others to check the correctness.

vog 9y ago
Moreover, in this context "code" is not just Python/C++/Java, but also spreadsheets! Publishing spreadsheets is perhaps an even more urgent issue than publishing other types of code. And of course the underlying data (anonymized) in case of statistical elicitation, but that's another topic. (Open Access, Open Data and Free Software are very closely related, though).
raverbashing 9y ago
Really, the academic world is sometimes obsessed with pseudocode. It might make some didactic sense, but it makes fewer and fewer sense as times go by when you can ship the actual code and shows actual results
fjfaase 9y ago
And even when it is made available, it is of very poor quality. I heard about someome who studied the programs used for analyzing DNA sequences. While tinkering with it, he found a bug, fixed it and submitted a pull request. Two years later, the pull request was still not merged by the maintainer of this 'standard' package. He also told that the software was of a poor quality. For example, in theory, four base pairs could be stored in one byte, instead they are simply storing the letters A,C,T, or G in a single byte for each pair.
- dkersten 9y ago
  Yeah, that’s what my friend was saying too. He mentioned one often-cited paper which, when he fixed a bug he found in the code, no longer supported the papers conclusion at all. He also complained about the poor quality in a lot of the code. :-(
- znfii 9y ago
  I haven't tested this myself, but from what I have heard it is not necessarily a good idea to store several different things in a single byte like this. While it may consume less memory there is a performance cost for each access since you may need to do masks and shifts. Of course, there is a non-trivial performance trade-off with more cycles but more objects fits in cache, etc. Some quick googling gave for example this source which seems to hint that under some situations it is not worth it at least https://stackoverflow.com/questions/4240974/when-is-it-worthwhile-to-use-bit-fields https://stackoverflow.com/questions/4240974/when-is-it-worth... Lastly, it seems like being quite harsh to call the code being of poor quality because of something like that. Unless it really is very important to get the extra memory, just the maintenance cost of packing things efficiently might make it not worth it. Even production code bases such as the JVM will happily use a single byte (or even more?) to store a boolean, and such things probably occur all over the place.
  - kirillseva 9y ago
    unless the author uses memory in the most efficient way possible (preferably writing LLVM-IR directly) it will be hard for reviewers to run DNA analysis algorithms on their smart fridges /s
deepnotderp 9y ago
Even worse is when you realize that's a growing concern in deep learning.

vog 9y ago

Moreover, in this context "code" is not just Python/C++/Java, but also spreadsheets! Publishing spreadsheets is perhaps an even more urgent issue than publishing other types of code. And of course the underlying data (anonymized) in case of statistical elicitation, but that's another topic. (Open Access, Open Data and Free Software are very closely related, though).

raverbashing 9y ago

Really, the academic world is sometimes obsessed with pseudocode. It might make some didactic sense, but it makes fewer and fewer sense as times go by when you can ship the actual code and shows actual results

fjfaase 9y ago

And even when it is made available, it is of very poor quality. I heard about someome who studied the programs used for analyzing DNA sequences. While tinkering with it, he found a bug, fixed it and submitted a pull request. Two years later, the pull request was still not merged by the maintainer of this 'standard' package. He also told that the software was of a poor quality. For example, in theory, four base pairs could be stored in one byte, instead they are simply storing the letters A,C,T, or G in a single byte for each pair.

dkersten 9y ago
Yeah, that’s what my friend was saying too. He mentioned one often-cited paper which, when he fixed a bug he found in the code, no longer supported the papers conclusion at all. He also complained about the poor quality in a lot of the code. :-(
znfii 9y ago
I haven't tested this myself, but from what I have heard it is not necessarily a good idea to store several different things in a single byte like this. While it may consume less memory there is a performance cost for each access since you may need to do masks and shifts. Of course, there is a non-trivial performance trade-off with more cycles but more objects fits in cache, etc. Some quick googling gave for example this source which seems to hint that under some situations it is not worth it at least https://stackoverflow.com/questions/4240974/when-is-it-worthwhile-to-use-bit-fields https://stackoverflow.com/questions/4240974/when-is-it-worth... Lastly, it seems like being quite harsh to call the code being of poor quality because of something like that. Unless it really is very important to get the extra memory, just the maintenance cost of packing things efficiently might make it not worth it. Even production code bases such as the JVM will happily use a single byte (or even more?) to store a boolean, and such things probably occur all over the place.
- kirillseva 9y ago
  unless the author uses memory in the most efficient way possible (preferably writing LLVM-IR directly) it will be hard for reviewers to run DNA analysis algorithms on their smart fridges /s

deepnotderp 9y ago

Even worse is when you realize that's a growing concern in deep learning.