3 ms·

Computational research is incredibly difficult because it's usually hard to see the effect of a bug. A triangle drawn on the wrong place of a screen can be eas

by andyv 8y ago

Computational research is incredibly difficult because it's usually hard to see the effect of a bug. A triangle drawn on the wrong place of a screen can be easy to see, but a typo in your integration subroutine? Hard to spot if you don't catch it when it is born. I also had a few do-overs at the end of my thesis, but fortunately had a cluster standing by...

BeetleB 8y ago
>A triangle drawn on the wrong place of a screen can be easy to see, but a typo in your integration subroutine? Hard to spot if you don't catch it when it is born. Well, there's also this notion of testing and regression. As I said in another comment a few days ago: >A few weeks ago I had a conversation with a friend of mine who is wrapping up his PhD. He pointed out that not one of his colleagues is concerned whether anyone can reproduce their work. They use a home grown simulation suite which only they have access to, and is constantly being updated with the worst software practices you can think of. No one in their team believes that the tool will give the same results they did 4 years ago. The troubling part is, no one sees that as being a problem. They got their papers published, and so the SW did its job.
- fouc 8y ago
  Wow. If it's not reproducible, can it even be called science?
  - BurningFrog 8y ago
    Perhaps it's a reproducible way to build careers?
  - slavik81 8y ago
    It's not really much different from a physical experiment, where the expectation is that you'll have to rebuild the experimental apparatus yourself to reproduce the experiment. An independent implementation of the experiment is neccessary for a full reproduction anyways. If you just run their code again, you'll end up with all their bugs again. (But, don't get me wrong. I like when researchers release their code. It's still very useful.)
    - ascar 8y ago
      This can't be upvoted enough. Makes me thinking that using published source code and reproducing/validating results is completely orthogonal. Maybe it's a good thing the source code for science gets not published.
      - BeetleB 8y ago
        I like the concept of rewriting all the code by an unbiased third party to see if they can reproduce the results, but in practice what this leads to is: 1. People will not bother. It took a lot of minds to come up with the software used (in my friend's case, several PhD's amount of work). No one is going to invest that much effort to invent their own software libraries to get it to work. 2. Even when you do write your own version of the software, there are a lot of subtleties involved in, say, computational physics. Choices you make (inadvertently) affect the convergence and accuracy. My producing a software that gives different results could mean I had a bug. It could mean they did. It could mean we both did. Until both our codes are in the open, no one can know. It is very unlikely that you'll have a case of one group's software giving one result and everyone else's giving another. More like everyone else giving different results. Case in point: https://physicstoday.scitation.org/do/10.1063/PT.6.1.20180822a/full/ https://physicstoday.scitation.org/do/10.1063/PT.6.1.2018082... HN discussion: https://news.ycombinator.com/item?id=17819420 https://news.ycombinator.com/item?id=17819420
  - BeetleB 8y ago
    This issue comes up often on HN, and it used to a lot in /r/science (maybe it still does - I left that subreddit years ago). If there's one thing I could convey to the world from what I learned from my time in academia, it is this: Most scientists at universities do not care about reproducibility.[1] Not only that, many people intentionally omit details from papers so that it is hard for rivals to reproduce their work - they want the edge so they can publish without competition. This isn't a shadowy conspiracy theory - this is what advisors openly tell their students. Search around on HN and reddit and you'll see people saying it. [1] My experience is in condensed matter physics - it may not apply to all of academia.
  - andyv 8y ago
    People doing science programming are the worst programmers in the world. The reason is that they are focused on a calculation and result, not the program. I helped a guy speed up his program once. He was sorting around 10^4 to 10^5 elements using a bubble sort (which he had reinvented).
- petters 8y ago
  >> being updated with the worst software practices you can think of. I can believe that. I have seen code in academia and it was all one or two-letter variables without even linebreaks between statements.

by andyv 8y ago

BeetleB 8y ago

>A triangle drawn on the wrong place of a screen can be easy to see, but a typo in your integration subroutine? Hard to spot if you don't catch it when it is born. Well, there's also this notion of testing and regression. As I said in another comment a few days ago: >A few weeks ago I had a conversation with a friend of mine who is wrapping up his PhD. He pointed out that not one of his colleagues is concerned whether anyone can reproduce their work. They use a home grown simulation suite which only they have access to, and is constantly being updated with the worst software practices you can think of. No one in their team believes that the tool will give the same results they did 4 years ago. The troubling part is, no one sees that as being a problem. They got their papers published, and so the SW did its job.

fouc 8y ago
Wow. If it's not reproducible, can it even be called science?
- BurningFrog 8y ago
  Perhaps it's a reproducible way to build careers?
- slavik81 8y ago
  It's not really much different from a physical experiment, where the expectation is that you'll have to rebuild the experimental apparatus yourself to reproduce the experiment. An independent implementation of the experiment is neccessary for a full reproduction anyways. If you just run their code again, you'll end up with all their bugs again. (But, don't get me wrong. I like when researchers release their code. It's still very useful.)
  - ascar 8y ago
    This can't be upvoted enough. Makes me thinking that using published source code and reproducing/validating results is completely orthogonal. Maybe it's a good thing the source code for science gets not published.
    - BeetleB 8y ago
      I like the concept of rewriting all the code by an unbiased third party to see if they can reproduce the results, but in practice what this leads to is: 1. People will not bother. It took a lot of minds to come up with the software used (in my friend's case, several PhD's amount of work). No one is going to invest that much effort to invent their own software libraries to get it to work. 2. Even when you do write your own version of the software, there are a lot of subtleties involved in, say, computational physics. Choices you make (inadvertently) affect the convergence and accuracy. My producing a software that gives different results could mean I had a bug. It could mean they did. It could mean we both did. Until both our codes are in the open, no one can know. It is very unlikely that you'll have a case of one group's software giving one result and everyone else's giving another. More like everyone else giving different results. Case in point: https://physicstoday.scitation.org/do/10.1063/PT.6.1.20180822a/full/ https://physicstoday.scitation.org/do/10.1063/PT.6.1.2018082... HN discussion: https://news.ycombinator.com/item?id=17819420 https://news.ycombinator.com/item?id=17819420
- BeetleB 8y ago
  This issue comes up often on HN, and it used to a lot in /r/science (maybe it still does - I left that subreddit years ago). If there's one thing I could convey to the world from what I learned from my time in academia, it is this: Most scientists at universities do not care about reproducibility.[1] Not only that, many people intentionally omit details from papers so that it is hard for rivals to reproduce their work - they want the edge so they can publish without competition. This isn't a shadowy conspiracy theory - this is what advisors openly tell their students. Search around on HN and reddit and you'll see people saying it. [1] My experience is in condensed matter physics - it may not apply to all of academia.
- andyv 8y ago
  People doing science programming are the worst programmers in the world. The reason is that they are focused on a calculation and result, not the program. I helped a guy speed up his program once. He was sorting around 10^4 to 10^5 elements using a bubble sort (which he had reinvented).
petters 8y ago
>> being updated with the worst software practices you can think of. I can believe that. I have seen code in academia and it was all one or two-letter variables without even linebreaks between statements.