5 ms·

This can only end if peer-reviewed journals require source code (and if possible datasets) to be made available as well. High-impact journals have the weight to

This can only end if peer-reviewed journals require source code (and if possible datasets) to be made available as well. High-impact journals have the weight to enforce such policies. It's true that third parties can apply methods easily to new data. But it is a testimony to the method, and references will help building the reputation of the original inventor. Another concern only addressed in the comments on this blog post is that most scientists do not produce beautiful programs. The reasons are twofold: - Programs are hacked together as quickly as possible to produce results. Scientists are mostly concerned with testing their theories, and not so much in producing software for public consumption. - Most scientists are not great programmers. Consequently, scientists usually do not want to make their source code available. This situation sucks, given that in many countries taxpayers fund science.

RBerenguel 16y ago
Yes, taxpayers fund science, but commenting, beautifying and documenting code is not what a physicist/mathematician/climate researcher wants to spend his time. Usually you want to be doing research, may it be by direct coding or by doing something else, related. Doing this kind of stuff is far worse than filling grant proposals or doing other bureaucratic stuff. On the other side, I disagree with "most scientists are not great programmers". What is a "great programmer"? In my definition, it is someone who can write a program to solve a problem without too much hassle. And a lot of scientists I know satisfy this to terrific levels. Of course, they use no orthogonality, nor source code control, nor do extreme programming and usually don't write test cases. They just do what is asked as quickly as possible to keep on doing what is needed to do.
- Maro 16y ago
  I disagree strongly about your definition of "great programmer". Your definition is of a "somebody who can program".
  - RBerenguel 16y ago
    I know a lot of people who "can program" and when confronted with a problem would spend a whole afternoon deciding if they can do it in the main function or need to code separate pieces and then a week to really do it. A great programmer just picks the problem and writes the code that solves it.
    - jemfinch 16y ago
      You've rather obviously never been exposed to real great programmers. Your "can program" definition is actually describing a really bad programmer, and your "great programmer" definition describes most kids who make it past one semester of high school computer science class.
      - RBerenguel 16y ago
        As a side note, your 'obviously' probably means that most mathematicians I know are therefore not 'great programmers'. Ok, maybe my definition is flawed. But when they have to do some program to solve anything, they'll find a way to do it.
      - RBerenguel 16y ago
        Probably not to 'real' great programmers. But I've given classes on numerical algorithms for the CS program in our university. None of these guys (and gals) except for one or two out of 70 or 80 will be better than "someone who can program".
  - kenjackson 16y ago
    "Somebody who can program" have written probably 99% of used applications in the world today. From Bill Joy to Donald Knuth to David Cutler to Linus Torvalds to Guy Steele to Jamie Zawinsky to Guido van Rossum to John Carmack, and almost everyone in between. Great programmers somehow only appear to write books and give pristine examples of how you build infinitely extensible architectures.
    - Maro 16y ago
      John Carmack produces the most maintainable and readable codebases. I happen to have the Quake3 source code up on my github at, so you can see for yourself: https://github.com/mtrencseni/quake3 https://github.com/mtrencseni/quake3 Donald Knuth is the author of literate programming, which is a framework for writing human-readable programs: http://www-cs-faculty.stanford.edu/~uno/lp.html http://www-cs-faculty.stanford.edu/~uno/lp.html
      - kenjackson 16y ago
        Have you read how each of them actually write their code? Both write them precisely how the other poster noted. And I've read the Quake 3 code extensively. Great code, but certainly not pristine, and I'm sure if you handed it to code review to virtually anyone you know, they'd find a whole bunch of sylistic and architectural issues with it. Like take a look at the playerDie code. You're telling me you wouldn't have said, "Rewrite this?" if you a colleague handed this to you? And yes, Knuth is the author of literate programming, but that's not how the code started out. Read his letters on computer science. Tarjan wrote a similar thing, I think in his ACM Turing Award lecture. I picked those names because they are the best our industry has. But even with that, they all pretty much write code the way the previous post noted.
- hnal943 16y ago
  Is it possible that in the midst of their no-comments, no-test-cases spaghetti they introduce a few bugs that affect the results? If so, they need to release their code to be scrutinized along with their findings.
  - RBerenguel 16y ago
    I don't see where I said it was spaghetti code. Sometimes a goto in C can save quite a few running cycles. Write a quick Runge-Kutta-Fehlberg integrator without it.
- EdiX 16y ago
  > Yes, taxpayers fund science, but commenting, beautifying and documenting code is not what a physicist/mathematician/climate researcher wants to spend his time. Usually you want to be doing research Hunting down all the right citations, putting results in good prose, formatting the paper to the standard of where you are trying to get it published, responding to peer reviews (as silly as they can get), doing peer reviews for others... all things that are not "doing research" (strictly speaking) and to some are not enjoyable. But they have to be done anyway, it's a quality standard that has to be met, everyone suffers for it and everyone benefits from it. It's just that the scientific community has let itself set really lousy standards for code sharing. It should be rectified.
Maro 16y ago
I have an MSc in CS and then did (still doing) a degree in Physics. I can report that you are right, most scientists are not great programmer. Most scientists just don't care about readability, maintainability, warnings, library dependencies, cross-platform issues, versioning, testing, etc. I think it'd be perfectly reasonable for society and the gov't to demand that scientists who produce results based on data and data processing make the data and the programs available to their peers and the general public, and also demand that it satisfies some reasonable software engineering expectations. (These will be set by their own peers anyway.) It is accepted by scientists that it is important to communicate research findings in the form of well-written human-readable papers. Over time the community must accept that including well-written machine-parsable data and code is just as important part of the scientific communication framework. Finally, a personal anecdote. I was contacted by a senior researcher to work on a cutting-edge numeric simulation project, and he sent me the source code, but made it clear that it is "top secret", even though it is gov't funded, as it is much better than its competitors, also gov't funded. He estimated that this advantage will last for several years, meaning several papers.
- RBerenguel 16y ago
  Deeply disturbing fact. The main researcher in our group works in cutting edge numeric simulations, and although he does not publish his code (mostly because "you can code it yourself! It is really easy, listen" and go on for 3 hours), if you were to ask, he would give it to you without any problems. But according to what you say, this means that he (still programs in Fortran) would need to start caring about versioning, cross-platform stuff (fortran, remember), readability (well, it's fortran, for me it is always unreadable) and maintainability. In addition to his usual duties of "doing research", giving lectures, answering students, advising PhD's and asking for grants. The problem is most likely to be solved when everyone in the field can code the algorithm in the paper. If the peer reviewer can write a program following the algorithm and applied on the data it validates the results, this ends up working well. I doubt (very profoundly) that giving the source will result in better peer reviews. The peer reviewer may have the code, but he will most likely compile, run and check that the numbers are the same.
  - 16y ago
Daishiman 16y ago
You'll fix source code quality problems the day that research grants can contemplate the money to pay for scientific programming, meaning people who know their shit on modeling and running simulations on distributed systems. Do bear in mind that those people are not cheap, so it's doubtful that anyone would want to incur the expense.

RBerenguel 16y ago

Yes, taxpayers fund science, but commenting, beautifying and documenting code is not what a physicist/mathematician/climate researcher wants to spend his time. Usually you want to be doing research, may it be by direct coding or by doing something else, related. Doing this kind of stuff is far worse than filling grant proposals or doing other bureaucratic stuff. On the other side, I disagree with "most scientists are not great programmers". What is a "great programmer"? In my definition, it is someone who can write a program to solve a problem without too much hassle. And a lot of scientists I know satisfy this to terrific levels. Of course, they use no orthogonality, nor source code control, nor do extreme programming and usually don't write test cases. They just do what is asked as quickly as possible to keep on doing what is needed to do.

Maro 16y ago
I disagree strongly about your definition of "great programmer". Your definition is of a "somebody who can program".
- RBerenguel 16y ago
  I know a lot of people who "can program" and when confronted with a problem would spend a whole afternoon deciding if they can do it in the main function or need to code separate pieces and then a week to really do it. A great programmer just picks the problem and writes the code that solves it.
  - jemfinch 16y ago
    You've rather obviously never been exposed to real great programmers. Your "can program" definition is actually describing a really bad programmer, and your "great programmer" definition describes most kids who make it past one semester of high school computer science class.
    - RBerenguel 16y ago
      As a side note, your 'obviously' probably means that most mathematicians I know are therefore not 'great programmers'. Ok, maybe my definition is flawed. But when they have to do some program to solve anything, they'll find a way to do it.
    - RBerenguel 16y ago
      Probably not to 'real' great programmers. But I've given classes on numerical algorithms for the CS program in our university. None of these guys (and gals) except for one or two out of 70 or 80 will be better than "someone who can program".
- kenjackson 16y ago
  "Somebody who can program" have written probably 99% of used applications in the world today. From Bill Joy to Donald Knuth to David Cutler to Linus Torvalds to Guy Steele to Jamie Zawinsky to Guido van Rossum to John Carmack, and almost everyone in between. Great programmers somehow only appear to write books and give pristine examples of how you build infinitely extensible architectures.
  - Maro 16y ago
    John Carmack produces the most maintainable and readable codebases. I happen to have the Quake3 source code up on my github at, so you can see for yourself: https://github.com/mtrencseni/quake3 https://github.com/mtrencseni/quake3 Donald Knuth is the author of literate programming, which is a framework for writing human-readable programs: http://www-cs-faculty.stanford.edu/~uno/lp.html http://www-cs-faculty.stanford.edu/~uno/lp.html
    - kenjackson 16y ago
      Have you read how each of them actually write their code? Both write them precisely how the other poster noted. And I've read the Quake 3 code extensively. Great code, but certainly not pristine, and I'm sure if you handed it to code review to virtually anyone you know, they'd find a whole bunch of sylistic and architectural issues with it. Like take a look at the playerDie code. You're telling me you wouldn't have said, "Rewrite this?" if you a colleague handed this to you? And yes, Knuth is the author of literate programming, but that's not how the code started out. Read his letters on computer science. Tarjan wrote a similar thing, I think in his ACM Turing Award lecture. I picked those names because they are the best our industry has. But even with that, they all pretty much write code the way the previous post noted.
hnal943 16y ago
Is it possible that in the midst of their no-comments, no-test-cases spaghetti they introduce a few bugs that affect the results? If so, they need to release their code to be scrutinized along with their findings.
- RBerenguel 16y ago
  I don't see where I said it was spaghetti code. Sometimes a goto in C can save quite a few running cycles. Write a quick Runge-Kutta-Fehlberg integrator without it.
EdiX 16y ago
> Yes, taxpayers fund science, but commenting, beautifying and documenting code is not what a physicist/mathematician/climate researcher wants to spend his time. Usually you want to be doing research Hunting down all the right citations, putting results in good prose, formatting the paper to the standard of where you are trying to get it published, responding to peer reviews (as silly as they can get), doing peer reviews for others... all things that are not "doing research" (strictly speaking) and to some are not enjoyable. But they have to be done anyway, it's a quality standard that has to be met, everyone suffers for it and everyone benefits from it. It's just that the scientific community has let itself set really lousy standards for code sharing. It should be rectified.

Maro 16y ago

I have an MSc in CS and then did (still doing) a degree in Physics. I can report that you are right, most scientists are not great programmer. Most scientists just don't care about readability, maintainability, warnings, library dependencies, cross-platform issues, versioning, testing, etc. I think it'd be perfectly reasonable for society and the gov't to demand that scientists who produce results based on data and data processing make the data and the programs available to their peers and the general public, and also demand that it satisfies some reasonable software engineering expectations. (These will be set by their own peers anyway.) It is accepted by scientists that it is important to communicate research findings in the form of well-written human-readable papers. Over time the community must accept that including well-written machine-parsable data and code is just as important part of the scientific communication framework. Finally, a personal anecdote. I was contacted by a senior researcher to work on a cutting-edge numeric simulation project, and he sent me the source code, but made it clear that it is "top secret", even though it is gov't funded, as it is much better than its competitors, also gov't funded. He estimated that this advantage will last for several years, meaning several papers.

RBerenguel 16y ago
Deeply disturbing fact. The main researcher in our group works in cutting edge numeric simulations, and although he does not publish his code (mostly because "you can code it yourself! It is really easy, listen" and go on for 3 hours), if you were to ask, he would give it to you without any problems. But according to what you say, this means that he (still programs in Fortran) would need to start caring about versioning, cross-platform stuff (fortran, remember), readability (well, it's fortran, for me it is always unreadable) and maintainability. In addition to his usual duties of "doing research", giving lectures, answering students, advising PhD's and asking for grants. The problem is most likely to be solved when everyone in the field can code the algorithm in the paper. If the peer reviewer can write a program following the algorithm and applied on the data it validates the results, this ends up working well. I doubt (very profoundly) that giving the source will result in better peer reviews. The peer reviewer may have the code, but he will most likely compile, run and check that the numbers are the same.
- 16y ago

Daishiman 16y ago

You'll fix source code quality problems the day that research grants can contemplate the money to pay for scientific programming, meaning people who know their shit on modeling and running simulations on distributed systems. Do bear in mind that those people are not cheap, so it's doubtful that anyone would want to incur the expense.