6 ms·

I was part of the GitHub team that helped get the code ready for public release. We fixed a few bugs, reduced memory consumption, made it portable across operat

I was part of the GitHub team that helped get the code ready for public release. We fixed a few bugs, reduced memory consumption, made it portable across operating systems, etc., but the code you see is largely what was written by Neil Ferguson and his team. Given your concerns, I would like to mention two things: 1) the code was originally a single source file, so I'm not surprised the module boundaries are imperfect - the code is many years old and the module boundaries have existed for only a few weeks. 2) this was not written by professional software engineers, but epidemiologists, and they were working with limited budget, time, and programming experience.

mytailorisrich 6y ago
If the point of the exercise is to allow public scrutiny of models then IMHO the code should not be changed "for public release". The original code as used to produce simulation results that were in turn used for reports/papers/policy should be made public.
- lbeltrame 6y ago
  > If the point of the exercise is to allow public scrutiny of models then IMHO the code should not be changed "for public release". Some code I've used from scientific papers in the past made a git tag at the time of publication, and said "Use version XXXX and the provided sample data to get the results in the paper", or something of the sort. This could allow review and improvements post-publication.
thu2111 6y ago
Epidemiologists are professional software engineers, in the sense that they are paid full time to write code and summarise the output in papers. They might not be "professional" in the sense of the quality of their work but let's not kid ourselves that these people spend half their days in the lab or taking swabs from children in Beijing. Other types of researchers do that - these guys just run sims.
- mping 6y ago
  Academia is geared towards paper publishing, so why would they bother with code quality? Even if in the long run it would benefit them, its at best a nice to have. Most academia doesn't share code or data anyway. It's like saying that programmers are professional English writers, so comments and such should be written in perfect English. IMHO the only valid criticism is if the the simulation modelling is sound and implemented accurately.
  - thu2111 6y ago
    At every place I've worked correct spelling and grammar in comments is in fact expected of developers, as is high quality documentation. (yes I know I've been lucky)
  - abainbridge 6y ago
    Papers should not be accepted unless they are accompanied by high quality source code. > IMHO the only valid criticism is if the the simulation modelling is sound and implemented accurately. I think the best way to tell if it is sound and implemented accurately is to first require that it is implemented as simply (to understand) as it can be. Even then, it is hard to reason about code. But without simplicity, the problem is ten times worse. I'm tempted to go further and say that source code is formalized thought. I've written a number of simulators in the past, sometimes purely to help me understand a system more fully. There's something beautiful about describing the behaviour of a system in elegant code. When there's nothing left to remove I get much greater confidence that I've fully groked the system. I'd say this is the area where the "computers are like a bicycle for the mind" quote is the most true.
jarym 6y ago
Review by someone claiming to be a ex-Googler here: https://lockdownsceptics.org/code-review-of-fergusons-model/ https://lockdownsceptics.org/code-review-of-fergusons-model/ If the original code was a single file and you’ve ‘fixed bugs’ then what is on GitHub can in NO way be ‘largely what was written’ This code was used to lockdown an entire country - defending the code now based on who it was written by and under what constraints is disingenuous. It must have been known that there were these issues and they ought to have been fixed before using the output for something that had such huge societal impact.
mcepl 6y ago
Isn’t this a great example, why epidemiologists shouldn’t write in C++? Aren’t there some Python modules for this kind of stuff? Edit: Oh this is very old code, that could be an excuse.

mytailorisrich 6y ago

If the point of the exercise is to allow public scrutiny of models then IMHO the code should not be changed "for public release". The original code as used to produce simulation results that were in turn used for reports/papers/policy should be made public.

lbeltrame 6y ago
> If the point of the exercise is to allow public scrutiny of models then IMHO the code should not be changed "for public release". Some code I've used from scientific papers in the past made a git tag at the time of publication, and said "Use version XXXX and the provided sample data to get the results in the paper", or something of the sort. This could allow review and improvements post-publication.

thu2111 6y ago

Epidemiologists are professional software engineers, in the sense that they are paid full time to write code and summarise the output in papers. They might not be "professional" in the sense of the quality of their work but let's not kid ourselves that these people spend half their days in the lab or taking swabs from children in Beijing. Other types of researchers do that - these guys just run sims.

mping 6y ago
Academia is geared towards paper publishing, so why would they bother with code quality? Even if in the long run it would benefit them, its at best a nice to have. Most academia doesn't share code or data anyway. It's like saying that programmers are professional English writers, so comments and such should be written in perfect English. IMHO the only valid criticism is if the the simulation modelling is sound and implemented accurately.
- thu2111 6y ago
  At every place I've worked correct spelling and grammar in comments is in fact expected of developers, as is high quality documentation. (yes I know I've been lucky)
- abainbridge 6y ago
  Papers should not be accepted unless they are accompanied by high quality source code. > IMHO the only valid criticism is if the the simulation modelling is sound and implemented accurately. I think the best way to tell if it is sound and implemented accurately is to first require that it is implemented as simply (to understand) as it can be. Even then, it is hard to reason about code. But without simplicity, the problem is ten times worse. I'm tempted to go further and say that source code is formalized thought. I've written a number of simulators in the past, sometimes purely to help me understand a system more fully. There's something beautiful about describing the behaviour of a system in elegant code. When there's nothing left to remove I get much greater confidence that I've fully groked the system. I'd say this is the area where the "computers are like a bicycle for the mind" quote is the most true.

jarym 6y ago

Review by someone claiming to be a ex-Googler here: https://lockdownsceptics.org/code-review-of-fergusons-model/ https://lockdownsceptics.org/code-review-of-fergusons-model/ If the original code was a single file and you’ve ‘fixed bugs’ then what is on GitHub can in NO way be ‘largely what was written’ This code was used to lockdown an entire country - defending the code now based on who it was written by and under what constraints is disingenuous. It must have been known that there were these issues and they ought to have been fixed before using the output for something that had such huge societal impact.

mcepl 6y ago

Isn’t this a great example, why epidemiologists shouldn’t write in C++? Aren’t there some Python modules for this kind of stuff? Edit: Oh this is very old code, that could be an excuse.