> I do not know how bad it was, but it cannot be good the code was bad. I do know because I reviewed the code and its issue tracker extensively. I then wrote a

> I do not know how bad it was, but it cannot be good the code was bad. I do know because I reviewed the code and its issue tracker extensively. I then wrote an article summarizing its problems that went viral and melted the server hosting it. The Imperial College code wasn't merely "bad". It was unusable. It produced what were effectively random numbers distributed in a way that looked right to the people who wrote it (in epidemiology there is no actual validation of models, reinforcing the researcher's prior expectations is considered validation instead). The government accepted the resulting predictions at face value because they looked scientific. In the private sector this behavior would have resulted in severe liability. ICL's code was similar to the Toyota engine control code. A selection of bug types in that codebase: buffer overflows, heap corruption, race conditions, typos in PRNG constants, extreme sensitivity to what exact CPU it was run on, and so on. The results changed completely between versions for no scientific reason. The bugs mattered a lot: the variation in computed bed demand between runs was larger than the entire UK emergency hospital building program, just due to bugs. The program was originally a 15,000 line C file where most variables had single letter names and were in global scope. The results were predictable. In one case they'd attempted to hand write a Fischer-Yates shuffle (very easy, I used to use it as an interview question), but because their coding style was so poor they got confused about what variable 'k' contained and ended up replacing the contents of an array meant to contain people's ages with random junk from the heap. There were tests! But commented out, because you can't test a codebase that overrun with non-determinism bugs. The biggest problem was the attitudes it revealed within academia. Institutionalized arrogance and stupidity ruled the day. Bug reports were blown off by saying that they didn't matter because the "scientists" just ran their simulation lots of times and took the average. Professional programmers who pointed out bugs were told they had no right to comment because they weren't experts. ICL administration claimed all criticism was "ideological" or was irrelevant because "epidemiology isn't a subfield of computer science". Others were told that they shouldn't raise the alarm, because otherwise scientists would just stop showing their code for peer review. Someone claimed the results must have been correct because bugs in C programs always cause crashes and the model didn't crash. One academic even argued it was the fault of the software industry, because C doesn't come with "warning labels"! The worst was the culture of lying it exposed. The idea you can fix software bugs by just running the program several times is obviously wrong, but later it turned out that their simulation was so slow they didn't even bother doing that! It had been run once. They were simultaneously claiming determinism bugs didn't matter whilst also fixing them. They claimed the software had been peer reviewed when it never had been. The problems spanned institutions and weren't specific to ICL, as academics from other universities stood up to defend them. The coup de grace: ICL found an academic at Cambridge who issued a "code check" claiming in its abstract that in fact he'd run the model and got the same results, so there were no reproducibility problems. The BBC and others ran with it, saying the whole thing was just a storm in a teacup and actually there weren't any problems. In reality the code check report went on to admit that every single number the author had got was different to those in Report 9, including some differences of up to 25%! This was considered a "replication" by the author because the shape of the resulting graph was similar. That's ignoring all the deep scientific problems with the work. Even if the code had been correct it wouldn't have yielded predictions that came close to reality. Outside of computer science I don't believe science can be trusted when software gets involved. The ICL model had been hacked on for over a decade. Nobody had noticed or fixed the problems in that time, and when they were spotted by outsiders, academia and their friends in the media collectively closed ranks to protect Prof Ferguson. Academia has no procedures or conventions in place to ensure software is correct. To this day, nothing was ever done and no fault was ever admitted. There was a successful coverup and that was the end of it. Again: in the private sector this kind of behavior would yield liabilities in the tens of millions of dollars range, if not worse.