17 ms·

A Code Glitch May Have Caused Errors in More Than a Hundred Published Studies

361 points by daddylonglegs 7y ago

609venezia 7y ago
This is another great example illustrating the need for reproducible research practices even in the hard sciences, in this case so that the papers in question could be easily checked after Luo's excellent finding. For an earlier discussion on the same topic, referencing other discussions: https://news.ycombinator.com/item?id=17819420 https://news.ycombinator.com/item?id=17819420 (Top comment: "This article harkens back to discussions of the "reproducibility crisis" in science, as discussed extensively here just recently (see link below). Where, in this case, not coughing up the code used in the simulations in a timely manner led to an apparently unnecessary multi-year dispute.")
ch33zer 7y ago
I feel like, where feasible, scientists should adopt Kubernetes or something so that the software they use is repeatable.
- deleted 7y ago
  [deleted]
- fwip 7y ago
  There's definitely a big push in that direction in the scientific community right now. More and more tools and pipelines are getting distributed as containerized workflows. Big projects have realized the need to make their code available and versioned just as they do their input data, side by side with hashes recorded all the way along and reproducibility made as simple as possible. Now we're starting to see it trickle down into less organized/large/disciplined projects as well.
  - ryukafalz 7y ago
    Containers as a technology are nice, but it's easy to fall back into the same traps that make software non-reproducible using containers as well. You can precisely specify all your dependencies, but it often takes a lot of effort to make that happen. I'm a fan of the approach Guix developers are taking for scientific computing, because it makes reproducible software simple enough for people to use without too many headaches: https://hpc.guix.info/blog/2019/10/towards-reproducible-jupyter-notebooks/ https://hpc.guix.info/blog/2019/10/towards-reproducible-jupy...
    - CamperBob2 7y ago
      Isn't the whole idea behind containers to eliminate external dependencies?
      - FakeComments 7y ago
        Sure, but how many people just... apt install foo ...without pinning a version of foo?
      - ryukafalz 7y ago
        In a sense. You're right that once a container is built, it has few external dependencies. But you need to get those dependencies from somewhere at build-time, and if you're not careful it's easy to do that in a way that makes it extremely difficult to rebuild that container in the future. To use a slightly more concrete example: let's say you're using a library in your container that has a severe bug. This bug results in incorrect computations, so you would like to upgrade to a fixed version. Now let's say that when you built that container initially, you installed packages in the Dockerfile by running e.g. "pip install <package>". The problem is that once this image is built, it's nontrivial to rebuild this image and ensure you're using the same dependencies you were the first time. In a sense, you've lost that information (though you can probably start to figure it out with close inspection of the image). Yes, there are usually ways around this with the language-specific package managers; Node has package-lock.json, Python has Pipfile.lock, etc. But it's not even close to being the default.
- javagram 7y ago
  Kubernetes and “repeatable” don’t exactly seem like they should go together, introducing the extremely high complexity of kubernetes seems likely to result in the opposite of repeatability. Just using a VM image or docker is simpler to use and also to understand.
peter303 7y ago
Most scientists are not software engineers so would not know about comprehensive testing. This was an eye opener to me when I moved from academia to a scientific software company. The situation is improving somewhat as many grad students open source their software in github. Then have include their makefiles, documentation, and some tests. In earlier days we we software engineers would roll our eyes when management suggested using 'free' university code to save money. Sometime bringing such software up to professional industry standards took more work than starting from scratch from published papers.
boulos 7y ago
It’s too bad that the code and correction seem to be locked down, but the linked Twitter discussion has an explanation [1]: the code expects Python’s glob to return a sorted list of files, but it doesn’t guarantee this (the main fix is to sort the result). I had thought reading the Vice article that this would be about case-sensitive vs insensitive filesystems. [1] https://mobile.twitter.com/bmarwell/status/1181821143835201536 https://mobile.twitter.com/bmarwell/status/11818211438352015...
- contravariant 7y ago
  To me this just raises more questions like "Why does the order of the files even matter?", and "If it does why wouldn't you make sure it was sorted, or better yet sort it in some reliable fashion?".
  - lupire 7y ago
    If you are writing a paper about the order in which a sequence of events happens, the order matters. It's the difference between cause and effect, the heart of science. Filename is often used as part of the data, as a timestamp. Obviously reliable sorting is important, which is why people are upset that python's glob() turned out to be unreliable
    - luckylion 7y ago
      > Obviously reliable sorting is important, which is why people are upset that python's glob() turned out to be unreliable A quick glance at the documentation didn't mention it, does glob even promise sorting the result? Relying on that to happen when it's not even promised seems negligent.
      - magicbuzz 7y ago
        Most developers look at the result from a function call rather than looking at the documentation.
      - perl4ever 7y ago
jacquesm 7y ago
A couple of years ago I looked over the alignment software used in a bunch of genetics studies and spotted a pretty trivial mistake that caused mis-alignment of the strings. This in turn led to wrong conclusions about the order in which mutations took place. This sort of thing is probably quite common: biologists are not computer programmers (though the combination is for sure occurring), they tend to treat the computer programs in roughly the same way as they would treat lab equipment, stuff goes in, other stuff come out. If it looks good and seems to work then it probably is good. But the complexity of the software is such that you need to check very carefully whether or not the software operates in the way you expect it to.
- jaddood 7y ago
  You could also choose to use formally verified software for sensitive research.
  - rhinoceraptor 7y ago
    Full-time developers don't (to a first approximation) use formal verification, much less research scientists.
    - stevenhuang 7y ago
      Other way around. Research scientists are the ones that may (or should) require additional rigor in their results. Formal verification would presumably help in that regard.
      - rhinoceraptor 7y ago
        I don't disagree, I'm just saying that we as full-time software developers have not made formal verification standard practice, or even approachable. I would bet the average software developer has never even heard of it. So to expect people who do not write software full-time to do so is a little crazy.
  - jacquesm 7y ago
jaddood 7y ago
Why not use formally-verified software?
- contravariant 7y ago
  Usually because the people capable of writing formally-verified software do not understand the specific subject area, and the people that understand the subject area are not generally capable of writing formally-verified software.
  - Nzen 7y ago
    For a perspective of a mathematician that came to evangelize theorem provers, I recommend Kevin Buzzard's MS 2019-09 presentation [0] about LEAN. He highlights cultural misunderstanding and apathy on both sides of the domain divide. He also references the idea that the people who might make the appropriate tools may not have stayed in academia. So, he's structured his courses around using LEAN with the indirect consequence that power users (undergrads) may choose to become open source committers. [0] https://www.youtube.com/watch?v=Dp-mQ3HxgDE https://www.youtube.com/watch?v=Dp-mQ3HxgDE One hour of presentation and then 15 min of q/a. My favorite is around 1:04:00 when someone asks a second time why he disprefers coq, and Buzzard complains that it can't represent some advanced quotient type that he'd have to work around. I'm reminded of [1] [1] https://prog21.dadgum.com/160.html https://prog21.dadgum.com/160.html Dangling by a trivial feature
    - abstractcontrol 7y ago
      Having been inspired by that video by Kevin Buzzard and finally finding something that would be worth formalizing, I am trying out Lean at the moment. I have about 2-3 months of Coq experience, so I can say that even without the quotient type, Lean is much better designed than Coq. I can't vouch for how it will do at scale since I've only started it out, but from what I can see, Lean fixes all the pain points that I had with Coq while going through Software Foundations. It has things like structural recursion (similar to Agda), dependent pattern matching (the biggest benefit of which would be proper variable naming), unicode, `calc` blocks, good IDE experience (it actually has autocomplete) with VS Code (I prefer it over Emacs and the inbuilt CoqIDE is broken on Windows), mutually recursive definitions and types, and various other things that are not at the top of my head. If I were to sum it up, the biggest issue with Coq is that it does not allow you to structure your code properly. This is kind of a big thing for me as a programmer.
throwaway57023 7y ago
Writing code in research is quite different from writing code in industry. For starters, in research, absolutely no one will read your code. Not your boss, not your peer reviewers, not your colleagues, not your tech-savvy users. No one. They just care about getting the right results and will complain if they don't, which most of the time steers you into producing correct code. Also, most programs usually have a maintainer team that's exactly one (1) person strong, and that person is usually an underpaid grad student or postdoc with a billion other tasks at hand and not much time left for that issue you opened three months ago. On the other hand, hey, you never have to fear the dreaded 'code review', so it's not all bad.
- lupire 7y ago
  Problem is, we don't want the results the author thinks are right; we want the results that are accurate.
  - chimi 7y ago
    The problem is, it's impossible to know if they are accurate without doing everything the computer was coded to do for you.
    - microcolonel 7y ago
      That's not necessarily true. We could start building correctness or equivalence proofs for the building blocks of research software, and maybe some day we could prove some meaningful equivalence between how the software is described, and how it actually works.
    - gmueckl 7y ago
      This is not entirely true. When I implement a simulation, I specifically look for specific properties of the simulated system that can be checked, for example time evolution of total energy or (angular) momentum. If you have a decent set of properties with non-linear relationships between them, it is actually quite hard to have a misbehaving simulation that still produces corrrect values for these properties. In fact, these checks have led me to the discovery of bugs that would otherwise have been impossible to find because the sim output was just plausible enough.
m3kw9 7y ago
Is kind of like if there is a CPU glitch
- shagie 7y ago
  From the article: > Luo’s results did not match up with the NMR values that Williams’ group had previously calculated, and according to Sun, when his students ran the code on their computers, they realized that different operating systems were producing different results. Sun then adjusted the code to fix the glitch, which had to do with how different operating systems sort files.
Gimpei 7y ago
There was a rumor when I was in econ grad school that there was a huge bug in Stata in the nineties affecting standard error calculations, rendering a large fraction of the studies from that time invalid.
- zhdc1 7y ago
  This should be easy to verify though. I would be more worried about some of the more popular Stata macros & R libraries. I've spent a good amount of time on several occasions trying to reconcile results from Stata and R, only to be stopped - I hope - by limited or technically dense documentation that I couldn't grok (...or the code was bad)
  - Gimpei 7y ago
    Should be, but coding standards among professors of econ are generally abysmal. It's a big gap in the educational system for social scientists.
- Dayshine 7y ago
  Stata doesn't even version its packages, so it's literally impossible to reproduce research as you can't retrieve the packages as they were then. The maintainer of a popular package could choose to change their api today and all code using it would break with no way to revert. Stata is insanity.
  - Gimpei 7y ago
    I agree. And it's expensive whereas R is free and a much much much better language.
Tistel 7y ago
Researchers should consider making their papers runnable. Jupyter notebooks being a great tool for it. Also, check out Knuth’s literate programming.
- zhdc1 7y ago
  This is standard practice in some fields. I'm helping an old colleague with one of their projects, and one of the requirements for publication is documented runnable code and all of the underlying data. I'm still surprised that this isn't expected in other research areas.
- gwd 7y ago
  I was going to say that would almost make things worse: right now these sorts of bugs are discovered by attempting to re-implement the described algorithm, and discovering that the described algorithm was wrong. If the code was available but wrong, most people would just run the code and find the same wrong answers as the original researchers. But actually I think that's wrong; far more people would run the code than currently try to re-implement the code; and it's likely that someone will notice something fishy about the code and report it.
  - erichocean 7y ago
    Well, we don’t have to wonder since this was done with PBRT. Result? An incredibly high quality codebase with many hundreds of subtle fixes over time coming from readers and researchers.
veezbo 7y ago
I've always felt like there should be a way to connect experienced software developers with research groups- something like this, for example, should have been never happened with code review. For example, one could imagine a program that connects developers with with research groups to help develop good code, or with conferences to test submitted code. Does anyone know if there already exists a program like that?
- tetron 7y ago
  Software Carpentry would be a good place to start: https://software-carpentry.org/ https://software-carpentry.org/
- zhdc1 7y ago
  A lot of the code used in research is hosted on Github and open for contributions. That would be an easy way to make an impact. Uni Zürich also has a group that tries to connect non-academics with researchers. The focus goes both ways, but it could be something to look into if you're good with code and want to assist in some way. Link: https://citizenscience.ch/en/ https://citizenscience.ch/en/
- therein 7y ago
  Something like this is sorely needed because we are abstracting away layers of implementation details and quirks from the users of computational software. They are in fact abstracted away so much that a doubt that the author might have about the correctness of the software is almost never brought up in a research paper. It is just assumed that whoever wrote the software did their due diligence. You often don't see their code, and even if you do, it is incomplete or impossible to run because while the publication is designed to allow the reader to replicate the study, the same doesn't necessarily apply to the computational component that crunched the numbers to reach the conclusion of the paper.
- pishpash 7y ago
  It's called an internship (for graduate students), but you're right that it should go the other way too but you'd run into a lot of money, IP, and institutional politics issues. Occasionally there are older industry people who work as staff scientists who get paid little and support the research effort in some capacity, usually IT. The biggest reasons why it doesn't happen as often are because in many cases, people who become professors have never done anything in industry, don't care unless funding is part of the conversation, or aren't motivated to have artifacts be reproducible. Their research group is a fiefdom over which they wield control, so unless they want to start a company on the side, they don't care about your programming skills and they don't like to give up any control.
WalterBright 7y ago
> “We all kind of assume that a computer program always spits out the correct answer.” I'm a bit surprised at this.
- _Wintermute 7y ago
  I'm not at all. Googling your problem and running your dataset through the first random R package on github you managed to install seems to be an accepted approach in the biomedical fields. If you're lucky they might even list which package it was in the paper's methods.
ken 7y ago
I've implemented algorithms I dug out of original research papers, which often included sample code. (That's why I first learned to read Fortran!) I've almost never gotten results that match the authors' exactly. Sometimes the bugs are obvious, and sometimes they're subtle. I'd estimate that 100% of sample implementations in published research papers have bugs. Researchers, even in computer science, are usually not skilled programmers. The product for them is the paper, not a program. It's the same category of problem as "enterprise software". Whenever the customer is not the user, the user gets screwed. With research, the customer is the journal.
- baby 7y ago
  Same here, not having done a PhD I’ve spent a lot of time just reproducing papers I liked and realized they were often very hand-wavy in the way they achieved their results, often exaggerating the impacts of the research.
- totalperspectiv 7y ago
  I totally agree on this. Having read papers and tried to implement, or even re-implement the same algorithm, even when their source code is available, it's hard to get results that match. I'm not a hard core TDD guy, but sometimes a few tests go a long way to making your code reproducible...or even just working correctly.
- softwaredoug 7y ago
  Academicia is suffering from a disease of uselessness where papers are written from a standpoint of getting credit. It’s not about other people understanding, reusing or learning how to use what they’ve found. Not to mention lacking in reproducability. Sadly this limits the usefulness of research for most. Yet for some reason we all still hold academia up on a pedestal
  - pishpash 7y ago
    The brokenness of academia starts with the exploitation of graduate students by educational institutions, which makes it a very lucrative enterprise to over-hire professors, each of whom is like an incubator for the school, which acts as an investor because they get a cut (upper five figures per year per student) no matter what. It's similar to college sports in a way. It's all driven by money. Most research is useless. Most professors are unneeded given the size of the problem space. Students see this and in the end, besides the very few true geniuses, it's not the most purely motivated that become professors, but the ones who most aggressively play the game in trumping up their results.
burntoutfire 7y ago
When I mentioned here some time ago, that maybe we shouldn't be trusting our climate science models so much, as they consist of millions of lines of poorly tested code, I was heavily criticized...
- moultano 7y ago
  Not trusting them is reasonable, but then where does that leave you? You don't know then whether they're underestimating or overestimating things. I'm strongly in favor of building a prior from the geological record and leaning on that as much or more than the models, but that doesn't lead to a very different conclusion.
  - burntoutfire 7y ago
    I guess that, at least compared to society in general, I'm a cognitive nihilist? I highly doubt that it is in our power to predict with useful accuracy something as complex as climate. If I'm not mistaken, we're still having problems modelling stuff a billion times less complex than climate (such as an air flow in turbines etc.)...
    - moultano 7y ago
      Generally, more uncertainty around climate change means you should be more supportive of climate change mitigation rather than less, because it means we can't rule out things that look like the P-T extinction.
      - burntoutfire 7y ago
        There's infinite number of events which we can't rule out. My approach is basically the opposite - if the evidence for something is super-sketchy, maybe we shouldn't spend trillions (or even quadrillions?) on preparing for it.
        moultano 7y ago
saboot 7y ago
Oh man, a sorting error of files due to filesystem differences.. A very similar issue happened to me while doing astronomy during undergrad. This was a software pipeline for a specific telescope and I followed the guide they provided. Essentially there were two directories, one for images, and another for error estimates. Files in both directories had the same filename. The software needed text documents of the images and errors. The guide listed to do a basic 'ls images/ > images_list.txt` `ls errors/ > errors_list.txt`. HOWEVER HYPERTHREADING WASN'T ACCOUNTED FOR. Between the images_list.txt and errors_list.txt about every 20 or 30 lines the file order would be swapped. Invalidating the analysis and producing poor images. It only took this undergrad two months to learn what was happening.
- liability 7y ago
  'Piles-of-files' is itself a bug in an era where sqlite exists and has bindings for virtually every language. Concurrent writing can almost always be reasonably avoided by keeping the processing parallel but serializing the writes. In cases where that earnestly isn't sufficient, a 'proper' RDBMS is a good option. Pile-of-files is 1960's era tech. We've learned much since then and our hardware is much more capable. Unless you're shooting for the retro-UNIX aesthetic for artistic reasons, it should be avoided.
  - kardos 7y ago
    I don't think this is one size fits all advice. Surely once your results are measured in TBs this will fall part.
BeetleB 7y ago
Copy pasting a comment I left almost a year ago: > A few weeks ago I had a conversation with a friend of mine who is wrapping up his PhD. He pointed out that not one of his colleagues is concerned whether anyone can reproduce their work. They use a home grown simulation suite which only they have access to, and is constantly being updated with the worst software practices you can think of. No one in their team believes that the tool will give the same results they did 4 years ago. The troubling part is, no one sees that as being a problem. They got their papers published, and so the SW did its job. My own experiences when I was in grad school (engineering, not CS): No one cares about code or code quality. In those days, no one used version control. The attitude all my fellow grad students had was: "I don't need to learn this stuff. Just need to publish this paper. When I become a PI, I'll simply make it my student/post doc's job"
nanis 7y ago
You have to dig really hard to find out what the specific problem was. It turns out this was the fix: def read_gaussian_outputfiles(): list_of_files = [] for file in glob.glob('*.out'): list_of_files.append(file) list_of_files.sort() return list_of_files That is, the original code never sorted the files and instead it trusted that a simple `glob` would give it files sorted in whatever is the desirable way. It is a reasonable mistake for a total noob to make (just like the hundreds of simulations which use the built-in `rand` function of whatever language they are using without even so much as mentioning this pertinent fact), this should be called a "programming error" instead of a "code glitch". Also, the error should be publicized up-front.
- mxcrossb 7y ago
  So many comments in this thread are saying that scientists are bad programmers, but even a great programmer can make mistakes like this. The solution isn’t related to better programmers, but instead for users to treat codes as instruments, and hence a verification and calibration step.
  - deleted 7y ago
    [deleted]
  - fauigerzigerk 7y ago
    I would say it depends on the thought process that caused the bug. There are a couple of possibilities I can think of: Did the author of the code simply forget to sort? That's equally likely to happen to anyone. Did they assume that the sort order of glob.glob() was reliable across different OSs and file systems? I don't think this would have happened to an experienced software developer. At least there would have been enough doubt to go read the first line of the docs. Did they not care whether the code worked anywhere outside their own personal setup? This is perhaps slightly more likely in an academic environment but I'm not completely sure about that. We'd have to know more about the specific circumstances. What I find more astonishing is that it took so long for this bug to be found. It's not exactly an edge case or a rounding error. It must have caused wildly incorrect results in many cases. So I think the more important differences may be on an organizational level rather than a question of individual competence.
sunstone 7y ago
Oh don't tell me it's a spreadsheet app.
- TallGuyShort 7y ago
  It says clearly it was a Python script
cameldrv 7y ago
On this specific bug, I really wish Python just returned a sorted directory listing on every platform. It goes back to Python's original philosophy of passing through OS level behavior rather than standardizing like Java.
mjcohen 7y ago
Retraction Watch has many, many, many examples of this type of thing. https://retractionwatch.com https://retractionwatch.com
vinni2 7y ago
This simple glitch in the original script calls into question the conclusions of a significant number of papers on a wide range of topics in a way that cannot be easily resolved from published information because the operating system is rarely mentioned,” In computer science it is a common practice to mention the experimental setup along with OS hardware configuration and third party libraries used for reproducibility purposes. Still many papers suffer from reproducibility. Now the conferences are demanding docker containers with all dependencies and configs. Which improves a bit better. But still lack of data is another reason why we can’t reproduce papers.
crocowhile 7y ago
I am a neuroscientist and a good portion of my laboratory's scientific output is code. I can offer a perspective. Every few months, a case like this one emerges. A published software contained a scientifically relevant bug and this may have had an influence on other studies that used that software. The reaction is always the same: blaming the naivete of the researchers because they are not professional coders. While I would certainly love to see more interaction between professional coders and scientists, I don't agree with the overall sentiment. I find it counterproductive and even naive for several reasons: 1) mistakes happen. Wikipedia has a long and certainly not complete list of software bugs that had major consequences ( https://en.wikipedia.org/wiki/List_of_software_bugs https://en.wikipedia.org/wiki/List_of_software_bugs ) and most of them come from professional coders in all kind of industries. There is no doubt whatsoever that code produced by scientists is in generall less streamed, less reviewed, and way uglier but is it on average more buggy where it matters? Unless we answer this question, the discussion is moot. 2) Scientists who produce software by themselves do this according to Open Source philosophy meaning that the code can be scrutinized by anyone. I'd rather use a possibly buggy but open-source software over one that I cannot even scrutinize. I think published results should only be accepted if they rely on open-source software: we would never accept a figure of a paper unless the protocol was 100% disclosed. This is one aspect where we can all immediately act. As a reviewer, I will never accept a paper that made use of new software unless the code is an integral part of the paper. I urge all my colleagues to do the same. Also, as authors, if you have to write software from scratch, take code readability into account exactly for this reason. Python is a good choice for this reason. R much less so. 3) we must encourage scientists to take on more coding, and not the opposite. If we keep telling them that they are not going to be good enough, we are not helping. A the moment the trade-off is very clear: either not-professionally coded software or no software whatsoever. Fields like crystallography have shown that the former option can still be extremely favorable.
nabla9 7y ago
The Reinhart-Rogoff error is probably the most famous Exel error in economics with huge consequences. Their 2010 a paper, Growth in a Time of Debt https://www.nber.org/papers/w15639 https://www.nber.org/papers/w15639 convincing dataset to show that when external debt reaches 60 percent of GDP, annual growth declines by about two percent. This only paper was used as top level politicians in the US and Europe to show that austerity is the only option. It started widely damaging pro-cyclical austerity policy during regression. The corrected result don't show any change in growth when debt to GDP ratio goes above 60 or 90 percent.
- dmix 7y ago
  The paper claimed a -0.1% decline at 90%, not -2% at 60% as you mentioned, but only in advanced countries, data in emerging countries was much worse. The Keynesian economists who countered showed their data when extended over a longer timeframe and without weighting was actually +2% at 90%, this too seems be limited to advanced economies. High debt ratios in emerging markets still seems to have a negative correlation AFAIK. Regardless, in practice, it looks like the study was completely ignored in the US anyway: https://fred.stlouisfed.org/fredgraph.png?g=FHS&nsh=1&width=600&height=400&trc=1 https://fred.stlouisfed.org/fredgraph.png?g=FHS&nsh=1&width=... Around 2010 the debt only dramatically increased above 100% instead of declining, and has continued to grow. In the EU, comparing debt to gdp in 2012 vs 2019 shows that there has been little decline there either: https://i0.wp.com/factsmaps.com/wp-content/uploads/2018/02/debt-to-gdp-ratio-european-countries.png?fit=2400%2C1800 https://i0.wp.com/factsmaps.com/wp-content/uploads/2018/02/d... https://www.statista.com/graphic/1/269684/national-debt-in-eu-countries-in-relation-to-gross-domestic-product-gdp.jpg https://www.statista.com/graphic/1/269684/national-debt-in-e...
  - nabla9 7y ago
    That's because the Democratic Party had majorities in both chambers. The paper was in the center of Rebublican policy led by Paul Ryan.
    - dmix 7y ago
dr_j_ 7y ago
Should have used MATLAB.
frankenbee 7y ago
Ok
TallGuyShort 7y ago
Most of the comments on here talking about repeatability, but I'm concerned about correctness too. It's $40 to have 48-hour access to the paper, and I'm not going to do that right now, but hear me out. The order in which the operating system listed the files made a difference. So just having it be the same everywhere doesn't necessarily fix the problem. What if it's the same everywhere, but still isn't the correct order for what the researcher intended? Everyone gets the same error, and that error isn't necessarily 0.
todd8 7y ago
I've been asked to consult on a number of programs used for scientific research or environmental modeling. In one case a science student was expected to work on a C++ simulation for his dissertation. The existing code base had been written by other grad students from previous years. It was clear that no one involved had any experience with C++ and it wasn't going to be possible for him to do a good job with the existing source code. Fortunately, in this case, the C++ based simulation was eventually dropped. Another time a team of professional civil engineers couldn't understand why their model was able to make predictions that perfectly matched their experimental data. It predicted a poorly understood physical phenomena (algae blooms) so perfectly that they asked me for help. In a couple of minutes I was able to explain to them how they had overfitted the model to the small sample of experimental data and that the model likely had no predictive value at all. I was asked once to debug a large program used to understand the dynamics of floods. The program contained some of the worst coding practices I've ever seen. For example, the same global variables were reused for completely different purposes in different parts of the program, sometimes X2DDRATE meant one thing and later on some paths it contained a completely different kind of measurement with entirely different units. This was done in an effort to "save memory" by not having too many variables. A different ecological model I helped repair needed to compute the integral of a function at one point and used the following method: pretend the function was a straight line so that the integral could be computed as the area of a triangle. The function was a simple, uncomplicated exponential, something that a high school math student could integrate exactly.

361 points by daddylonglegs 7y ago

This is another great example illustrating the need for reproducible research practices even in the hard sciences, in this case so that the papers in question could be easily checked after Luo's excellent finding. For an earlier discussion on the same topic, referencing other discussions: https://news.ycombinator.com/item?id=17819420 https://news.ycombinator.com/item?id=17819420 (Top comment: "This article harkens back to discussions of the "reproducibility crisis" in science, as discussed extensively here just recently (see link below). Where, in this case, not coughing up the code used in the simulations in a timely manner led to an apparently unnecessary multi-year dispute.")

ch33zer 7y ago

I feel like, where feasible, scientists should adopt Kubernetes or something so that the software they use is repeatable.

deleted 7y ago
[deleted]
fwip 7y ago
There's definitely a big push in that direction in the scientific community right now. More and more tools and pipelines are getting distributed as containerized workflows. Big projects have realized the need to make their code available and versioned just as they do their input data, side by side with hashes recorded all the way along and reproducibility made as simple as possible. Now we're starting to see it trickle down into less organized/large/disciplined projects as well.
- ryukafalz 7y ago
  Containers as a technology are nice, but it's easy to fall back into the same traps that make software non-reproducible using containers as well. You can precisely specify all your dependencies, but it often takes a lot of effort to make that happen. I'm a fan of the approach Guix developers are taking for scientific computing, because it makes reproducible software simple enough for people to use without too many headaches: https://hpc.guix.info/blog/2019/10/towards-reproducible-jupyter-notebooks/ https://hpc.guix.info/blog/2019/10/towards-reproducible-jupy...
  - CamperBob2 7y ago
    Isn't the whole idea behind containers to eliminate external dependencies?
    - FakeComments 7y ago
      Sure, but how many people just... apt install foo ...without pinning a version of foo?
    - ryukafalz 7y ago
      In a sense. You're right that once a container is built, it has few external dependencies. But you need to get those dependencies from somewhere at build-time, and if you're not careful it's easy to do that in a way that makes it extremely difficult to rebuild that container in the future. To use a slightly more concrete example: let's say you're using a library in your container that has a severe bug. This bug results in incorrect computations, so you would like to upgrade to a fixed version. Now let's say that when you built that container initially, you installed packages in the Dockerfile by running e.g. "pip install <package>". The problem is that once this image is built, it's nontrivial to rebuild this image and ensure you're using the same dependencies you were the first time. In a sense, you've lost that information (though you can probably start to figure it out with close inspection of the image). Yes, there are usually ways around this with the language-specific package managers; Node has package-lock.json, Python has Pipfile.lock, etc. But it's not even close to being the default.
javagram 7y ago
Kubernetes and “repeatable” don’t exactly seem like they should go together, introducing the extremely high complexity of kubernetes seems likely to result in the opposite of repeatability. Just using a VM image or docker is simpler to use and also to understand.

peter303 7y ago

Most scientists are not software engineers so would not know about comprehensive testing. This was an eye opener to me when I moved from academia to a scientific software company. The situation is improving somewhat as many grad students open source their software in github. Then have include their makefiles, documentation, and some tests. In earlier days we we software engineers would roll our eyes when management suggested using 'free' university code to save money. Sometime bringing such software up to professional industry standards took more work than starting from scratch from published papers.

boulos 7y ago

It’s too bad that the code and correction seem to be locked down, but the linked Twitter discussion has an explanation [1]: the code expects Python’s glob to return a sorted list of files, but it doesn’t guarantee this (the main fix is to sort the result). I had thought reading the Vice article that this would be about case-sensitive vs insensitive filesystems. [1] https://mobile.twitter.com/bmarwell/status/1181821143835201536 https://mobile.twitter.com/bmarwell/status/11818211438352015...

contravariant 7y ago
To me this just raises more questions like "Why does the order of the files even matter?", and "If it does why wouldn't you make sure it was sorted, or better yet sort it in some reliable fashion?".
- lupire 7y ago
  If you are writing a paper about the order in which a sequence of events happens, the order matters. It's the difference between cause and effect, the heart of science. Filename is often used as part of the data, as a timestamp. Obviously reliable sorting is important, which is why people are upset that python's glob() turned out to be unreliable
  - luckylion 7y ago
    > Obviously reliable sorting is important, which is why people are upset that python's glob() turned out to be unreliable A quick glance at the documentation didn't mention it, does glob even promise sorting the result? Relying on that to happen when it's not even promised seems negligent.
    - magicbuzz 7y ago
      Most developers look at the result from a function call rather than looking at the documentation.
    - perl4ever 7y ago

jacquesm 7y ago

A couple of years ago I looked over the alignment software used in a bunch of genetics studies and spotted a pretty trivial mistake that caused mis-alignment of the strings. This in turn led to wrong conclusions about the order in which mutations took place. This sort of thing is probably quite common: biologists are not computer programmers (though the combination is for sure occurring), they tend to treat the computer programs in roughly the same way as they would treat lab equipment, stuff goes in, other stuff come out. If it looks good and seems to work then it probably is good. But the complexity of the software is such that you need to check very carefully whether or not the software operates in the way you expect it to.

jaddood 7y ago
You could also choose to use formally verified software for sensitive research.
- rhinoceraptor 7y ago
  Full-time developers don't (to a first approximation) use formal verification, much less research scientists.
  - stevenhuang 7y ago
    Other way around. Research scientists are the ones that may (or should) require additional rigor in their results. Formal verification would presumably help in that regard.
    - rhinoceraptor 7y ago
      I don't disagree, I'm just saying that we as full-time software developers have not made formal verification standard practice, or even approachable. I would bet the average software developer has never even heard of it. So to expect people who do not write software full-time to do so is a little crazy.
- jacquesm 7y ago

jaddood 7y ago

Why not use formally-verified software?

contravariant 7y ago
Usually because the people capable of writing formally-verified software do not understand the specific subject area, and the people that understand the subject area are not generally capable of writing formally-verified software.
- Nzen 7y ago
  For a perspective of a mathematician that came to evangelize theorem provers, I recommend Kevin Buzzard's MS 2019-09 presentation [0] about LEAN. He highlights cultural misunderstanding and apathy on both sides of the domain divide. He also references the idea that the people who might make the appropriate tools may not have stayed in academia. So, he's structured his courses around using LEAN with the indirect consequence that power users (undergrads) may choose to become open source committers. [0] https://www.youtube.com/watch?v=Dp-mQ3HxgDE https://www.youtube.com/watch?v=Dp-mQ3HxgDE One hour of presentation and then 15 min of q/a. My favorite is around 1:04:00 when someone asks a second time why he disprefers coq, and Buzzard complains that it can't represent some advanced quotient type that he'd have to work around. I'm reminded of [1] [1] https://prog21.dadgum.com/160.html https://prog21.dadgum.com/160.html Dangling by a trivial feature
  - abstractcontrol 7y ago
    Having been inspired by that video by Kevin Buzzard and finally finding something that would be worth formalizing, I am trying out Lean at the moment. I have about 2-3 months of Coq experience, so I can say that even without the quotient type, Lean is much better designed than Coq. I can't vouch for how it will do at scale since I've only started it out, but from what I can see, Lean fixes all the pain points that I had with Coq while going through Software Foundations. It has things like structural recursion (similar to Agda), dependent pattern matching (the biggest benefit of which would be proper variable naming), unicode, `calc` blocks, good IDE experience (it actually has autocomplete) with VS Code (I prefer it over Emacs and the inbuilt CoqIDE is broken on Windows), mutually recursive definitions and types, and various other things that are not at the top of my head. If I were to sum it up, the biggest issue with Coq is that it does not allow you to structure your code properly. This is kind of a big thing for me as a programmer.

throwaway57023 7y ago

Writing code in research is quite different from writing code in industry. For starters, in research, absolutely no one will read your code. Not your boss, not your peer reviewers, not your colleagues, not your tech-savvy users. No one. They just care about getting the right results and will complain if they don't, which most of the time steers you into producing correct code. Also, most programs usually have a maintainer team that's exactly one (1) person strong, and that person is usually an underpaid grad student or postdoc with a billion other tasks at hand and not much time left for that issue you opened three months ago. On the other hand, hey, you never have to fear the dreaded 'code review', so it's not all bad.

lupire 7y ago
Problem is, we don't want the results the author thinks are right; we want the results that are accurate.
- chimi 7y ago
  The problem is, it's impossible to know if they are accurate without doing everything the computer was coded to do for you.
  - microcolonel 7y ago
    That's not necessarily true. We could start building correctness or equivalence proofs for the building blocks of research software, and maybe some day we could prove some meaningful equivalence between how the software is described, and how it actually works.
  - gmueckl 7y ago
    This is not entirely true. When I implement a simulation, I specifically look for specific properties of the simulated system that can be checked, for example time evolution of total energy or (angular) momentum. If you have a decent set of properties with non-linear relationships between them, it is actually quite hard to have a misbehaving simulation that still produces corrrect values for these properties. In fact, these checks have led me to the discovery of bugs that would otherwise have been impossible to find because the sim output was just plausible enough.

m3kw9 7y ago

Is kind of like if there is a CPU glitch

shagie 7y ago
From the article: > Luo’s results did not match up with the NMR values that Williams’ group had previously calculated, and according to Sun, when his students ran the code on their computers, they realized that different operating systems were producing different results. Sun then adjusted the code to fix the glitch, which had to do with how different operating systems sort files.

Gimpei 7y ago

There was a rumor when I was in econ grad school that there was a huge bug in Stata in the nineties affecting standard error calculations, rendering a large fraction of the studies from that time invalid.

zhdc1 7y ago
This should be easy to verify though. I would be more worried about some of the more popular Stata macros & R libraries. I've spent a good amount of time on several occasions trying to reconcile results from Stata and R, only to be stopped - I hope - by limited or technically dense documentation that I couldn't grok (...or the code was bad)
- Gimpei 7y ago
  Should be, but coding standards among professors of econ are generally abysmal. It's a big gap in the educational system for social scientists.
Dayshine 7y ago
Stata doesn't even version its packages, so it's literally impossible to reproduce research as you can't retrieve the packages as they were then. The maintainer of a popular package could choose to change their api today and all code using it would break with no way to revert. Stata is insanity.
- Gimpei 7y ago
  I agree. And it's expensive whereas R is free and a much much much better language.

Tistel 7y ago

Researchers should consider making their papers runnable. Jupyter notebooks being a great tool for it. Also, check out Knuth’s literate programming.

zhdc1 7y ago
This is standard practice in some fields. I'm helping an old colleague with one of their projects, and one of the requirements for publication is documented runnable code and all of the underlying data. I'm still surprised that this isn't expected in other research areas.
gwd 7y ago
I was going to say that would almost make things worse: right now these sorts of bugs are discovered by attempting to re-implement the described algorithm, and discovering that the described algorithm was wrong. If the code was available but wrong, most people would just run the code and find the same wrong answers as the original researchers. But actually I think that's wrong; far more people would run the code than currently try to re-implement the code; and it's likely that someone will notice something fishy about the code and report it.
- erichocean 7y ago
  Well, we don’t have to wonder since this was done with PBRT. Result? An incredibly high quality codebase with many hundreds of subtle fixes over time coming from readers and researchers.

veezbo 7y ago

I've always felt like there should be a way to connect experienced software developers with research groups- something like this, for example, should have been never happened with code review. For example, one could imagine a program that connects developers with with research groups to help develop good code, or with conferences to test submitted code. Does anyone know if there already exists a program like that?

tetron 7y ago
Software Carpentry would be a good place to start: https://software-carpentry.org/ https://software-carpentry.org/
zhdc1 7y ago
A lot of the code used in research is hosted on Github and open for contributions. That would be an easy way to make an impact. Uni Zürich also has a group that tries to connect non-academics with researchers. The focus goes both ways, but it could be something to look into if you're good with code and want to assist in some way. Link: https://citizenscience.ch/en/ https://citizenscience.ch/en/
therein 7y ago
Something like this is sorely needed because we are abstracting away layers of implementation details and quirks from the users of computational software. They are in fact abstracted away so much that a doubt that the author might have about the correctness of the software is almost never brought up in a research paper. It is just assumed that whoever wrote the software did their due diligence. You often don't see their code, and even if you do, it is incomplete or impossible to run because while the publication is designed to allow the reader to replicate the study, the same doesn't necessarily apply to the computational component that crunched the numbers to reach the conclusion of the paper.
pishpash 7y ago
It's called an internship (for graduate students), but you're right that it should go the other way too but you'd run into a lot of money, IP, and institutional politics issues. Occasionally there are older industry people who work as staff scientists who get paid little and support the research effort in some capacity, usually IT. The biggest reasons why it doesn't happen as often are because in many cases, people who become professors have never done anything in industry, don't care unless funding is part of the conversation, or aren't motivated to have artifacts be reproducible. Their research group is a fiefdom over which they wield control, so unless they want to start a company on the side, they don't care about your programming skills and they don't like to give up any control.

WalterBright 7y ago

> “We all kind of assume that a computer program always spits out the correct answer.” I'm a bit surprised at this.

_Wintermute 7y ago
I'm not at all. Googling your problem and running your dataset through the first random R package on github you managed to install seems to be an accepted approach in the biomedical fields. If you're lucky they might even list which package it was in the paper's methods.

ken 7y ago

I've implemented algorithms I dug out of original research papers, which often included sample code. (That's why I first learned to read Fortran!) I've almost never gotten results that match the authors' exactly. Sometimes the bugs are obvious, and sometimes they're subtle. I'd estimate that 100% of sample implementations in published research papers have bugs. Researchers, even in computer science, are usually not skilled programmers. The product for them is the paper, not a program. It's the same category of problem as "enterprise software". Whenever the customer is not the user, the user gets screwed. With research, the customer is the journal.

baby 7y ago
Same here, not having done a PhD I’ve spent a lot of time just reproducing papers I liked and realized they were often very hand-wavy in the way they achieved their results, often exaggerating the impacts of the research.
totalperspectiv 7y ago
I totally agree on this. Having read papers and tried to implement, or even re-implement the same algorithm, even when their source code is available, it's hard to get results that match. I'm not a hard core TDD guy, but sometimes a few tests go a long way to making your code reproducible...or even just working correctly.
softwaredoug 7y ago
Academicia is suffering from a disease of uselessness where papers are written from a standpoint of getting credit. It’s not about other people understanding, reusing or learning how to use what they’ve found. Not to mention lacking in reproducability. Sadly this limits the usefulness of research for most. Yet for some reason we all still hold academia up on a pedestal
- pishpash 7y ago
  The brokenness of academia starts with the exploitation of graduate students by educational institutions, which makes it a very lucrative enterprise to over-hire professors, each of whom is like an incubator for the school, which acts as an investor because they get a cut (upper five figures per year per student) no matter what. It's similar to college sports in a way. It's all driven by money. Most research is useless. Most professors are unneeded given the size of the problem space. Students see this and in the end, besides the very few true geniuses, it's not the most purely motivated that become professors, but the ones who most aggressively play the game in trumping up their results.

burntoutfire 7y ago

When I mentioned here some time ago, that maybe we shouldn't be trusting our climate science models so much, as they consist of millions of lines of poorly tested code, I was heavily criticized...

moultano 7y ago
Not trusting them is reasonable, but then where does that leave you? You don't know then whether they're underestimating or overestimating things. I'm strongly in favor of building a prior from the geological record and leaning on that as much or more than the models, but that doesn't lead to a very different conclusion.
- burntoutfire 7y ago
  I guess that, at least compared to society in general, I'm a cognitive nihilist? I highly doubt that it is in our power to predict with useful accuracy something as complex as climate. If I'm not mistaken, we're still having problems modelling stuff a billion times less complex than climate (such as an air flow in turbines etc.)...
  - moultano 7y ago
    Generally, more uncertainty around climate change means you should be more supportive of climate change mitigation rather than less, because it means we can't rule out things that look like the P-T extinction.
    - burntoutfire 7y ago
      There's infinite number of events which we can't rule out. My approach is basically the opposite - if the evidence for something is super-sketchy, maybe we shouldn't spend trillions (or even quadrillions?) on preparing for it.
      - moultano 7y ago

saboot 7y ago

Oh man, a sorting error of files due to filesystem differences.. A very similar issue happened to me while doing astronomy during undergrad. This was a software pipeline for a specific telescope and I followed the guide they provided. Essentially there were two directories, one for images, and another for error estimates. Files in both directories had the same filename. The software needed text documents of the images and errors. The guide listed to do a basic 'ls images/ > images_list.txt` `ls errors/ > errors_list.txt`. HOWEVER HYPERTHREADING WASN'T ACCOUNTED FOR. Between the images_list.txt and errors_list.txt about every 20 or 30 lines the file order would be swapped. Invalidating the analysis and producing poor images. It only took this undergrad two months to learn what was happening.

liability 7y ago
'Piles-of-files' is itself a bug in an era where sqlite exists and has bindings for virtually every language. Concurrent writing can almost always be reasonably avoided by keeping the processing parallel but serializing the writes. In cases where that earnestly isn't sufficient, a 'proper' RDBMS is a good option. Pile-of-files is 1960's era tech. We've learned much since then and our hardware is much more capable. Unless you're shooting for the retro-UNIX aesthetic for artistic reasons, it should be avoided.
- kardos 7y ago
  I don't think this is one size fits all advice. Surely once your results are measured in TBs this will fall part.

BeetleB 7y ago

Copy pasting a comment I left almost a year ago: > A few weeks ago I had a conversation with a friend of mine who is wrapping up his PhD. He pointed out that not one of his colleagues is concerned whether anyone can reproduce their work. They use a home grown simulation suite which only they have access to, and is constantly being updated with the worst software practices you can think of. No one in their team believes that the tool will give the same results they did 4 years ago. The troubling part is, no one sees that as being a problem. They got their papers published, and so the SW did its job. My own experiences when I was in grad school (engineering, not CS): No one cares about code or code quality. In those days, no one used version control. The attitude all my fellow grad students had was: "I don't need to learn this stuff. Just need to publish this paper. When I become a PI, I'll simply make it my student/post doc's job"

nanis 7y ago

You have to dig really hard to find out what the specific problem was. It turns out this was the fix: def read_gaussian_outputfiles(): list_of_files = [] for file in glob.glob('*.out'): list_of_files.append(file) list_of_files.sort() return list_of_files That is, the original code never sorted the files and instead it trusted that a simple `glob` would give it files sorted in whatever is the desirable way. It is a reasonable mistake for a total noob to make (just like the hundreds of simulations which use the built-in `rand` function of whatever language they are using without even so much as mentioning this pertinent fact), this should be called a "programming error" instead of a "code glitch". Also, the error should be publicized up-front.

mxcrossb 7y ago
So many comments in this thread are saying that scientists are bad programmers, but even a great programmer can make mistakes like this. The solution isn’t related to better programmers, but instead for users to treat codes as instruments, and hence a verification and calibration step.
- deleted 7y ago
  [deleted]
- fauigerzigerk 7y ago
  I would say it depends on the thought process that caused the bug. There are a couple of possibilities I can think of: Did the author of the code simply forget to sort? That's equally likely to happen to anyone. Did they assume that the sort order of glob.glob() was reliable across different OSs and file systems? I don't think this would have happened to an experienced software developer. At least there would have been enough doubt to go read the first line of the docs. Did they not care whether the code worked anywhere outside their own personal setup? This is perhaps slightly more likely in an academic environment but I'm not completely sure about that. We'd have to know more about the specific circumstances. What I find more astonishing is that it took so long for this bug to be found. It's not exactly an edge case or a rounding error. It must have caused wildly incorrect results in many cases. So I think the more important differences may be on an organizational level rather than a question of individual competence.

sunstone 7y ago

Oh don't tell me it's a spreadsheet app.

TallGuyShort 7y ago
It says clearly it was a Python script

cameldrv 7y ago

On this specific bug, I really wish Python just returned a sorted directory listing on every platform. It goes back to Python's original philosophy of passing through OS level behavior rather than standardizing like Java.

mjcohen 7y ago

Retraction Watch has many, many, many examples of this type of thing. https://retractionwatch.com https://retractionwatch.com

vinni2 7y ago

This simple glitch in the original script calls into question the conclusions of a significant number of papers on a wide range of topics in a way that cannot be easily resolved from published information because the operating system is rarely mentioned,” In computer science it is a common practice to mention the experimental setup along with OS hardware configuration and third party libraries used for reproducibility purposes. Still many papers suffer from reproducibility. Now the conferences are demanding docker containers with all dependencies and configs. Which improves a bit better. But still lack of data is another reason why we can’t reproduce papers.

crocowhile 7y ago

I am a neuroscientist and a good portion of my laboratory's scientific output is code. I can offer a perspective. Every few months, a case like this one emerges. A published software contained a scientifically relevant bug and this may have had an influence on other studies that used that software. The reaction is always the same: blaming the naivete of the researchers because they are not professional coders. While I would certainly love to see more interaction between professional coders and scientists, I don't agree with the overall sentiment. I find it counterproductive and even naive for several reasons: 1) mistakes happen. Wikipedia has a long and certainly not complete list of software bugs that had major consequences ( https://en.wikipedia.org/wiki/List_of_software_bugs https://en.wikipedia.org/wiki/List_of_software_bugs ) and most of them come from professional coders in all kind of industries. There is no doubt whatsoever that code produced by scientists is in generall less streamed, less reviewed, and way uglier but is it on average more buggy where it matters? Unless we answer this question, the discussion is moot. 2) Scientists who produce software by themselves do this according to Open Source philosophy meaning that the code can be scrutinized by anyone. I'd rather use a possibly buggy but open-source software over one that I cannot even scrutinize. I think published results should only be accepted if they rely on open-source software: we would never accept a figure of a paper unless the protocol was 100% disclosed. This is one aspect where we can all immediately act. As a reviewer, I will never accept a paper that made use of new software unless the code is an integral part of the paper. I urge all my colleagues to do the same. Also, as authors, if you have to write software from scratch, take code readability into account exactly for this reason. Python is a good choice for this reason. R much less so. 3) we must encourage scientists to take on more coding, and not the opposite. If we keep telling them that they are not going to be good enough, we are not helping. A the moment the trade-off is very clear: either not-professionally coded software or no software whatsoever. Fields like crystallography have shown that the former option can still be extremely favorable.

nabla9 7y ago

The Reinhart-Rogoff error is probably the most famous Exel error in economics with huge consequences. Their 2010 a paper, Growth in a Time of Debt https://www.nber.org/papers/w15639 https://www.nber.org/papers/w15639 convincing dataset to show that when external debt reaches 60 percent of GDP, annual growth declines by about two percent. This only paper was used as top level politicians in the US and Europe to show that austerity is the only option. It started widely damaging pro-cyclical austerity policy during regression. The corrected result don't show any change in growth when debt to GDP ratio goes above 60 or 90 percent.

dmix 7y ago
The paper claimed a -0.1% decline at 90%, not -2% at 60% as you mentioned, but only in advanced countries, data in emerging countries was much worse. The Keynesian economists who countered showed their data when extended over a longer timeframe and without weighting was actually +2% at 90%, this too seems be limited to advanced economies. High debt ratios in emerging markets still seems to have a negative correlation AFAIK. Regardless, in practice, it looks like the study was completely ignored in the US anyway: https://fred.stlouisfed.org/fredgraph.png?g=FHS&nsh=1&width=600&height=400&trc=1 https://fred.stlouisfed.org/fredgraph.png?g=FHS&nsh=1&width=... Around 2010 the debt only dramatically increased above 100% instead of declining, and has continued to grow. In the EU, comparing debt to gdp in 2012 vs 2019 shows that there has been little decline there either: https://i0.wp.com/factsmaps.com/wp-content/uploads/2018/02/debt-to-gdp-ratio-european-countries.png?fit=2400%2C1800 https://i0.wp.com/factsmaps.com/wp-content/uploads/2018/02/d... https://www.statista.com/graphic/1/269684/national-debt-in-eu-countries-in-relation-to-gross-domestic-product-gdp.jpg https://www.statista.com/graphic/1/269684/national-debt-in-e...
- nabla9 7y ago
  That's because the Democratic Party had majorities in both chambers. The paper was in the center of Rebublican policy led by Paul Ryan.
  - dmix 7y ago

dr_j_ 7y ago

Should have used MATLAB.

frankenbee 7y ago

TallGuyShort 7y ago

Most of the comments on here talking about repeatability, but I'm concerned about correctness too. It's $40 to have 48-hour access to the paper, and I'm not going to do that right now, but hear me out. The order in which the operating system listed the files made a difference. So just having it be the same everywhere doesn't necessarily fix the problem. What if it's the same everywhere, but still isn't the correct order for what the researcher intended? Everyone gets the same error, and that error isn't necessarily 0.

todd8 7y ago

I've been asked to consult on a number of programs used for scientific research or environmental modeling. In one case a science student was expected to work on a C++ simulation for his dissertation. The existing code base had been written by other grad students from previous years. It was clear that no one involved had any experience with C++ and it wasn't going to be possible for him to do a good job with the existing source code. Fortunately, in this case, the C++ based simulation was eventually dropped. Another time a team of professional civil engineers couldn't understand why their model was able to make predictions that perfectly matched their experimental data. It predicted a poorly understood physical phenomena (algae blooms) so perfectly that they asked me for help. In a couple of minutes I was able to explain to them how they had overfitted the model to the small sample of experimental data and that the model likely had no predictive value at all. I was asked once to debug a large program used to understand the dynamics of floods. The program contained some of the worst coding practices I've ever seen. For example, the same global variables were reused for completely different purposes in different parts of the program, sometimes X2DDRATE meant one thing and later on some paths it contained a completely different kind of measurement with entirely different units. This was done in an effort to "save memory" by not having too many variables. A different ecological model I helped repair needed to compute the integral of a function at one point and used the following method: pretend the function was a straight line so that the integral could be computed as the area of a triangle. The function was a simple, uncomplicated exponential, something that a high school math student could integrate exactly.