3 ms·

As a friend of mine in academia is discovering...it can be quite hard to tell if it is broke.

rprospero 11y ago
I remember once trying to get an older professor on board with the concept of unit testing. He couldn't wrap his head around the idea. What kind of madmen were the Computer Science department hiring that programmers were wasting their time writing code so trivial that you knew what it would return before it ever ran?
- jmount 11y ago
  I found the way to sell unit testing to the academics is the following. The unit test isn't a proof that the code works on all inputs, it is a demonstration that there exists at least one set of inputs the code works for. As we know: a lot of code doesn't even meet the seemingly easy weaker second condition.
rspeer 11y ago
About two years ago I encountered a bizarre bug where, if I asked SciPy for the eigenvalues of a particular small matrix that I was using as a test case, it would consistently give a different result on my desktop computer than anywhere else. But when I tried to isolate the test case, it went away. It would only happen if I ran three other test cases in a particular order first. Or if I ran that one test case 12 times in a row, it would fail the twelfth time. I really wanted to find out what was going on. I looked through the code, from SciPy to ARPACK to the underlying ATLAS calls, at which point it became completely opaque to me. I still don't know whether it was the fault of ARPACK or ATLAS or what, but I just put the test cases in a different order, they consistently passed in that order just like they passed for everyone else, and a few system upgrades later the problem didn't happen anymore.
- GFK_of_xmaspast 11y ago
  A similar thing happened to me too, I would get some kind of internal error in a wrapped ARPACK and it would cause the entire python process to crash. It was reliable in that a given sequence of events would always trigger a crash, but not reliable in that we couldn't really alter those events to minimize things. We never bothered reporting it.
- hyperbovine 11y ago
  ATLAS compiles differently on different machines ("Automatically Tuned Linear Algebra Software") so this doesn't come as a complete surprise. Agree that that's a really annoying bug though. Do you happen to remember the matrix?

I remember once trying to get an older professor on board with the concept of unit testing. He couldn't wrap his head around the idea. What kind of madmen were the Computer Science department hiring that programmers were wasting their time writing code so trivial that you knew what it would return before it ever ran?

jmount 11y ago
I found the way to sell unit testing to the academics is the following. The unit test isn't a proof that the code works on all inputs, it is a demonstration that there exists at least one set of inputs the code works for. As we know: a lot of code doesn't even meet the seemingly easy weaker second condition.

rspeer 11y ago

About two years ago I encountered a bizarre bug where, if I asked SciPy for the eigenvalues of a particular small matrix that I was using as a test case, it would consistently give a different result on my desktop computer than anywhere else. But when I tried to isolate the test case, it went away. It would only happen if I ran three other test cases in a particular order first. Or if I ran that one test case 12 times in a row, it would fail the twelfth time. I really wanted to find out what was going on. I looked through the code, from SciPy to ARPACK to the underlying ATLAS calls, at which point it became completely opaque to me. I still don't know whether it was the fault of ARPACK or ATLAS or what, but I just put the test cases in a different order, they consistently passed in that order just like they passed for everyone else, and a few system upgrades later the problem didn't happen anymore.

GFK_of_xmaspast 11y ago
A similar thing happened to me too, I would get some kind of internal error in a wrapped ARPACK and it would cause the entire python process to crash. It was reliable in that a given sequence of events would always trigger a crash, but not reliable in that we couldn't really alter those events to minimize things. We never bothered reporting it.
hyperbovine 11y ago
ATLAS compiles differently on different machines ("Automatically Tuned Linear Algebra Software") so this doesn't come as a complete surprise. Agree that that's a really annoying bug though. Do you happen to remember the matrix?