5 ms·

The most expensive bug that I can talk about publicly that I've encountered: I used to work on software for a very expensive (started at 70k) DNA/protein analy

by earl 17y ago

The most expensive bug that I can talk about publicly that I've encountered: I used to work on software for a very expensive (started at 70k) DNA/protein analysis hardware solution. This was in the late 90s, and our GUI was a couple million lines of MFC code. I was responsible for an analysis package written in cross platform (linux and windows) C++, but the main UI and all the complex interface was MFC. My then boss was a unix idiot who hated Windows. Which is fine and all, but a Windows product was paying our salaries. So one day he decided to rewrite my (well tested, though not unit tested) file handling code written using the win32 API and port it to posix. Not to accomplish anything different, mind you, and not to add it to the cross platform bits since it was useless in isolation and, oh yeah, we had multiple millions of lines of MFC so nothing was ever getting ported. In any case, during this "upgrade", he found a function called DirTouch, which was intended to make sure certain directories existed during the normal course of saving data. Well, this gentleman subtly changed the semantics from "create the directory if it doesn't exist" to "create the directory if it doesn't exist, but if it does already, then silently delete whatever is there". This wasn't the whole bug, but this change of semantics to a destructive function was the root cause. This change got shipped. One of our customers killed literally more than $1.3MM worth of data, since each machine run might cost $100K given lab time, reagents, prep, etc.

vsync 17y ago
What happened to him? And did he try to blame you? In his mind I'm sure all he did was port your code, which must after all have been buggy.
- earl 17y ago
  Well... it was a shitshow. We were just told by a livid CEO that a major university (the bio world is surprisingly small and it's pretty easy to get a terrible rep) had a major data loss, etc. We were on the phone with them while simultaneously overnighting the computer to data recovery trying to figure out what happened. It took two frantic days before I figured out what the problem is, patched it, and started the emergency release process for all the other customers. But yeah, I initially took responsibility for the mess, since it was most likely my code that did it. After figuring out precisely what happened, and having to demonstrate to my boss that the old code did not delete stuff out of directories, that responsibility got walked back a bit. Still... I got "laid off" within 5 months of that. Basically, there was blame to go around: the university should have had a comprehensive backup solution in place, particularly given the expense involved in creating the data; I should have been more careful to not needlessly call this directory touch operation while creating cached bits in the data files (the "data file" that the user thought of was actually a directory because we were running into fat16 and fat32 file size limits), but at the end of the day, my boss took code that used to work and turned it into code that not only didn't work but was broken in the worst way possible for no other reason than to fuck about with posix idiocy. In a giant win32/mfc app.

by earl 17y ago

vsync 17y ago

What happened to him? And did he try to blame you? In his mind I'm sure all he did was port your code, which must after all have been buggy.

earl 17y ago
Well... it was a shitshow. We were just told by a livid CEO that a major university (the bio world is surprisingly small and it's pretty easy to get a terrible rep) had a major data loss, etc. We were on the phone with them while simultaneously overnighting the computer to data recovery trying to figure out what happened. It took two frantic days before I figured out what the problem is, patched it, and started the emergency release process for all the other customers. But yeah, I initially took responsibility for the mess, since it was most likely my code that did it. After figuring out precisely what happened, and having to demonstrate to my boss that the old code did not delete stuff out of directories, that responsibility got walked back a bit. Still... I got "laid off" within 5 months of that. Basically, there was blame to go around: the university should have had a comprehensive backup solution in place, particularly given the expense involved in creating the data; I should have been more careful to not needlessly call this directory touch operation while creating cached bits in the data files (the "data file" that the user thought of was actually a directory because we were running into fat16 and fat32 file size limits), but at the end of the day, my boss took code that used to work and turned it into code that not only didn't work but was broken in the worst way possible for no other reason than to fuck about with posix idiocy. In a giant win32/mfc app.