4 ms·

Ask HN: What’s the worst bug you’ve written or encountered?

6 points by palmdeezy 6y ago

actionowl 6y ago
I've probably seen worst than this but this one comes to mind. A sysadmin had (I can only assume accidentally) removed the SUID bit from the crontab binary causing an application we used that invoked the binary to not work. To even allow this to work we had a custom SELinux module that we suspected had the issue. I ended up comparing systrace output between a working instance and the broken instance (in prod) to track it down.
wryoak 6y ago
I can't think of a particular bug, but I encountered one database so horribly misused, so poorly configured, for which no staff was given proper training, and with no data sanity checks in place, such that the university department using it had lost in excess of $700K revenue through one of its services over the course of 3 years.
- brittpart_ 6y ago
  yikes
AnimalMuppet 6y ago
Worst in what sense? Did the most damage, or hardest to find, or just most mind-blowing? We had a customer whose Unix environment didn't pass a stderr file handle to running programs. So if you tried to print something to stderr, it scribbled on whatever file happened to be open on handle number 2. This got reported to us as a bug. After looking at it, we told them to fix their insane runtime environment.
- kazinator 6y ago
  So, process starts with fd 2 being unused; opens something, and that something gets fd 2, which now looks like stderr?
  - AnimalMuppet 6y ago
    Yes. Not just looks like stderr, but by definition is stderr. (Except that it wasn't intended to be any such thing...)
- veddox 6y ago
  Not a bug as such, but always a lovely war story: rebuilding Unix after scragging /bin (https://pulsar.sternwarte.uni-erlangen.de/wilms/recreation/crash.html https://pulsar.sternwarte.uni-erlangen.de/wilms/recreation/c...)
croo 6y ago
Most mind boggling: a three character long comment addition in java code broke the build pipeline. It was 100% reproducable. Nothing else was in the commit. Reverting fixed it, re commiting broke the build again. It happened because the mvn on the build server ran out of memory exactly because of that 3 character addition.

6 points by palmdeezy 6y ago

I've probably seen worst than this but this one comes to mind. A sysadmin had (I can only assume accidentally) removed the SUID bit from the crontab binary causing an application we used that invoked the binary to not work. To even allow this to work we had a custom SELinux module that we suspected had the issue. I ended up comparing systrace output between a working instance and the broken instance (in prod) to track it down.

wryoak 6y ago

I can't think of a particular bug, but I encountered one database so horribly misused, so poorly configured, for which no staff was given proper training, and with no data sanity checks in place, such that the university department using it had lost in excess of $700K revenue through one of its services over the course of 3 years.

brittpart_ 6y ago
yikes

AnimalMuppet 6y ago

Worst in what sense? Did the most damage, or hardest to find, or just most mind-blowing? We had a customer whose Unix environment didn't pass a stderr file handle to running programs. So if you tried to print something to stderr, it scribbled on whatever file happened to be open on handle number 2. This got reported to us as a bug. After looking at it, we told them to fix their insane runtime environment.

kazinator 6y ago
So, process starts with fd 2 being unused; opens something, and that something gets fd 2, which now looks like stderr?
- AnimalMuppet 6y ago
  Yes. Not just looks like stderr, but by definition is stderr. (Except that it wasn't intended to be any such thing...)
veddox 6y ago
Not a bug as such, but always a lovely war story: rebuilding Unix after scragging /bin (https://pulsar.sternwarte.uni-erlangen.de/wilms/recreation/crash.html https://pulsar.sternwarte.uni-erlangen.de/wilms/recreation/c...)

croo 6y ago

Most mind boggling: a three character long comment addition in java code broke the build pipeline. It was 100% reproducable. Nothing else was in the commit. Reverting fixed it, re commiting broke the build again. It happened because the mvn on the build server ran out of memory exactly because of that 3 character addition.