5 ms·

The first serious bug I ever dealt with professionally was a result of the hazards of mmap(). This was 1995, and I was working on AIX with a system that used a

by beat 7y ago

The first serious bug I ever dealt with professionally was a result of the hazards of mmap(). This was 1995, and I was working on AIX with a system that used a series of shared memory buffers for IPC. It was originally written with shmat(), and on AIX (at least in those days), shmat was limited to three shared segments, so we had a lot of performance-wrecking blocking going on while waiting for the buffers to be cleared. My first being original-idea development was rewriting with mmap so I could use an arbitrary (and programmable) number of buffers, tuning blocking against memory consumption, with logging to track performance for tuning. It was very cool. Worked great! Until it went to production. In production, it crashed every time it ran, shortly after starting up. Since we were doing seasonal production, backing out my change also backed out other necessary changes. It was very embarrassing and frustrating. Worse, I could not replicate the problem in testing! It only happened on the prod servers. And, as a wet behind the ears junior programmer, everyone assumed I'd just screwed up and was too dumb to understand how. So I wrote a test program, divorced from our regular code, to test mmap() itself. Turned out that it ran fine on dev/test servers, but on the prod servers, it would randomly overwrite 1k memory pages with nulls. Yeah. Once i convinced the senior engineers and my manager, I got to report an OS bug to IBM. Who were like "What's wrong with your code, really?" I wound up sending them my C test code and the compiled executable, along with results. It turned out the bug was caused by the order in which OS patches had been applied on the servers. One of the first rules in the marvelous book The Pragmatic Programmer is "Select() isn't broken". Yeah, but sometimes it is. And if a junior programmer came to me today reporting a bug in OS memory management, my first response would be "What's wrong with your code, really?"

ajross 7y ago
Yeah. It's sometimes hard to remember just How Bad software was even in living memory. These days, we all just assume that the basics all work and are surprised to see bugs, which we vote up to the top of HN. But it wasn't always like that, things just failed in crazy ways, at all levels of the stack. At the start of the dotcom boom, a server uptime measured in months was considered notable, and the idea of a client system lasting a week between some failure or voodoo reboot was laughable. Now... I dunno, when was the last time I bounced my phone? Genuinely can't remember.
- pcwalton 7y ago
  Yep, this is part of the reason why it drives me crazy when people say the '90s were the heyday of computing. Computers were awful back then.
  - nexuist 7y ago
    I think they still have a point. It was the heyday of computing precisely because they were awful; there was always something to work on or improve. Fixing bugs in operating systems offered clear improvements to the livelihoods of thousands or millions. It's no longer that easy now. Most critical things don't suck that hard anymore, so it's hard to find a way to feel special, or feel like it's a special time to be a part of. Even in startup land most work done is "build this website" or "build this app" - what the software does could be cool and easy enough to get passionate about, but the hope that "we're going to use this code to revolutionize the entire world for the better" that was omnipresent in '90s hacker culture has pretty much been murdered by the sins of Facebook and others in Big Tech. Now it's "what can I do to make sure the stuff I work on hurts the least amount of people as possible?"
    - knorker 7y ago
      On the flip side, while OpenSSH may be great and high quality now, and wasn't in the '90s, in the '90s people used telnet on the public Internet, and telnet worked.[1] IOW: It wasn't basic tech and great opportunity at the same time. Today you can work on immature things like OpenBSD's MPLS implementation and who knows, maybe in 20 years people will say that back in 2019 there were so many things that could be improved about that basic tech! (cue replies anti-labelswitching dweebs) [1] Not me, I used ssh since before OpenSSH.
      - icedchai 7y ago
        Telnet and rlogin were both very common. I remember installing ssh on my Slackware Linux box back in 1996. I had to build it from source... fun times!
  - 7y ago
mcguire 7y ago
AIX 3.2.5, around the same time, had a different mmap issue. If you mmap a file and write a byte into the mapped region, but beyond the end of the file, you get a signal; in handling the signal, you can extend the file. You also have 1. Made an error, as documented in the mmap man page, 2. Done something that works on JFS, 3. In fact, done what JFS does itself under the covers, and 4. Made a huge, huge mistake. If you did this on a NFS filesystem, it deadlocks the filesystem. If you are using an automounter, it turns into a tar-baby: IBM's automountd was single threaded. Any attempt to touch that filesystem deadlocks. Any attempt to mount a new filesystem blocks waiting on automountd. The only solution was to reboot the system. This is a bit of a problem if you have a database implementation class all trying this at the same time. I reported this to IBM and their response was, "it's documented as not working, whaddya want?" I considered releasing it on BUGTRAQ, but I didn't. The AIX 4 multithready rewrite fixed it, BTW.
- beat 7y ago
  Wow! Glad I missed that one!
- smsm42 7y ago
  NFS deadlocks are not unique to AIX. I've seen NFS in the state where any attempt to touch it locks up the process doing it (in the state where it can't even be killed - yes, the 'D' state) and if you need something from that process, good luck - reboot is pretty much the only way out. Nowdays, I think, NFS drivers in Linux became a bit saner but before that, if you get D's on NFS, you're out of luck.
loeg 7y ago
> One of the first rules in the marvelous book The Pragmatic Programmer is "Select() isn't broken". Yeah, but sometimes it is. poll() was broken on early versions of Mac OS X for quite a while[1] :-). And I thought I recalled select() being broken too, but might be mistaken. Looks like some reports of kqueue() being broken too.[2][3] [1]: https://daniel.haxx.se/blog/2016/10/11/poll-on-mac-10-12-is-broken/ https://daniel.haxx.se/blog/2016/10/11/poll-on-mac-10-12-is-... [2]: http://pod.tst.eu/http://cvs.schmorp.de/libev/ev.pod#OS_X_AND_DARWIN_BUGS http://pod.tst.eu/http://cvs.schmorp.de/libev/ev.pod#OS_X_AN... [3]: https://discussions.apple.com/thread/4783301?tstart=0 https://discussions.apple.com/thread/4783301?tstart=0
ak217 7y ago
I had a similarly maddening issue (circa 2014) once where small chunks of memory would randomly get overwritten on PV-virtualized AWS instances, but not on HVM-virtualized instances. We spent an enormous amount of time chasing this problem; the most senior engineer on the team wrote a test case that would have to write 100TB of data before reliably reproducing the issue. We never did figure out what caused it. Eventually we migrated all our instances to be HVM-only, and the problem went away.
user5994461 7y ago
>>> One of the first rules in the marvelous book The Pragmatic Programmer is "Select() isn't broken". Yeah, but sometimes it is. How come you give this example just now? This can't be a coincidence. select() just caused us a major production outage. FYI: select() is broken on pretty much all Linux kernels up to very recent ones. Doesn't work when there are more than 1024 opened file descriptors, not a lot for a server app. Side effects include crashing the application.
- ptr 7y ago
  Was your problem something other than the FD_SETSIZE limitation of select?
- bogomipz 7y ago
  >"FYI: select() is broken on pretty much all Linux kernels up to very recent ones." Can you elaborate or might you have some links? What was the cause and resolution?
  - zzzcpan 7y ago
    No, it's the least broken event notification mechanism on all systems. And it doesn't have a 1024 FDs limit on most systems either, as long as it used directly via syscall or through an event loop library that does that. But in general for all event notification mechanism on all systems, it's incorrect to assume that reported FDs are actually ready, because that could cause a busy waiting loop, and incorrect to assume that unreported FDs are not ready, because that could cause timeouts.
  - user5994461 7y ago
    See first paragraph in section BUGS. http://man7.org/linux/man-pages/man2/select.2.html http://man7.org/linux/man-pages/man2/select.2.html If you google for select 1024 file descriptors, you will find a ton of issues affecting a ton of libraries. The correction seems to be using poll(), but in our case was to remove the routine that wasn't needed anymore.
speedplane 7y ago
> One of the first rules in the marvelous book The Pragmatic Programmer is "Select() isn't broken". Yeah, but sometimes it is. And if a junior programmer came to me today reporting a bug in OS memory management, my first response would be "What's wrong with your code, really?" In my years of software development I had the fun but unfortunate experience of coming across not one, but two compiler bugs. The program would segfault on a seemingly innocuous line, but after looking a the assembly, it was clear the compiler was doing something screwy. These types of bugs are exceedingly rare, and they can be maddening to debug, but when you catch them you get a nice sense of satisfaction.
techslave 7y ago
this is an example of where experience matters. kids these days don’t get to experience deep bugs like this. and they are the worse for it.

by beat 7y ago

ajross 7y ago

Yeah. It's sometimes hard to remember just How Bad software was even in living memory. These days, we all just assume that the basics all work and are surprised to see bugs, which we vote up to the top of HN. But it wasn't always like that, things just failed in crazy ways, at all levels of the stack. At the start of the dotcom boom, a server uptime measured in months was considered notable, and the idea of a client system lasting a week between some failure or voodoo reboot was laughable. Now... I dunno, when was the last time I bounced my phone? Genuinely can't remember.

pcwalton 7y ago
Yep, this is part of the reason why it drives me crazy when people say the '90s were the heyday of computing. Computers were awful back then.
- nexuist 7y ago
  I think they still have a point. It was the heyday of computing precisely because they were awful; there was always something to work on or improve. Fixing bugs in operating systems offered clear improvements to the livelihoods of thousands or millions. It's no longer that easy now. Most critical things don't suck that hard anymore, so it's hard to find a way to feel special, or feel like it's a special time to be a part of. Even in startup land most work done is "build this website" or "build this app" - what the software does could be cool and easy enough to get passionate about, but the hope that "we're going to use this code to revolutionize the entire world for the better" that was omnipresent in '90s hacker culture has pretty much been murdered by the sins of Facebook and others in Big Tech. Now it's "what can I do to make sure the stuff I work on hurts the least amount of people as possible?"
  - knorker 7y ago
    On the flip side, while OpenSSH may be great and high quality now, and wasn't in the '90s, in the '90s people used telnet on the public Internet, and telnet worked.[1] IOW: It wasn't basic tech and great opportunity at the same time. Today you can work on immature things like OpenBSD's MPLS implementation and who knows, maybe in 20 years people will say that back in 2019 there were so many things that could be improved about that basic tech! (cue replies anti-labelswitching dweebs) [1] Not me, I used ssh since before OpenSSH.
    - icedchai 7y ago
      Telnet and rlogin were both very common. I remember installing ssh on my Slackware Linux box back in 1996. I had to build it from source... fun times!
- 7y ago

mcguire 7y ago

AIX 3.2.5, around the same time, had a different mmap issue. If you mmap a file and write a byte into the mapped region, but beyond the end of the file, you get a signal; in handling the signal, you can extend the file. You also have 1. Made an error, as documented in the mmap man page, 2. Done something that works on JFS, 3. In fact, done what JFS does itself under the covers, and 4. Made a huge, huge mistake. If you did this on a NFS filesystem, it deadlocks the filesystem. If you are using an automounter, it turns into a tar-baby: IBM's automountd was single threaded. Any attempt to touch that filesystem deadlocks. Any attempt to mount a new filesystem blocks waiting on automountd. The only solution was to reboot the system. This is a bit of a problem if you have a database implementation class all trying this at the same time. I reported this to IBM and their response was, "it's documented as not working, whaddya want?" I considered releasing it on BUGTRAQ, but I didn't. The AIX 4 multithready rewrite fixed it, BTW.

beat 7y ago
Wow! Glad I missed that one!
smsm42 7y ago
NFS deadlocks are not unique to AIX. I've seen NFS in the state where any attempt to touch it locks up the process doing it (in the state where it can't even be killed - yes, the 'D' state) and if you need something from that process, good luck - reboot is pretty much the only way out. Nowdays, I think, NFS drivers in Linux became a bit saner but before that, if you get D's on NFS, you're out of luck.

loeg 7y ago

> One of the first rules in the marvelous book The Pragmatic Programmer is "Select() isn't broken". Yeah, but sometimes it is. poll() was broken on early versions of Mac OS X for quite a while[1] :-). And I thought I recalled select() being broken too, but might be mistaken. Looks like some reports of kqueue() being broken too.[2][3] [1]: https://daniel.haxx.se/blog/2016/10/11/poll-on-mac-10-12-is-broken/ https://daniel.haxx.se/blog/2016/10/11/poll-on-mac-10-12-is-... [2]: http://pod.tst.eu/http://cvs.schmorp.de/libev/ev.pod#OS_X_AND_DARWIN_BUGS http://pod.tst.eu/http://cvs.schmorp.de/libev/ev.pod#OS_X_AN... [3]: https://discussions.apple.com/thread/4783301?tstart=0 https://discussions.apple.com/thread/4783301?tstart=0

ak217 7y ago

I had a similarly maddening issue (circa 2014) once where small chunks of memory would randomly get overwritten on PV-virtualized AWS instances, but not on HVM-virtualized instances. We spent an enormous amount of time chasing this problem; the most senior engineer on the team wrote a test case that would have to write 100TB of data before reliably reproducing the issue. We never did figure out what caused it. Eventually we migrated all our instances to be HVM-only, and the problem went away.

user5994461 7y ago

>>> One of the first rules in the marvelous book The Pragmatic Programmer is "Select() isn't broken". Yeah, but sometimes it is. How come you give this example just now? This can't be a coincidence. select() just caused us a major production outage. FYI: select() is broken on pretty much all Linux kernels up to very recent ones. Doesn't work when there are more than 1024 opened file descriptors, not a lot for a server app. Side effects include crashing the application.

ptr 7y ago
Was your problem something other than the FD_SETSIZE limitation of select?
bogomipz 7y ago
>"FYI: select() is broken on pretty much all Linux kernels up to very recent ones." Can you elaborate or might you have some links? What was the cause and resolution?
- zzzcpan 7y ago
  No, it's the least broken event notification mechanism on all systems. And it doesn't have a 1024 FDs limit on most systems either, as long as it used directly via syscall or through an event loop library that does that. But in general for all event notification mechanism on all systems, it's incorrect to assume that reported FDs are actually ready, because that could cause a busy waiting loop, and incorrect to assume that unreported FDs are not ready, because that could cause timeouts.
- user5994461 7y ago
  See first paragraph in section BUGS. http://man7.org/linux/man-pages/man2/select.2.html http://man7.org/linux/man-pages/man2/select.2.html If you google for select 1024 file descriptors, you will find a ton of issues affecting a ton of libraries. The correction seems to be using poll(), but in our case was to remove the routine that wasn't needed anymore.

speedplane 7y ago

> One of the first rules in the marvelous book The Pragmatic Programmer is "Select() isn't broken". Yeah, but sometimes it is. And if a junior programmer came to me today reporting a bug in OS memory management, my first response would be "What's wrong with your code, really?" In my years of software development I had the fun but unfortunate experience of coming across not one, but two compiler bugs. The program would segfault on a seemingly innocuous line, but after looking a the assembly, it was clear the compiler was doing something screwy. These types of bugs are exceedingly rare, and they can be maddening to debug, but when you catch them you get a nice sense of satisfaction.

techslave 7y ago

this is an example of where experience matters. kids these days don’t get to experience deep bugs like this. and they are the worse for it.