3 ms·

Sounds vaguely similar to the trickiest bug I ever had (and the first really hard bug I ever dealt with on my own). Mine was in AIX 3.2.5, circa 1995. We were b

by beat 11y ago

Sounds vaguely similar to the trickiest bug I ever had (and the first really hard bug I ever dealt with on my own). Mine was in AIX 3.2.5, circa 1995. We were buffering a latency-prone data stream between read and write processes using shared memory buffers. The original design used shmat(), which was limited to three buffers total on AIX. I rewrote the buffering using mmap() to create memory-mapped anonymous file buffers - the number was effectively unlimited and could be tuned via configuration. Around 100 buffers gave optimum performance, with a 300%+ throughput improvement in real-world conditions - a huge win on a huge problem! Then it blew up in production. Like hard crashes shortly after starting. Upon investigation, I found that entire pages of mapped memory were being overwritten with nulls, more or less randomly (4096 bytes at a time). Turned out the bug was in mmap() due to the order patches were applied on various servers. The dev/qa servers were patched at different times than the production servers. That, that was hairy. And for a junior programmer to have to explain this to the tech leads and IBM support - I don't even know how many times I heard variants on "What's wrong with your code, really? mmap() isn't buggy!" Ah, the 1990s. No Stack Overflow. No ssl (not even http). We did network programming by wrapping raw sockets in C and writing stream parsers in lex and yacc. Kids these days, they don't know hacking!

tjradcliffe 11y ago
I once #defined strlen to return a short. This was in the days of the 16-bit to 32-bit transition on Windows, and I was running a project to eliminate spurious warnings from our code. There were several hundred places where the return value of strlen was being assigned to a short, every one of which created a warning. I was a senior developer but it was my first commerical job--I'd been in academia previously. I ran the change by the the most senior technical people in the company, all the way up to the guy who'd written the original application as employee #1. He OK'd it because, as he said, "strlen returns as short on the Mac anyway, and since our code runs on Mac as well as Windows it's a limitation we have to respect anyways." A few years later the company stopped supporting the Mac. A few years after that (and well after I'd left the company) one user site started getting crashes. I heard later that it took four senior devs a week to track down the cause, and much head-scratching because strlen was documented to return an int. They eventually found my #define, along with a comment that this eliminated so-many hundred warnings in the build, and that the change had been approved by the most senior people (I wasn't totally naive). It turned out the problem at the specific site having the issue was users putting the entire contents of files into what amounted to tool-tips. It was totally unexpected user behaviour, but they'd found a place they could cache some useful data and we let them do it, so it should have worked. Today I'd write a script that auto-edited all the cases where the problem occured, and regression test the hell out of it, but yeah: the '90's were a different time!
- unwind 11y ago
  I'd just like to point out that at least in the draft C89 standard (http://port70.net/~nsz/c/c89/c89-draft.html#4.11.6.3 http://port70.net/~nsz/c/c89/c89-draft.html#4.11.6.3), strlen() had the same prototype as it has today: in other words it returns size_t. Not int.
huhtenberg 11y ago
> raw sockets Raw _TCP_ sockets probably. Streaming over raw RAW sockets - now, that'd be pretty damn hardcore indeed :)
- beat 11y ago
  Oh yeah. That's sort of insane coding. Still, it's weird to think that 20 years ago, just having TCP was kind of a luxury. I don't ever want to have to manage socket timeouts again.
zrm 11y ago
> circa 1995 How about this one from just last year: I'm writing an application that encrypts packets using datagram TLS with OpenSSL. Take a raw packet, send it through OpenSSL, it comes out the same on the other end and everything looks fine. Then I try load testing it and start getting mangled packets. Application data where there should be IP headers and vice versa. So get out the debugger and find out that the data going into SSL_write() on one end isn't the same as the data coming out of SSL_read() on the other end. No TLS error, just mangled data. So I install the OpenSSL symbols and discover that the data going into the cipher is exactly the same as the data that gets decrypted on the other side, hence no HMAC verification failure. But OpenSSL by default compresses data before encrypting it, and decompression output is not the data that was originally compressed. TLS requires records to be delivered reliably. Datagram TLS, by contrast, is like UDP. Packets can be lost. And if they are, you can't use stateful compression or the lost data creates a hole in the decompressor's stream and corrupts the output. But OpenSSL was doing exactly that. So disable compression and the problem disappears instantly. (After three days in a debugger.)
- jmhodges 11y ago
  FYI, using TLS compression makes you susceptible to the CRIME attack. I think I have a ticket with OpenSSL for them to turn that off by default, but I don't think they've done it, yet. Glad you got there accidentally!

by beat 11y ago

tjradcliffe 11y ago

I once #defined strlen to return a short. This was in the days of the 16-bit to 32-bit transition on Windows, and I was running a project to eliminate spurious warnings from our code. There were several hundred places where the return value of strlen was being assigned to a short, every one of which created a warning. I was a senior developer but it was my first commerical job--I'd been in academia previously. I ran the change by the the most senior technical people in the company, all the way up to the guy who'd written the original application as employee #1. He OK'd it because, as he said, "strlen returns as short on the Mac anyway, and since our code runs on Mac as well as Windows it's a limitation we have to respect anyways." A few years later the company stopped supporting the Mac. A few years after that (and well after I'd left the company) one user site started getting crashes. I heard later that it took four senior devs a week to track down the cause, and much head-scratching because strlen was documented to return an int. They eventually found my #define, along with a comment that this eliminated so-many hundred warnings in the build, and that the change had been approved by the most senior people (I wasn't totally naive). It turned out the problem at the specific site having the issue was users putting the entire contents of files into what amounted to tool-tips. It was totally unexpected user behaviour, but they'd found a place they could cache some useful data and we let them do it, so it should have worked. Today I'd write a script that auto-edited all the cases where the problem occured, and regression test the hell out of it, but yeah: the '90's were a different time!

unwind 11y ago
I'd just like to point out that at least in the draft C89 standard (http://port70.net/~nsz/c/c89/c89-draft.html#4.11.6.3 http://port70.net/~nsz/c/c89/c89-draft.html#4.11.6.3), strlen() had the same prototype as it has today: in other words it returns size_t. Not int.

huhtenberg 11y ago

> raw sockets Raw _TCP_ sockets probably. Streaming over raw RAW sockets - now, that'd be pretty damn hardcore indeed :)

beat 11y ago
Oh yeah. That's sort of insane coding. Still, it's weird to think that 20 years ago, just having TCP was kind of a luxury. I don't ever want to have to manage socket timeouts again.

zrm 11y ago

> circa 1995 How about this one from just last year: I'm writing an application that encrypts packets using datagram TLS with OpenSSL. Take a raw packet, send it through OpenSSL, it comes out the same on the other end and everything looks fine. Then I try load testing it and start getting mangled packets. Application data where there should be IP headers and vice versa. So get out the debugger and find out that the data going into SSL_write() on one end isn't the same as the data coming out of SSL_read() on the other end. No TLS error, just mangled data. So I install the OpenSSL symbols and discover that the data going into the cipher is exactly the same as the data that gets decrypted on the other side, hence no HMAC verification failure. But OpenSSL by default compresses data before encrypting it, and decompression output is not the data that was originally compressed. TLS requires records to be delivered reliably. Datagram TLS, by contrast, is like UDP. Packets can be lost. And if they are, you can't use stateful compression or the lost data creates a hole in the decompressor's stream and corrupts the output. But OpenSSL was doing exactly that. So disable compression and the problem disappears instantly. (After three days in a debugger.)

jmhodges 11y ago
FYI, using TLS compression makes you susceptible to the CRIME attack. I think I have a ticket with OpenSSL for them to turn that off by default, but I don't think they've done it, yet. Glad you got there accidentally!