11 ms·

The trickiest bug I've encountered

183 points by kenjackson 11y ago

digi_owl 11y ago
When a dev goes "why would anyone do that?!?" you know you are in for a bad day...
- dredmorbius 11y ago
  When it's the QA manager saying it, leave.
- slazaro 11y ago
  Don't ever report bugs or UI annoyances to the Adobe team then. It's so infuriating.
tonetheman 11y ago
I might agree with at least one thing the MS guy said. I really do not trust the OS so I would be zeroing out memory. That is me, I am an un-trusting soul.
- kleer001 11y ago
  > That is me, I am an un-trusting soul. Same here. I think it's a rare person that's been living and working with computers for more than 20 years that doesn't develop an innate distrust of them or at the very least a subtle set of supersticious about how they work, how they're supposed to work, and the best way to get your work done.
- ikeboy 11y ago
  >I really do not trust the OS so I would be zeroing out memory. If you don't trust the OS, don't use it. If you use it, you're giving it access to all your files, full stop.
  - randyrand 11y ago
    Sound simple but reality is much different. What if the intended users of your software use it?
    - ikeboy 11y ago
      Anyone using your software on a system they don't trust should have no expectation of the data being safe.
- banachtarski 11y ago
  How do you trust the OS to zero out the memory when you tell it to then? You're being facetious.
- ww520 11y ago
  Zeroing your own pages won't help. What if the OS reads in your super secret file into its cache pages which you have no control of. And those cache pages somehow got written to another process's page.
- raverbashing 11y ago
  Sure, when are you doing that? Remember, the bug "can cause arbitrary host OS pages to be written to the target file in some circumstances" something like do_encryption(origin, destination) save_to_disk(destination) You're clearing origin right before save_to_disk? Oh I'm so sorry, destination had only been allocated for you, writing to it caused some other pages to be evicted and other processes being scheduled and the bug to be triggered and your data is still there. You lost
nathanb 11y ago
In my day job, I have to write reams of complicated code that can slow down the system or make maintenance more annoying...just because a user can do something (even though doing that is unsupported). That's what it means to write business-class software. Nobody worth having as a customer is going to build their business on your platform if your attitude whenever something goes wrong is "you shouldn't have been doing that in the first place". (I am surprised to hear this story though. I actually found a bug in the NTFS buffer cache a few years ago which was introduced in (as I recall) Windows Server 2012. Maybe the Server organization are way more on the ball than the consumer OS organization, which is definitely possible. But they took it seriously and fixed it in a patch.)
- angersock 11y ago
  There's a balance, though. Oftentimes, you serve the business folks better by preventing them from doing something really silly, and instead letting them optimize their workflow not to include silliness. Unfortunately, it's tricky to tell ahead of time which features are bad ideas and which aren't.
  - tjradcliffe 11y ago
    It's pretty easy at first order: features that users want are good ideas. I agree at second order users want a lot of features that solve their problems in ways that are less clever than one might like, but in the vast majority of cases the right answer from development is, "Users know what they need better than I do, so if they ask for something, I'll do my best to implement it even if I don't totally understand why." That answer exposes the deeper answer, too: "My job as a developer is to understand user needs so that our software can help them fulfill them, so if I don't understand why someone needs a feature I should dig in further before implementing it so I don't implement the wrong thing or the right thing in the wrong way." Not always possible given shedule constraints, though. Admitedly, those rules don't seem to apply in this case, since if the OS allows you to do something that corrupts data, that's a problem no matter why the user wants to do it. If it can corrupt data, the OS shouldn't allow you to do it, end of story.
ksk 11y ago
When you reported the bug did you include a repro? I would appreciate it if someone could post it here. To be specific, I wanted to know the specific conditions the article talks about under which this causes an issue. Flush will cause writes (obviously) so issuing writes to a file which is pending a flush is an interesting scenario.
- vollmond 11y ago
  >> We reported the bug to Microsoft and included a short program that easily reproduced the issue. Now we just need a copy of it.
j_s 11y ago
Am I correct in believing that from a high-level view, this NTFS error resembles the ext4 bug currently on the front page? https://news.ycombinator.com/item?id=9576917 https://news.ycombinator.com/item?id=9576917 Edit: That must be why this post from 2013 (update the title!) discussing an issue fixed in 2011 hit the front page...
- jimrandomh 11y ago
  Similar, but the NTFS one was worse in that it was potentially a security vulnerability, since the corrupted data that it wrote inappropriately into the middle of files came from elsewhere in the filesystem, rather than zeroes. (Software is hard and often buggy, filesystems included. Check your backups! Even if your filesystem is perfect, I spilled water on my laptop yesterday, and you could, too.)
  - mikeash 11y ago
    If data isn't backed up, it doesn't exist. If it's not backed up offsite, it's not backed up. These are words to live or die by. I don't think it's likely that I'll spill water on your laptop, though.
    - hnyc 11y ago
      It seems so far that I'm immortal.
    - janfoeh 11y ago
      3) if you haven't successfully tried to restore it, it's not backed up. As the old saying goes, nobody cares about backups — people care only about restores.
      - mikeash 11y ago
        That is an excellent and often overlooked point!
      - quotedmycode 11y ago
vermooten 11y ago
How about fixing the white-text-on-a-pale background bug while you're at it?
Diamons 11y ago
I love the blog design
- snarkyturtle 11y ago
  I hate it, skinny text, low contrast, non-consistent background. It's everything wrong with modern web design. If, say, as you scrolled down the gradient went away, I'd be fine with it, but since it's fixed it's distracting.
  - joshuapants 11y ago
    I care less about the gradient than the text and contrast. It's a bit distracting but not to the point where it alone really hinders me.
  - Benjamin_Dobell 11y ago
    Agreed! I genuinely wanted to read the article but found myself squinting just to try read the text on that awful background. It was excruciating so I just gave up.
freebish 11y ago
In retrospect, I wonder if there was a better way to report that bug. Clearly the first person didn't get it, at all.
- perlgeek 11y ago
  Public disclosure as a security bug (information leak)?
beat 11y ago
Sounds vaguely similar to the trickiest bug I ever had (and the first really hard bug I ever dealt with on my own). Mine was in AIX 3.2.5, circa 1995. We were buffering a latency-prone data stream between read and write processes using shared memory buffers. The original design used shmat(), which was limited to three buffers total on AIX. I rewrote the buffering using mmap() to create memory-mapped anonymous file buffers - the number was effectively unlimited and could be tuned via configuration. Around 100 buffers gave optimum performance, with a 300%+ throughput improvement in real-world conditions - a huge win on a huge problem! Then it blew up in production. Like hard crashes shortly after starting. Upon investigation, I found that entire pages of mapped memory were being overwritten with nulls, more or less randomly (4096 bytes at a time). Turned out the bug was in mmap() due to the order patches were applied on various servers. The dev/qa servers were patched at different times than the production servers. That, that was hairy. And for a junior programmer to have to explain this to the tech leads and IBM support - I don't even know how many times I heard variants on "What's wrong with your code, really? mmap() isn't buggy!" Ah, the 1990s. No Stack Overflow. No ssl (not even http). We did network programming by wrapping raw sockets in C and writing stream parsers in lex and yacc. Kids these days, they don't know hacking!
- tjradcliffe 11y ago
  I once #defined strlen to return a short. This was in the days of the 16-bit to 32-bit transition on Windows, and I was running a project to eliminate spurious warnings from our code. There were several hundred places where the return value of strlen was being assigned to a short, every one of which created a warning. I was a senior developer but it was my first commerical job--I'd been in academia previously. I ran the change by the the most senior technical people in the company, all the way up to the guy who'd written the original application as employee #1. He OK'd it because, as he said, "strlen returns as short on the Mac anyway, and since our code runs on Mac as well as Windows it's a limitation we have to respect anyways." A few years later the company stopped supporting the Mac. A few years after that (and well after I'd left the company) one user site started getting crashes. I heard later that it took four senior devs a week to track down the cause, and much head-scratching because strlen was documented to return an int. They eventually found my #define, along with a comment that this eliminated so-many hundred warnings in the build, and that the change had been approved by the most senior people (I wasn't totally naive). It turned out the problem at the specific site having the issue was users putting the entire contents of files into what amounted to tool-tips. It was totally unexpected user behaviour, but they'd found a place they could cache some useful data and we let them do it, so it should have worked. Today I'd write a script that auto-edited all the cases where the problem occured, and regression test the hell out of it, but yeah: the '90's were a different time!
meshko 11y ago
The experience of dealing with Microsoft's support sounds exactly the same as what I had to deal with. In my situation there was no workaround possible short of rebooting the system (it was a kernel resource leak), so after spending 3 months iterating through different ways to reproduce the problem (including "we don't support Windows on VMWare, after asking me to send them a VM image to reproduce the problem), going through 3 levels of support, I got to the people who were able to get me a fix. Alas, it was a private patch which was only available upon request, which didn't help much as I was working for an ISV.
zamalek 11y ago
Getting Microsoft to fix bugs is hard. We had a bug in the .Net runtime. Took ages to reach even agreement on the memory dumps: probably because bugs in the .Net runtime are like unicorns, even I had a level of disbelief that .Net was to blame. Once you prove it, it's completely free. Ultimately memory dumps are the way to go when it comes to MSFT bug reports, if you can catch their bug red-handed and snap it to disk things go real smooth.
- sseveran 11y ago
  Yeah, I found a great one in .net. If you used HttpWebRequest to fetch a url that had an invalid GZIP header the app would hard crash. This is because the request and decompression was being handled on a threadpool thread. An exception would be thrown and no one would be around to catch. Fortunately this was in a web crawler so it took quite a while to build a repro case and an exact diagnoses. I believe it was fixed 4 or 5 years after I reported it in 2008.
- koyote 11y ago
  How did you report the bug? I always found it incredibly difficult to report anything to Microsoft and usually only get a 'we have reported it to engineers and it will be fixed some day' message.
ww520 11y ago
Ahhh, the fun with caching in the file system. Incidentally one of trickiest bugs I encountered also had to do with the Windows Cache Manager. It was the interaction between the Cache Manager and the Memory Manager in managing the rare transient state ModifiedNoWrite of the cache pages when dealing with reentry IO read requests. The cache page status became Modified when its content was filled in from disk, but it's marked as NoWrite to avoid being flushed out by the Memory Manager. The physical page backing the cache page can't be reused since it's dirty (Modified) but the Memory Manager can't flush it out (NoWrite). Slowly over time as more pages are read, the system would run out of physical pages. The Cache Manager was supposed to change the cache page status back to Standby after the read returned from lower layer. But with reentry IO read requests, it won't do it when upper layer IO request buffer passed straight down. The work around was to allocate a separate buffer to interact with the Cache Manager and copied the content back to the upper layer IO request buffer, incurring an extra copy. At the end I intimately knew about the Windows Cache Manager and Memory Manager more than I needed to know.
alexdowad 11y ago
To me, stuff like this is a powerful argument for using open-source software (or at least software which you can access the source for), whenever possible. When things go very wrong, at least you can dig into the source code and try to fix it.
- MaulingMonkey 11y ago
  It's certainly an argument for having source access. For OS / "platform" level bugs it's no panacea however - you may very well need to keep your workaround, either because you can't rely on end users upgrading to the latest version, or simply need a stopgap until you finish the patch, submit the patch upstream, address all the issues brought up in the review, have it integrated into the next stable release, have it trickle downstream into the stable releases of individual distros, etc etc etc...
dba7dba 11y ago
I think the trickiest bug that I heard of was USAF F22 jets losing all computer systems (navigation, communication etc etc) as it crossed date line while flying to Hawaii for the first deployment on the island. The flight of 4 jets were able to return to US mainland only because the accompanying tanker was able to guide them back.
chris_wot 11y ago
Yeah, well Microsoft never managed to find why Liberation fonts caused a BSOD: https://news.ycombinator.com/item?id=5468390 https://news.ycombinator.com/item?id=5468390
beagle3 11y ago
Ah, reminds me of a Windows bug that I chased in Win2000 Server. (For all I know it might still be there - I've stopped writing Windows code in 2007). When you wrote to a file (using WriteFile or fwrite()), it first extended the file length, and then committed the buffer. This is supposed to be atomic - that is, you should never be able to see the length already extended but the data not yet there. And it apparently was atomic if both reads and writes came from the local machine, or both came from the network - however, if the write was on the local machine but the read was from the network, locking was missing, and it WAS possible to read zeros instead of the real data (but only because of a race condition - reading the same file again later would give the expected answer) Tried to get Microsoft to at least confirm this bug, to no avail - there was no one interested in talking to a lone freelance developer back then.

183 points by kenjackson 11y ago

digi_owl 11y ago

When a dev goes "why would anyone do that?!?" you know you are in for a bad day...

dredmorbius 11y ago
When it's the QA manager saying it, leave.
slazaro 11y ago
Don't ever report bugs or UI annoyances to the Adobe team then. It's so infuriating.

tonetheman 11y ago

I might agree with at least one thing the MS guy said. I really do not trust the OS so I would be zeroing out memory. That is me, I am an un-trusting soul.

kleer001 11y ago
> That is me, I am an un-trusting soul. Same here. I think it's a rare person that's been living and working with computers for more than 20 years that doesn't develop an innate distrust of them or at the very least a subtle set of supersticious about how they work, how they're supposed to work, and the best way to get your work done.
ikeboy 11y ago
>I really do not trust the OS so I would be zeroing out memory. If you don't trust the OS, don't use it. If you use it, you're giving it access to all your files, full stop.
- randyrand 11y ago
  Sound simple but reality is much different. What if the intended users of your software use it?
  - ikeboy 11y ago
    Anyone using your software on a system they don't trust should have no expectation of the data being safe.
banachtarski 11y ago
How do you trust the OS to zero out the memory when you tell it to then? You're being facetious.
ww520 11y ago
Zeroing your own pages won't help. What if the OS reads in your super secret file into its cache pages which you have no control of. And those cache pages somehow got written to another process's page.
raverbashing 11y ago
Sure, when are you doing that? Remember, the bug "can cause arbitrary host OS pages to be written to the target file in some circumstances" something like do_encryption(origin, destination) save_to_disk(destination) You're clearing origin right before save_to_disk? Oh I'm so sorry, destination had only been allocated for you, writing to it caused some other pages to be evicted and other processes being scheduled and the bug to be triggered and your data is still there. You lost

nathanb 11y ago

In my day job, I have to write reams of complicated code that can slow down the system or make maintenance more annoying...just because a user can do something (even though doing that is unsupported). That's what it means to write business-class software. Nobody worth having as a customer is going to build their business on your platform if your attitude whenever something goes wrong is "you shouldn't have been doing that in the first place". (I am surprised to hear this story though. I actually found a bug in the NTFS buffer cache a few years ago which was introduced in (as I recall) Windows Server 2012. Maybe the Server organization are way more on the ball than the consumer OS organization, which is definitely possible. But they took it seriously and fixed it in a patch.)

angersock 11y ago
There's a balance, though. Oftentimes, you serve the business folks better by preventing them from doing something really silly, and instead letting them optimize their workflow not to include silliness. Unfortunately, it's tricky to tell ahead of time which features are bad ideas and which aren't.
- tjradcliffe 11y ago
  It's pretty easy at first order: features that users want are good ideas. I agree at second order users want a lot of features that solve their problems in ways that are less clever than one might like, but in the vast majority of cases the right answer from development is, "Users know what they need better than I do, so if they ask for something, I'll do my best to implement it even if I don't totally understand why." That answer exposes the deeper answer, too: "My job as a developer is to understand user needs so that our software can help them fulfill them, so if I don't understand why someone needs a feature I should dig in further before implementing it so I don't implement the wrong thing or the right thing in the wrong way." Not always possible given shedule constraints, though. Admitedly, those rules don't seem to apply in this case, since if the OS allows you to do something that corrupts data, that's a problem no matter why the user wants to do it. If it can corrupt data, the OS shouldn't allow you to do it, end of story.

ksk 11y ago

When you reported the bug did you include a repro? I would appreciate it if someone could post it here. To be specific, I wanted to know the specific conditions the article talks about under which this causes an issue. Flush will cause writes (obviously) so issuing writes to a file which is pending a flush is an interesting scenario.

vollmond 11y ago
>> We reported the bug to Microsoft and included a short program that easily reproduced the issue. Now we just need a copy of it.

j_s 11y ago

Am I correct in believing that from a high-level view, this NTFS error resembles the ext4 bug currently on the front page? https://news.ycombinator.com/item?id=9576917 https://news.ycombinator.com/item?id=9576917 Edit: That must be why this post from 2013 (update the title!) discussing an issue fixed in 2011 hit the front page...

jimrandomh 11y ago
Similar, but the NTFS one was worse in that it was potentially a security vulnerability, since the corrupted data that it wrote inappropriately into the middle of files came from elsewhere in the filesystem, rather than zeroes. (Software is hard and often buggy, filesystems included. Check your backups! Even if your filesystem is perfect, I spilled water on my laptop yesterday, and you could, too.)
- mikeash 11y ago
  If data isn't backed up, it doesn't exist. If it's not backed up offsite, it's not backed up. These are words to live or die by. I don't think it's likely that I'll spill water on your laptop, though.
  - hnyc 11y ago
    It seems so far that I'm immortal.
  - janfoeh 11y ago
    3) if you haven't successfully tried to restore it, it's not backed up. As the old saying goes, nobody cares about backups — people care only about restores.
    - mikeash 11y ago
      That is an excellent and often overlooked point!
    - quotedmycode 11y ago

vermooten 11y ago

How about fixing the white-text-on-a-pale background bug while you're at it?

Diamons 11y ago

I love the blog design

snarkyturtle 11y ago
I hate it, skinny text, low contrast, non-consistent background. It's everything wrong with modern web design. If, say, as you scrolled down the gradient went away, I'd be fine with it, but since it's fixed it's distracting.
- joshuapants 11y ago
  I care less about the gradient than the text and contrast. It's a bit distracting but not to the point where it alone really hinders me.
- Benjamin_Dobell 11y ago
  Agreed! I genuinely wanted to read the article but found myself squinting just to try read the text on that awful background. It was excruciating so I just gave up.

freebish 11y ago

In retrospect, I wonder if there was a better way to report that bug. Clearly the first person didn't get it, at all.

perlgeek 11y ago
Public disclosure as a security bug (information leak)?

beat 11y ago

Sounds vaguely similar to the trickiest bug I ever had (and the first really hard bug I ever dealt with on my own). Mine was in AIX 3.2.5, circa 1995. We were buffering a latency-prone data stream between read and write processes using shared memory buffers. The original design used shmat(), which was limited to three buffers total on AIX. I rewrote the buffering using mmap() to create memory-mapped anonymous file buffers - the number was effectively unlimited and could be tuned via configuration. Around 100 buffers gave optimum performance, with a 300%+ throughput improvement in real-world conditions - a huge win on a huge problem! Then it blew up in production. Like hard crashes shortly after starting. Upon investigation, I found that entire pages of mapped memory were being overwritten with nulls, more or less randomly (4096 bytes at a time). Turned out the bug was in mmap() due to the order patches were applied on various servers. The dev/qa servers were patched at different times than the production servers. That, that was hairy. And for a junior programmer to have to explain this to the tech leads and IBM support - I don't even know how many times I heard variants on "What's wrong with your code, really? mmap() isn't buggy!" Ah, the 1990s. No Stack Overflow. No ssl (not even http). We did network programming by wrapping raw sockets in C and writing stream parsers in lex and yacc. Kids these days, they don't know hacking!

tjradcliffe 11y ago
I once #defined strlen to return a short. This was in the days of the 16-bit to 32-bit transition on Windows, and I was running a project to eliminate spurious warnings from our code. There were several hundred places where the return value of strlen was being assigned to a short, every one of which created a warning. I was a senior developer but it was my first commerical job--I'd been in academia previously. I ran the change by the the most senior technical people in the company, all the way up to the guy who'd written the original application as employee #1. He OK'd it because, as he said, "strlen returns as short on the Mac anyway, and since our code runs on Mac as well as Windows it's a limitation we have to respect anyways." A few years later the company stopped supporting the Mac. A few years after that (and well after I'd left the company) one user site started getting crashes. I heard later that it took four senior devs a week to track down the cause, and much head-scratching because strlen was documented to return an int. They eventually found my #define, along with a comment that this eliminated so-many hundred warnings in the build, and that the change had been approved by the most senior people (I wasn't totally naive). It turned out the problem at the specific site having the issue was users putting the entire contents of files into what amounted to tool-tips. It was totally unexpected user behaviour, but they'd found a place they could cache some useful data and we let them do it, so it should have worked. Today I'd write a script that auto-edited all the cases where the problem occured, and regression test the hell out of it, but yeah: the '90's were a different time!

meshko 11y ago

The experience of dealing with Microsoft's support sounds exactly the same as what I had to deal with. In my situation there was no workaround possible short of rebooting the system (it was a kernel resource leak), so after spending 3 months iterating through different ways to reproduce the problem (including "we don't support Windows on VMWare, after asking me to send them a VM image to reproduce the problem), going through 3 levels of support, I got to the people who were able to get me a fix. Alas, it was a private patch which was only available upon request, which didn't help much as I was working for an ISV.

zamalek 11y ago

Getting Microsoft to fix bugs is hard. We had a bug in the .Net runtime. Took ages to reach even agreement on the memory dumps: probably because bugs in the .Net runtime are like unicorns, even I had a level of disbelief that .Net was to blame. Once you prove it, it's completely free. Ultimately memory dumps are the way to go when it comes to MSFT bug reports, if you can catch their bug red-handed and snap it to disk things go real smooth.

sseveran 11y ago
Yeah, I found a great one in .net. If you used HttpWebRequest to fetch a url that had an invalid GZIP header the app would hard crash. This is because the request and decompression was being handled on a threadpool thread. An exception would be thrown and no one would be around to catch. Fortunately this was in a web crawler so it took quite a while to build a repro case and an exact diagnoses. I believe it was fixed 4 or 5 years after I reported it in 2008.
koyote 11y ago
How did you report the bug? I always found it incredibly difficult to report anything to Microsoft and usually only get a 'we have reported it to engineers and it will be fixed some day' message.

ww520 11y ago

Ahhh, the fun with caching in the file system. Incidentally one of trickiest bugs I encountered also had to do with the Windows Cache Manager. It was the interaction between the Cache Manager and the Memory Manager in managing the rare transient state ModifiedNoWrite of the cache pages when dealing with reentry IO read requests. The cache page status became Modified when its content was filled in from disk, but it's marked as NoWrite to avoid being flushed out by the Memory Manager. The physical page backing the cache page can't be reused since it's dirty (Modified) but the Memory Manager can't flush it out (NoWrite). Slowly over time as more pages are read, the system would run out of physical pages. The Cache Manager was supposed to change the cache page status back to Standby after the read returned from lower layer. But with reentry IO read requests, it won't do it when upper layer IO request buffer passed straight down. The work around was to allocate a separate buffer to interact with the Cache Manager and copied the content back to the upper layer IO request buffer, incurring an extra copy. At the end I intimately knew about the Windows Cache Manager and Memory Manager more than I needed to know.

alexdowad 11y ago

To me, stuff like this is a powerful argument for using open-source software (or at least software which you can access the source for), whenever possible. When things go very wrong, at least you can dig into the source code and try to fix it.

MaulingMonkey 11y ago
It's certainly an argument for having source access. For OS / "platform" level bugs it's no panacea however - you may very well need to keep your workaround, either because you can't rely on end users upgrading to the latest version, or simply need a stopgap until you finish the patch, submit the patch upstream, address all the issues brought up in the review, have it integrated into the next stable release, have it trickle downstream into the stable releases of individual distros, etc etc etc...

dba7dba 11y ago

I think the trickiest bug that I heard of was USAF F22 jets losing all computer systems (navigation, communication etc etc) as it crossed date line while flying to Hawaii for the first deployment on the island. The flight of 4 jets were able to return to US mainland only because the accompanying tanker was able to guide them back.

chris_wot 11y ago

Yeah, well Microsoft never managed to find why Liberation fonts caused a BSOD: https://news.ycombinator.com/item?id=5468390 https://news.ycombinator.com/item?id=5468390

beagle3 11y ago

Ah, reminds me of a Windows bug that I chased in Win2000 Server. (For all I know it might still be there - I've stopped writing Windows code in 2007). When you wrote to a file (using WriteFile or fwrite()), it first extended the file length, and then committed the buffer. This is supposed to be atomic - that is, you should never be able to see the length already extended but the data not yet there. And it apparently was atomic if both reads and writes came from the local machine, or both came from the network - however, if the write was on the local machine but the read was from the network, locking was missing, and it WAS possible to read zeros instead of the real data (but only because of a race condition - reading the same file again later would give the expected answer) Tried to get Microsoft to at least confirm this bug, to no avail - there was no one interested in talking to a lone freelance developer back then.