10 ms·

OMG, as I was reading this I thought, "man, this reminds me of a bug I ran into with std::string back in 2000", A few sentences later, and this is also about st

OMG, as I was reading this I thought, "man, this reminds me of a bug I ran into with std::string back in 2000", A few sentences later, and this is also about std::string and the STL. Mine was different though, after tracking down a memory leak that was happening with the creation of just new empty string, I discovered in the stdlib that there was a shared pointer to the empty string with a reference count of how many locations were using it (ironic that this was intended to save allocations). It turned out this was on Intel and we had what was rare at the time, a multi-processor system. It turned out that the std::string empty string reference count was just doing a vanilla ++, no locking, nothing, variable not marked volatile, nothing. A few emails with a guy in Australia, a little inline assembly to call a new atomic increment on the counter, and the bug was fixed. That took two weeks to track down, mostly because it didn't even cross my mind that it wasn't in my code. From that point on, I realized you can't trust libraries blindly, even one of the most used and broadly adopted ones out there.

bostik 10y ago
> you can't trust libraries blindly, even one of the most used and broadly adopted ones There is a corollary to development and debugging. When things break in mysterious ways, we tend to go through a familiar song and dance. As experience, skills and even personal networks grow, we can find ourselves diving ever further in the following chain. 1. "It must be in my code." -- hours of debugging 2. "Okay, it must be somewhere in our codebase." -- days of intense debugging and code spelunking 3. "It HAS TO be in the third party libraries" -- days of issue tracker excavations and never-before-enabled profiling runs 4. "It can't possibly be in stdlib..." -- more of the same, but now profiling the core runtime libraries 5. "Please let this not be a compiler bug" -- you become intensely familiar with mailing list archives 6. "I will not debug drivers. I will not debug drivers. I will not debug drivers." 7. "I don't even know anyone who could help me figure out the kernel innards." 8. "NOBODY understands filesystems!" 9. "What do you mean 'firmware edge case'?" And the final stage, the one I have witnessed only one person ever achieve: 10. "Where is my chip lab grade oscilloscope?" Apart from bullheadedness, this chain also highlights another trait of a good developer. Humility.
- kabdib 10y ago
  I've done the oscilloscope thing, though it was only a vanilla couple-of-hundred Mhz scope on some pins that were bugging me, and not the full deal: "Gang way, we're cracking the lid of that thing and going in." That sounds exciting, I'd love to see it. Once, I used a chunk of ice to cool down a chip, and that made it work. The hardware guys were unimpressed. But hey, they've got cans of Chill and they use them a lot, and this software guy took a while to realize the reason the board worked in the morning and was dead by lunch, and worked for a little while again after lunch, was temperature related. There were some devs who tracked down a nasty bug in a processor's TLB. I only heard about that one, wish I had been there. I only had to deal with the fallout in the hypervisor. Note: If you have to spend 20ms hunting down and killing lies with all interrupts turned off and everything basically stopped in its tracks, you are no longer a real-time operating system.
  - bostik 10y ago
    Heh. It could be telling that I had to look up the expansion for TLB. CPU cache implementation... holy crap. My ex-coworker has done the vanilla scope thing too and has a 400MHz scope at home. For some reason people like this are not too uncommon in Finnish oldskool[tm] IT scene. I remember how he isolated a latency and concurrency bug to an expensive interrupt handler. Rewriting isolated parts of core kernel code to make a really tricky problem go away was one of his more hardcore skills. I'm not even near his level. My own experience is limited to slightly nibbling the edges of file system and block cache behaviour. It's a brave person who dares dive into that code. Not me. But I do know one person who regularly works with decapped chips. He works for a company who do extremely low-level hardware investigations. Now that's hardcore.
    - kabdib 10y ago
      Cache bugs are one of the fun ones. You think you're losing your mind and the people around you would probably agree. A couple of weeks go by, your spouse is ready to fire you, your boss wants to divorce you, and every waking moment is full of race conditions. Four-way stops on the drive to work are a source of stress and you punch buttons in the elevator and worry about firmware bugs. Then you get to your desk and there's the setup, a laughably small board for all the trouble it's made, and it's time for single combat, Sherlock Holmes style. When you find the problem it's usually a blinding flash of realization that illuminates a tiny, eensy bit of code that you tweak and make right in a couple of minutes. Invariably the mistake was pretty stupid. The glory moment is over quickly because you know all the test cases will pass and that you've just nailed another one. You've got bragging rights during one lunch, but that's it. It's off to more mundane bugs in the mortal world, and you feel a little sad. I need to do hardware again.
deleted 10y ago
[deleted]
kazinator 10y ago
> It turned out that the std::string empty string reference count was just doing a vanilla ++, no locking, nothing, variable not marked volatile, nothing. The whole implementation is smells of silliness, because there is no need to track how many references there are to a global null string, which need not even be dynamically allocated.
- bluejekyll 10y ago
  I know; I've often wondered if this was changed, never went back to look.
  - daemin 10y ago
    Probably a few years ago, around C++11 time. Where the standard made it not possible to have a Copy-on-Write implementation of string.

bostik 10y ago

> you can't trust libraries blindly, even one of the most used and broadly adopted ones There is a corollary to development and debugging. When things break in mysterious ways, we tend to go through a familiar song and dance. As experience, skills and even personal networks grow, we can find ourselves diving ever further in the following chain. 1. "It must be in my code." -- hours of debugging 2. "Okay, it must be somewhere in our codebase." -- days of intense debugging and code spelunking 3. "It HAS TO be in the third party libraries" -- days of issue tracker excavations and never-before-enabled profiling runs 4. "It can't possibly be in stdlib..." -- more of the same, but now profiling the core runtime libraries 5. "Please let this not be a compiler bug" -- you become intensely familiar with mailing list archives 6. "I will not debug drivers. I will not debug drivers. I will not debug drivers." 7. "I don't even know anyone who could help me figure out the kernel innards." 8. "NOBODY understands filesystems!" 9. "What do you mean 'firmware edge case'?" And the final stage, the one I have witnessed only one person ever achieve: 10. "Where is my chip lab grade oscilloscope?" Apart from bullheadedness, this chain also highlights another trait of a good developer. Humility.

kabdib 10y ago
I've done the oscilloscope thing, though it was only a vanilla couple-of-hundred Mhz scope on some pins that were bugging me, and not the full deal: "Gang way, we're cracking the lid of that thing and going in." That sounds exciting, I'd love to see it. Once, I used a chunk of ice to cool down a chip, and that made it work. The hardware guys were unimpressed. But hey, they've got cans of Chill and they use them a lot, and this software guy took a while to realize the reason the board worked in the morning and was dead by lunch, and worked for a little while again after lunch, was temperature related. There were some devs who tracked down a nasty bug in a processor's TLB. I only heard about that one, wish I had been there. I only had to deal with the fallout in the hypervisor. Note: If you have to spend 20ms hunting down and killing lies with all interrupts turned off and everything basically stopped in its tracks, you are no longer a real-time operating system.
- bostik 10y ago
  Heh. It could be telling that I had to look up the expansion for TLB. CPU cache implementation... holy crap. My ex-coworker has done the vanilla scope thing too and has a 400MHz scope at home. For some reason people like this are not too uncommon in Finnish oldskool[tm] IT scene. I remember how he isolated a latency and concurrency bug to an expensive interrupt handler. Rewriting isolated parts of core kernel code to make a really tricky problem go away was one of his more hardcore skills. I'm not even near his level. My own experience is limited to slightly nibbling the edges of file system and block cache behaviour. It's a brave person who dares dive into that code. Not me. But I do know one person who regularly works with decapped chips. He works for a company who do extremely low-level hardware investigations. Now that's hardcore.
  - kabdib 10y ago
    Cache bugs are one of the fun ones. You think you're losing your mind and the people around you would probably agree. A couple of weeks go by, your spouse is ready to fire you, your boss wants to divorce you, and every waking moment is full of race conditions. Four-way stops on the drive to work are a source of stress and you punch buttons in the elevator and worry about firmware bugs. Then you get to your desk and there's the setup, a laughably small board for all the trouble it's made, and it's time for single combat, Sherlock Holmes style. When you find the problem it's usually a blinding flash of realization that illuminates a tiny, eensy bit of code that you tweak and make right in a couple of minutes. Invariably the mistake was pretty stupid. The glory moment is over quickly because you know all the test cases will pass and that you've just nailed another one. You've got bragging rights during one lunch, but that's it. It's off to more mundane bugs in the mortal world, and you feel a little sad. I need to do hardware again.

deleted 10y ago

[deleted]

kazinator 10y ago

> It turned out that the std::string empty string reference count was just doing a vanilla ++, no locking, nothing, variable not marked volatile, nothing. The whole implementation is smells of silliness, because there is no need to track how many references there are to a global null string, which need not even be dynamically allocated.

bluejekyll 10y ago
I know; I've often wondered if this was changed, never went back to look.
- daemin 10y ago
  Probably a few years ago, around C++11 time. Where the standard made it not possible to have a Copy-on-Write implementation of string.