9 ms·

> you can't trust libraries blindly, even one of the most used and broadly adopted ones There is a corollary to development and debugging. When things break in

• kabdib 10y ago
  I've done the oscilloscope thing, though it was only a vanilla couple-of-hundred Mhz scope on some pins that were bugging me, and not the full deal: "Gang way, we're cracking the lid of that thing and going in." That sounds exciting, I'd love to see it. Once, I used a chunk of ice to cool down a chip, and that made it work. The hardware guys were unimpressed. But hey, they've got cans of Chill and they use them a lot, and this software guy took a while to realize the reason the board worked in the morning and was dead by lunch, and worked for a little while again after lunch, was temperature related. There were some devs who tracked down a nasty bug in a processor's TLB. I only heard about that one, wish I had been there. I only had to deal with the fallout in the hypervisor. Note: If you have to spend 20ms hunting down and killing lies with all interrupts turned off and everything basically stopped in its tracks, you are no longer a real-time operating system.
  • bostik 10y ago
    Heh. It could be telling that I had to look up the expansion for TLB. CPU cache implementation... holy crap. My ex-coworker has done the vanilla scope thing too and has a 400MHz scope at home. For some reason people like this are not too uncommon in Finnish oldskool[tm] IT scene. I remember how he isolated a latency and concurrency bug to an expensive interrupt handler. Rewriting isolated parts of core kernel code to make a really tricky problem go away was one of his more hardcore skills. I'm not even near his level. My own experience is limited to slightly nibbling the edges of file system and block cache behaviour. It's a brave person who dares dive into that code. Not me. But I do know one person who regularly works with decapped chips. He works for a company who do extremely low-level hardware investigations. Now that's hardcore.
    • kabdib 10y ago
      Cache bugs are one of the fun ones. You think you're losing your mind and the people around you would probably agree. A couple of weeks go by, your spouse is ready to fire you, your boss wants to divorce you, and every waking moment is full of race conditions. Four-way stops on the drive to work are a source of stress and you punch buttons in the elevator and worry about firmware bugs. Then you get to your desk and there's the setup, a laughably small board for all the trouble it's made, and it's time for single combat, Sherlock Holmes style. When you find the problem it's usually a blinding flash of realization that illuminates a tiny, eensy bit of code that you tweak and make right in a couple of minutes. Invariably the mistake was pretty stupid. The glory moment is over quickly because you know all the test cases will pass and that you've just nailed another one. You've got bragging rights during one lunch, but that's it. It's off to more mundane bugs in the mortal world, and you feel a little sad. I need to do hardware again.
      • nikanj 10y ago
        Your writing reminds me of https://www.usenix.org/system/files/1311_05-08_mickens.pdf https://www.usenix.org/system/files/1311_05-08_mickens.pdf
  • steve_gh 10y ago
• CiaranMcNulty 10y ago
  11. "Hm maybe I should check my code again... ah there's the bug"
  • doubleplusgood 10y ago
    "oh, this config shouldn't be linked to /dev/null..."
  • fnord123 10y ago
    if (featureFlags[HN_DEBUG_HIER_FLAGS] = null) { /* Who won't this trigger!?!?!? */ } ...oh god, kill me now.
• doubleplusgood 10y ago
  I feel lucky to only have reached item 4; I wouldn't feel confident beyond it, anyway.
  • marcosdumay 10y ago
    I don't think anybody is ever confident beyond level 1. The worst part is those "why the fuck does everything work when I plug a logical analyzer?" moments.
    • doubleplusgood 10y ago
      I'm usually well into #3 before I realize I'm in too deep
• s_kilk 10y ago
  I've only seen level 5 personally, with a C# compiler bug that would omit totally valid `else` branches.
  • kyberias 10y ago
    Care to elaborate? Which C# compiler?
    • s_kilk 10y ago
      The official one, I think it was in .NET 3, but it was a few years ago at an old job, so I'm a bit hazy on the details. Basically we had a bug where a whole conditional branch was being skipped, and we traced it down to the branch being omitted entirely from the compiled IR. And no, it was nothing fancy, just something like: if (customer.country == "US") { doSomething(); } else { doDifferentThing(); } The whole `else` branch was simply missing from the compiled program. If I remember correctly, we got around it by doing something like: bool isUsCustomer = customer.country == "US" if (isUsCustomer) { doSomething(); } else { doDifferentThing(); } Anyway, the point is that the compiler fucked up it's handling of if/else statements, but only at that specific part of the code, leading to a few wasted days of effort tracking down the problem.
      • SideburnsOfDoom 10y ago
        > Anyway, the point is that the compiler fucked up it's handling of if/else statements, but only at that specific part of the code It would have to be specific. Put it this way: if this was a general bug and the "else" was always omitted, how long would it last before being found and fixed? Related, if you were to say to me "I found the issue, the compiler isn't correctly handling if/else statements" Then my first thought would be about your medication not about the compiler.
        • s_kilk 10y ago
          > if you were to say to me "I found the issue, the compiler isn't correctly handling if/else statements" Then my first thought would be about your medication not about the compiler And yet, it happened :) And the senior engineers at the company looked at it and confirmed it was a compiler bug. Their best guess was that something about that part of the code was putting the compiler in a funny state, causing it to skip that particular `else` branch. We reported it to Microsoft, but never heard anything back.
• rjsw 10y ago
  The furthest that I have got down the list was trying to bring up the first prototype of a board that had been designed with too long traces on the PCB between the SoC and DRAM. If you tried to read a location in memory you got the value of the page table entry for that address rather than the address contents.
  • kosma 10y ago
    I once had to debug a poorly-designed board where the CPU would lock up if you did a DRAM burst write with at least 3 of the 5 highest bits of the word set (yes, I narrowed the test case down that far). A quick look at the layout confirmed that those traces were routed directly under the crystal oscillator without any form of ground shielding... (We ended up underclocking the CPU by about 20% because there wasn't enough time for a redesign. Sigh. It's a miracle the thing even worked in the first place...)
    • kabdib 10y ago
      ... then your power supply goes marginal (because it will) and well . . . never been there :-)
  • fest 10y ago
    I've had the opportunity spend about a week debugging incorrect configuration of SDRAM by BSP team. At first I blamed third party library with no source code available. Then it occurred to me that my initial SDRAM tests were doing word-by-word access. Third party library used memset which was optimised to use DMA for bulk transfers, which failed to write subsequent words in the same transaction. An easy, one bit change in configuration registers of SDRAM fixed it. A week well spent!
  • kps 10y ago
    Similar: my new driver crashes the machine. A couple days debugging. Triple-check every register value. All good. It doesn't crash when I single-step! A couple more days debugging. Finally get it: the machine crashes when two ports are enabled close enough in time. Go talk to the hardware guys. “Yeah, we know, power traces are fixed in the next rev.”
• frik 10y ago
  I've done the oscilloscope thing, but it was for IoT stuff - debugging broken a I2C communication with Arduino (8-bit 16MHz ATMega CPU). The Arduino software stack is not huge, there is no operating system involved. Our application is the only thing that runs on bare slow hardware with very limited memory. But this also makes debugging harder. The IDE is limited, you debug over serial output. You have to reflash the flash-memory after every re-compile, which can take a minute. Building a IoT system for very specific tasks that has to run reliable for years without interruption, I would still use a 8-bit tiny ATMega CPU (e.g. Arduino), and to control this tiny CPU and do some networking stuff with a control center using a 32bit ARM CPU (e.g. RPi).
  • garaetjjte 10y ago
    > The IDE is limited, you debug over serial output. You have to reflash the flash-memory after every re-compile uh, you know that AVRs have debugWIRE (smaller parts) or JTAG (bigger parts)?
• kosma 10y ago
  > 10. "Where is my chip lab grade oscilloscope?" 11. "Shit, where do I borrow a spectrum analyzer and a set of near field probes? These things cost an arm and a leg!" Yes, STM32F1 MCUs generate inference that jams GPS receivers. No, it's not documented anywhere.
  • yzmtf2008 10y ago
    And here's the documentation for future generations! :P
  • northern_lights 10y ago
    any MCU can "jam" a GPS receiver if the board is laid out improperly or without enough shielding
    • kosma 9y ago
      A bare, free-floating STM32F103 with literally nothing but a LiPo battery connected with two wires, running the blinky.c demo, will completely jam many GPS receivers when placed next to the antenna.
  • makomk 10y ago
    You'd be surprised how much you can find out just with a $10 TV tuner dongle and a piece of coax with a short section of the outer braid trimmed off at once end.
    • ethbro 10y ago
      You're describing my college tv antenna. Coax is cheap. An actual digital antenna is like... 50 ramen equivalents.
      • eltoozero 10y ago
        50 ramen = $10 USD
    • kosma 9y ago
• RX14 10y ago
  This is the worst thing I've ever found, still not solved: https://github.com/crystal-lang/crystal/issues/4127 https://github.com/crystal-lang/crystal/issues/4127
  • lloeki 10y ago
    You must be kidding me. We hit a bug with the mysql2 gem where the client randomly crashes in libmariadbclient (but not libmysqlclient) only on debian (using Arch Linux and OS X for dev, but exact same versions of everything) and only for database names of length 25. And 28, but we cannot reproduce it on the repro docker image we made. And only if there are enough aliases in the query (could be as low as 5 but could need as much as 20+). And only if 'active_record' is require'd, but even when it's not used at all. And never ever under GDB or Valgrind, making it the perfect heisenbug. That's a lot of stars to align there, but when they do, hell breaks lose just often enough to be sure it's not completely random, and obviously this hit one of our most finicky customers, and only in production because of course "#{customer}_production".size == 28 (and not 25, because nah that'd have been too easy to be able to reproduce the bug right away). [0] bug: https://github.com/brianmario/mysql2/issues/822 https://github.com/brianmario/mysql2/issues/822 [1] repro: https://github.com/adhoc-gti/mysql2_pointer_bug https://github.com/adhoc-gti/mysql2_pointer_bug
• SideburnsOfDoom 10y ago
  I have also seen developers far too keen to blame the library before exhausting the most likely case that the issue is caused by the local code (step 1 and 2), or at any rate is fixable in it. It's a well-known syndrome. The classic motto for it is "'SELECT' isn't broken" https://blog.codinghorror.com/the-first-rule-of-programming-its-always-your-fault/ https://blog.codinghorror.com/the-first-rule-of-programming-...
  • ethbro 10y ago
    This too. For every one time it's the parent article, 99 times it's my code.
• gone35 10y ago
  11. "Do we have the IP core for this?" 12. "Where is my electron microscope?"
  • shabble 10y ago
    ..13 "We're gonna need some time on the FIB workstation"[1] [1] http://www.electronicdesign.com/eda/fib-circuit-edit-becomes-increasingly-valuable-advanced-node-design http://www.electronicdesign.com/eda/fib-circuit-edit-becomes...
    • nickpsecurity 10y ago
      That's the one I should've thought of. I said fab but who trusts them to know what's on it!? Haha.
• arglwarg 10y ago
  This one goes to eleven: 11. "On some setups, clients get a corrupted stack when swapped back in from kernel." (https://bugs.launchpad.net/ubuntu/+source/linux/+bug/745836 https://bugs.launchpad.net/ubuntu/+source/linux/+bug/745836)
• uep 10y ago
  From my experience this is normal for embedded development; particularly for consumer electronics. Part of the reason some developers in this space have to wear so many hats is that the pace in consumer electronics is unforgiving. I don't think my current employer is unusual either. Hopefully, the number of frameworks at the top, and the size of your individual programs are relatively small (so that 1-3 aren't nightmares by themselves). In my experience, 4-5 are seldomly the problem (thanks Linaro!). I suspect the ratio of C to C++ is significantly larger in embedded systems though. In general, PowerPC/MIPS/ARM toolchains and drivers are not as mature as x86/AMD64. 6-8 tend to occur because CPU vendors usually have their own "blessed" toolchains and BSPs that have diverged from their upstream projects. Fortunately, this means that it's often the case that someone else has already fixed the problem. It's just as often that a driver has not been tested for your use-case since the last time that particular driver's infrastructure was refactored inside the kernel. Or... you wrote the driver and made the mistake (or it might be something from 9/10). 9-10 happen because we're often using hardware that is new and has not had all of its errata discovered yet. When products need to ship, we're regularly going through this stack. I've seen every one of these, even in just the last 4 years.
  • mighty_atomic_c 10y ago
    Can confirm. I had a trippy experience where I had on one monitor some RTL+simulation for our chip up for view, on another I had the PCB schematic I had helped design, and on my third I had the GUI and embedded toolchain development environments up, and on my desk I had an oscilloscope measuring that PCB running that firmware. It was basically rolling through the list and really fun!
    • rbanffy 10y ago
      Indeed. I too had once the dubious pleasure of having an oscilloscope on my desk, between two computers and a prototype.
• TimJYoung 10y ago
  Spot on. The experience level of a developer is directly proportional to how fast he/she assumes that the problem is in someone else's code. ;-)
  • Natanael_L 10y ago
    That one depends a lot on who your coworkers are too.
    • TimJYoung 10y ago
      I was thinking more of the code in the OS, compiler, run-time, etc. I should have made that more clear in my comment.
• tripzilch 10y ago
  Ooofff, that list made my stomach churn, more stuff of nightmares! All debugging post-mortems of this level should be written in Lovecraftian style. ... it's not widely known, whispers attribute it to a transcription error, unsure when it started, copied through ancient manuscripts, that the Dead Thing that lies dreaming at the bottom of the ocean, is actually named ... C++hulhu
• nickpsecurity 10y ago
  Regarding No 10: http://cs.dartmouth.edu/~sergey/cs258/2010/D2T1%20-%20Kris%20Kaspersky%20-%20Remote%20Code%20Execution%20Through%20Intel%20CPU%20Bugs.pdf http://cs.dartmouth.edu/~sergey/cs258/2010/D2T1%20-%20Kris%2...
• syngrog66 10y ago
  went through these kinds of stages a few times in my career. once it led me to discover a leak in a major travel website's purchase flow caused by Java's Thread class, related to thread groups. most recently, I was writing some Linux auth code in C, reached a point where I could rule out my code, and found a bug in sudo. freaking sudo. (also related to groups, though the Linux user kind, not Java threads.)
• asdfgadsfgasfdg 10y ago
  For devs using higher level languages it is more like: 1. "It must be my code" -- minutes of debugging 2. "It must be in our codebase" -- hours of debugging 3. "Third party library or framework" -- If library use a different library, if framework accept the bug and work around it whilst cursing framework choice.
• Balgair 10y ago
  Regarding #10: Oh Lord, I've been there too many times to count.... One of the more memorable time was with an old timing distribution system. The thing would pretty much just send out clock pulses to networked machines and this cost a lot of money to do properly (very abridged here). This particular one was acting 'funky' and came back in. In testing it, we go really weird behavior. True to your list, I think we went 1, 2, 5, 6 (no drivers, per se), 7 (for about 5 days), 8, 9, 10. At 10, we finally plugged in the o-scope and started debugging the PCB vias and connections themselves. Things were getting really wacky now. The Faraday cage that was the testing room had to be re-grounded, we thought, as the wires themselves were still carrying current even when the power was dis-connected. One of the guys brought in his old hand-held impact hammer to drive a new copper stake into the peeled up linoleum and through the foundation of the building. Still, we got strange results. Like really strange results that, to us, were worthy of a Nobel Prize, as we had thus far proved to ourselves that physics herself was broken inside the lab. For reference: a lot of people worked in there, so having stuff about in all kinds of dis-repair was typical. I remember, long after the pizza had gone cold and the Mt. Dew was flat, I was looking up at the ceiling of the room. I saw an old RF horn hanging from the roof, kinda held together with the connecting wires. 'Hey, if that thing was on, would it do anything?' The other techs' eyes all lit up. Turns out, one of the guys was doing something with the horn for some other test. He had left for an extended backpacking vacation and accidentally still had the thing on. The broadcasting from the horn was adding the small amount of current to all our wires, thereby causing the whole box to go out of whack just enough to cause all the issues. At about 4 am, we finally got the box re-configured, the original problem from the customer solved, and all of it packed up and ready to overnight out to the customer for when the UPS store opened at 7am, about 3 hours from then. The poor guy got back from vacation to that mess of an email inbox and many meetings. It was an honest mistake, and he bought us all 12 packs for the trouble. Still, when you think you have proved that physics is broken, I think that will qualify as step 12.
• Beltiras 9y ago
  This is the most concise list of debugging layers I have seen. I'm mostly commenting so that I get this into my comment feed and can locate it easier.
  • bostik 9y ago
    In that case, feel free to bookmark this one: http://bostik.iki.fi/aivoituksia/random/developer-debugging-progression.html http://bostik.iki.fi/aivoituksia/random/developer-debugging-... This got more attention that I thought possible, so I decided to pull it out as an item all by itself.