53 ms·

Print this file, your printer will jam (2008)

492 points by segfaultbuserr 6y ago

hawktheslayer 6y ago
This is why I love bugs, especially the sporadic ones--often you grow in understanding of the underlying system or code as you solve them.
- segfaultbuserr 6y ago
  > especially the sporadic ones Yes, but there's only a fine line between "sporadic but solvable by analysis" and "too sporadic to be reproduced or solved..." Device driver problems are often from the second category... Nevertheless, it can get interesting if the problem is reproducible like the "500 miles email" or this printer.
  - marcan_42 6y ago
    Nothing is too sporadic to be reproduced, you just haven't found how to reproduce it yet :) I once solved a once-in-a-petabyte bug at Google, when I was still working there. One of our systems ran in two data centers on the same data, and the cross-check was detecting a record offset mismatch but not a data mismatch (?!); retrying the process almost always fixed it. Long story short, there was a buffer overread in Google's bespoke zlib compressor that caused it to emit slightly less efficient compressed data (one or two bytes longer) nondeterministically, even though the data was perfectly well-formed. We were detecting compressed records being at different offsets, even though the uncompressed data matched. Another Google one involved doing a postmortem on a single instance of bad data ending up on a system. I traced that back purely from logs to having been caused by a kernel panic two systems upstream, that caused disk data to not be flushed while metadata was, so on reboot the system picked up garbage from the disk sectors in the file, which turned out to be well formed data from a different file and it trickled down the layers. Also, this one has made the rounds on HN a couple times (second time I fix a golang runtime heisenbug too; first one was also fun): https://marcan.st/2017/12/debugging-an-evil-go-runtime-bug/ https://marcan.st/2017/12/debugging-an-evil-go-runtime-bug/ And I recently fixed an OBS bug that has been randomly crashing audio for streamers for at least 3 years. I found a reliable repro that involved scripting monitoring mode toggles dozens of times per second; along the way I found and fixed 3 other bugs. That one was exacerbated by some of the OBS developers having built a rhetoric that those failures were caused by user configuration mistakes, because they often went away after changes - when what was actually going on was that users were blindly trying things to solve the problem, and sometimes stumbling on a somewhat stable workaround. (That one hasn't been reviewed/merged, still in the pipes). Then there was that bug in The Homebrew Channel that I introduced after adding freetype/TTF support... That was a heap corruption that was crashing things way later (embedded baremetal code, so no memory protection or debugging tools beyond remote gdb). Finding that one ended up involving dumping memory around the corrupted heap and following pointers to what eventually turned out to be graphics queue entries, then following texture pointers and teaching myself how to read swizzled character textures from hexdumps... I eventually found I was not allocating entries for space characters in strings, but was incrementing the index on spaces anyway. Lots of fun bug hunting stories...
ovi256 6y ago
Any change of state in a system with inertia should have hysteresis! In this case, a minimum duration before restarting would have done the trick.
- segfaultbuserr 6y ago
  > Any change of state in a system with inertia should have hysteresis! Or noise.
mikewarot 6y ago
Back in the days of Windows 98, the HP LaserJet caused no end of headaches for me. The problem that was introduced by the compiler they used to build the print driver. To compute the paper size, margins, etc., floating point operations were used... but by then the driver had allowed the libraries that emulated the floating point processor to be unloaded, to save memory. (Due to the optimizing compiler) Any operation that involved computing the page size would crash, only if the floating point library had idled out of memory. I ended up forcing the library to be loaded at boot on all machines, and then things were stable.
- sharken 6y ago
  No wonder that one of the DevOps success stories revolves around HP printer drivers and firmware: https://itrevolution.com/the-amazing-devops-transformation-of-the-hp-laserjet-firmware-team-gary-gruver/ https://itrevolution.com/the-amazing-devops-transformation-o...
- cesarb 6y ago
  One thing which I always considered a misdesign in Windows, is its tendency to run third-party code in-process. This includes COM objects, printer drivers, and many more. We had a problem similar to yours at a previous company. The software was written in a proprietary language which had its own runtime, and that runtime expected a particular value for the floating point control word. There are some third-party DLLs, including some printer drivers and Explorer extensions, which when loaded overwrite the floating point control word with their own notion of what should be the correct value (this behavior, as far as I could determine, is introduced by the compiler used to build these third-party DLLs). The unexpected value for the floating point control word would then lead to crashes, but these crashes would not happen immediately, and they would only happen if the user had done something which led to one of these third-party DLLs being loaded, like opening/saving a file (the open/save dialog in Windows loads Explorer extensions) or printing (which loads the printer driver). The solution was to, after every point in the code which opens a file chooser or prints anything, call into a small stub DLL we wrote, which had code to reset the floating point control word to what the runtime expected.
  - 6y ago
Pxtl 6y ago
It's reassuring to know that printers have always been crap.
- dylan604 6y ago
  My 2 least favorite things about computers: Printing and Fonts.
  - jnwatson 6y ago
    Importantly, it was the two things that early Macs got right. Other than price, a 1990 Mac/LaserWriter combo is superior than a modern equivalent.
    - thih9 6y ago
      Current status of apple printer setup seems right to me too. I'm happy that I can print wirelessly from macOS or iOS, without installing any drivers or configuring anything (other than the wifi password).
      - jamiek88 6y ago
        Yeah Apple typically get printing and audio right. Hi res too. Lots of things they aren’t so strong on but they have always been good at the above.
- bananamerica 6y ago
  https://www.wired.com/story/why-do-printers-still-suck/ https://www.wired.com/story/why-do-printers-still-suck/
- coryrc 6y ago
  Printers being crap led to the Free Software movement and the GPL
  - gpvos 6y ago
avisser 6y ago
My favorite bug like this was me in my buddy's 2000 VW Jetta. The car had some electrical issues, but my buddy once remarked that the windows stopped working whenever he gave me a ride. I'm 6'4" and always slide the seat back whenever I get in a car. Turns out the circuitboard controlling the windows was under the passenger seat and had come loose. Moving the seat back caused a short, "breaking" the power windows. Turns out, the windows _did_ stop working when he gave me a ride. edit: "I have him" => "he gave me"
- jacquesm 6y ago
  I had a Citroen ID 19B which had a pretty neat quirk: the starter relay would only get power if you switched on the headlights. I don't know if that was on purpose or not (some primitive anti-theft device) but I left it in because it was actually a neat feature.
  - ComputerGuru 6y ago
    It’s a legal requirement to have headlights on any time the car is in operation (day or night) in some territories and manufacturers had different ways of enforcing it. Some unconditionally turned on the headlights for models sold in those countries, others had special variants earmarked for export to those countries. It seems Citroen found a clever way that didn’t require too much fiddling to meet the legal requirement.
    - jacquesm 6y ago
      Not in 1968, the year that car was manufactured in, and none of the other ID's that I've seen had this particular quirk, it was either done by one of the (many) previous owners or a fluke. The wiring in those cars got very brittle over time and shorts were pretty common. That car came from France to NL (I imported it). Anyway, it was a fun gimmick and it helped to dissuade people from taking it for a ride. I never did find out if it was on purpose or accidental, that dash was not one to disassemble if you didn't need to.
      - 6y ago
Evidlo 6y ago
Here's a semi-related question that maybe somebody can answer. Why is it that despite listening on TCP 9100, some printers will just print the source code when you send them a PDF or Postscript file with netcat? Is there some special PJL command to put it into Postscript mode? Are there any networked printers around nowadays that don't support Postscript?
- jagged-chisel 6y ago
  “...despite listening...” Yes, there’s a protocol behind that port. Just like there’s one behind port 22, 80, 443... you can’t just dump http to 443 and get a useable document in return.
  - tonyedgecombe 6y ago
    Actually that is exactly how the printers listened on port 9100. You open a connection and dump the raw print file. This "protocol" was introduced by HP with their JetDirect print servers.
  - jolmg 6y ago
    > you can’t just dump http to 443 and get a useable document in return. Not on 443, but on 80 you basically would. You'd just have to skip the response headers.
- buckminster 6y ago
  A lot of printers, by default, function like an ancient teletype. They interpret incoming data as something like ASCII and print out the text. You need to send the correct escape sequence (i.e. a sequence of bytes starting with 0x1B) to switch into a more modern mode.
- jschwartzi 6y ago
  A lot of Postscript printers are configured to print source code when there’s a syntax error in the file. My 2017 HP laser printer has this as an optional feature.
  - 6y ago
loloquwowndueo 6y ago
Lol “OpenOffice can’t print on Tuesdays” https://bugs.launchpad.net/ubuntu/+source/cupsys/+bug/255161/comments/28 https://bugs.launchpad.net/ubuntu/+source/cupsys/+bug/255161...
11thEarlOfMar 6y ago
Tell me if this is a bug... After driving a new car to work for 2 years, the traction control started kicking in on certain curves. I take a windy road to work and had never had traction control activate. When it kicked in, it pulsed the right front brake and slowed the car quickly, likely unnerving the guy tailgating me. This happened several times, both going to and from work. The road was perfectly dry, no snow or rain. Same turns at the same speed as always. The clues are was that my rear tires had worn and I replaced them a few days before. And, I drive a Tesla Model 3. It's a rear-wheel drive version and the front tires still had good tread so I didn't replace them. Why the right front tire? What I finally brained out, which you may have quickly realized, is that the diameters of the worn front and new rear tires were then slightly different. The front had a lower diameter than the rear and therefore turned at a slightly faster rate when driving. When I took a hard bank, the outside wheels also turned faster than the inside as they do in any turn. That increased the delta in spin rate the most between the outside worn front tire and the inside new rear tire. The traction control determined that the right front must be slipping and pulsed the brake on it. I replaced the front tires and everything went back to normal.
- cowsandmilk 6y ago
  Why not just move the tires from the front wheels to the back?
  - bbddg 6y ago
    Yeah isn't this why you should rotate your tires regularly?
    - jhoechtl 6y ago
      You get downvoted but yes, that's the reason for swapping front to back and vice versa.
  - crazyjncsu 6y ago
    Just so the laymen here don’t think this is some kind of hack, this is what is referred to as “rotating your tires”, which should sound familiar. The problem is that tires are often not rotatable due to staggered setups where the front and back tires are different sizes (very common) or due to directionality of tires (less common).
famousactress 6y ago
I may have told this story before (perhaps back in 2008 if this was posted then) -- but my first gig in tech was QA for Hewlett Packard testing printer/scanner/copier units. At some point I scanned a photo of my then-girlfriend and crashed the unit. Was able to do it repeatedly, provided I scanned the image in straight/cleanly. I didn't come across any other images that reproduced the issue, only this one photo. The underlying bug was a buffer overrun of some sort, was fixed before shipping -- and the image became a part of the durable regression test suite for that department.
- ComputerGuru 6y ago
  Not surprised there was a bug, but I am surprised it was patched (before or after shipping). I have an “enterprise grade” HP scanner (3500 F1) with buggy Twain drivers I had to manually disassemble and patch that will crash any time you try to duplex scan a single-sided sheet of paper. The blank page detection algorithm caused a null pointer dereference if two-sided mode was selected but the second side was elided. The bug reproduced 100% of the time with all available versions of their drivers in any application using the TWAIN rather than WIA or proprietary scanner interface. It was as simple as scanning any one-sided document in duplex mode. I escalated the issue to their engineers who kept insisting the issue was with my PC until the product was EOL’d just a year or two after production. I had other hardware issues pop up and ended up getting a Fujitsu for ADF document scanning and keeping the HP for scanning photos with the flatbed. Not many stand-alone hi-res dual color ADF/flatbed scanners out there.
  - ryandrake 6y ago
    If you're in software, you probably know this, but your bug was likely ignored not because it couldn't be reproduced, but because the engineers on the project were prioritizing feature work for the next version of the scanner, and did not have any support from management for fixing bugs in an already-shipping product. Sadly, driver software quality tends to max out at the level where customers won't return the hardware for a refund, and then the company abandons it and works on the next shiny new device.
thewebcount 6y ago
I hit a similar issue in OpenGL on macOS about 10 years ago. We had a fairly complex set of draw calls. It worked fine on high-end machines, and was perfectly usable, if a bit slow, on mid-range machines, but on low-end iMacs with crappy Intel graphics cards, it would cause the machine to panic and reboot. It ended up being that the draw calls filled up the smaller command buffer on the low end card, and the driver would crash. In talking with the implementors, they explained the situation, so I asked, "How do I determine that I'm filling up the command buffer?" The answer was basically, "The size of the command buffer is an implementation detail and there's no way to query OpenGL for it or for the size of the commands you're adding." Gee, thanks! We ended just breaking up the draw calls into more smaller calls and it solved the problem, but this entire bug was such an ordeal. Thankfully we're moving away from OpenGL and things are a bit more sane in Metal. Had I not had access to the OpenGL team at Apple at the time, there's no way I would have been able to figure out what was going on. I'm not sure if I would have thought to do the same work in smaller batches.
- marcan_42 6y ago
  That's a driver bug.
- cmg 6y ago
  I wonder if this was related to a kernel panic bug I ran into in Chrome around the same time! Long story short, we were using a library (Modernizr) on our web product that checked for some kind of OpenGL support that we didn't actually need, and just running that code would trigger a driver bug. Disabling that fixed it!
quercusa 6y ago
From the comments, the 500-mile email: http://www.ibiblio.org/harris/500milemail.html http://www.ibiblio.org/harris/500milemail.html
SoSoRoCoCo 6y ago
For the last 5~7 years, thousands of software-only programmers have been discovering Arduino and Raspberry Pi, and have been learning about the real-world consequences of interfacing to hardware. I help out at a local hackerspace and the excitement of someone discovering a software/hardware interface bug is contagious. Here's an example: someone was trying to make a plant watering machine and was using a FET to control the DC motor. The wire to the FET was about 15 feet long so the slope from the poor GPIO was horrendous, and the motor would basically brownout because it couldn't overcome the initial stall di/dt. The idea that not all edges are perfect 90 degrees was eye opening to him.
- myself248 6y ago
  Oh, dang, yeah. Hackerspace here too, and software people colliding face-first with electrical reality is an endless source of learning, amusement, and project delays. Some quality time with an oscilloscope goes a long way, but still cannot overcome all stubbornness.
- robocat 6y ago
  Is that because many software programmers use “genetic coding” techniques where bugs are randomly fixed until the software works? Or were you mostly dealing with newbie programmers? It’s not like software doesn’t have similar failure modes to hardware, right?
  - SoSoRoCoCo 6y ago
    I don't think it was... maybe? I've seen people hack circuits they way they hack code: keep trying small tweaks until it works. Then when they connect two circuits and it doesn't work, they go back to the fundamental RTFM. It could also be the skillset: circuits are more a creative art form, logic is more... logic. (But that doesn't always hold because I've encountered cache/CAM RTL designs that were beautiful.) Failure modes are different. A transistor can go bad if you accidentally drove it too hard, so now you might need to replace every part until the circuit works. And maybe one of the xtors in your box is bad. Or maybe a wire isn't making a good contact. Just the physical aspects alone without bringing an oscilloscope into the mix ("Why is my signal so noisy? Oh, it's the fluorescent lights, or the cheap USB hub noise in 400MHz spectrum, or the wall-wart...") can be maddening. Mostly I think the people I helped out seemed to forget that "digital" only exists inside the O/S, whereas the real world is analog, and real-world debuggers are more abstract than using, say, gdb or journalctl. But then I work with MY mentors, and they have an even larger set of debug skills based on extensive component knowledge that reminds me of chess grandmasters who know all the openings. E.g, being told I'm using the wrong OpAmp because that particular device's temperature sensitivity isn't suitable for frequency-range I'm designing.
lambda 6y ago
My dad had a similar story about a bug report. This was back in the day when terminals still used paper rather than CRTs; actual teletypes. The bug report was something like "if I print a page feed on such and such terminal model, it will catch fire." The bug report form (which was paper in those days) had a few of the usual checkboxes and fields to fill in, including one for "is the problem reproducible?" That box was checked. Someone had actually gone to the effort of standing by their terminal with a fire extinguisher, running the same command and putting out the fire when it caught fire again. It turns out the problem was in some software that emulated page feeds by doing repeated line feeds. Some overflow or underflow bug caused an integer to wrap around, so instead of advancing a few lines to the next page, it would try to advance thousands or millions of lines. This particular terminal model was a new, high speed one, so it would try to advance the paper really fast. It was using those perforated strips at the sides, with the pegs that went into the holes. The pegs and holes weren't perfectly aligned, so the pegs would sometimes clip the holes a bit, causing some bits of paper to shred and get kicked up as dust. Along with a spark, I don't recall if it was from the motor or just static buildup caused by the friction of the paper, and the paper dust would ignite, causing the terminal to catch fire. I just love the image of whoever submitted that bug report, seeing the "is it reproducible" checkbox on the form, and standing by the terminal with a fire extinguisher to put it out while doing the same thing to make sure the problem was reproducible.
- segfaultbuserr 6y ago
  > so instead of advancing a few lines to the next page, it would try to advance thousands or millions of lines. This particular terminal model was a new, high speed one, so it would try to advance the paper really fast. [...] causing some bits of paper to shred and get kicked up as dust. Along with a spark [...] causing the terminal to catch fire. Great story! It really gives a new interpretation to the classic phase "halt and catch fire." The original interpretation was the apocalyptic myth that the CPU starts switching the bus really fast, overheats the bus drivers, and eventually ignites the electronics, which doesn't make much sense. Now we have a more realistic one. To be precise, not "halt and catch fire", but "CRLF and catch fire"? BTW, legends also say that the flammability of early printers was the origin of Unix's "lp0 on fire!" joke error message, which can still be seen in Linux for USB printers. /* * Get and print printer errors. */ static const char *usblp_messages[] = { "ok", "out of paper", "off-line", "on fire" };
satysin 6y ago
I got my start at Xerox working in their photocopier "QA" department. I say "QA" because it was more a factory setup than quality assurance office. Anyway we had test cases in folders with instructions on the front to configure the copier. Just the usual things like two-page scan to offset or collated with staples and a cover sheet, etc. Every now and then a "strange" case would be sent down of a certain job type that would cause the copier to break and hit "UM" or "unscheduled maintenance" exactly like this story. Had some interesting things pop up such as very creative ways people managed to bypass the currency detection to prevent printing (totally crap) currency copies or causing the printer to fail correctly printing the Machine Identification Code, even one scan to email job that caused a crash in the SMTP client so it would endlessly loop sending the same scan to the same email address every 5 minutes (300 second timeout) until you rebooted the machine. Fun times! This was actually how I got started with software development. I was interested in the copiers ESS (Electronic Sub System) which was a very basic Intel system running LynxOS essentially bolted to the back of the copier. I was chatting to some of the OS devs about RTOS and how it differed from a "normal" OS so learnt C with books they gave me and answered many of my questions. From there I managed to get a job up in the "real" QA department testing the printer drivers for Windows 9x, NT4, 2000 and the "soon to be released" Windows XP (or Whistler as it was known during development). I learnt how to use the Windows Debugger and about printer driver development which almost made me want to leave humanity and live in a cave somewhere in South America (: Luckily I got to move into the "nicer" world of C++/MFC and then .NET (anyone remember Managed Extensions for C++ aka C++/CLI?!). Like I said, Fun Times!
germanka 6y ago
Definitely a bug.
csours 6y ago
This brings me back to night shift production support: My infrastructure buddy came to my desk at about 1 AM and said, 'Hey, I emailed you a Word doc, could you print it for me?'. I opened it and hit print and my computer immediately went to blue screen. He said 'Yea, that happened to me too, I wondered if it was just my computer'. Nowadays, the suspicion meter is much more sensitive.
dheera 6y ago
Considering PostScript is Turing complete, it is still possible, and will always be possible, to make a PostScript file that takes 3 seconds to interpret, despite advances in CPU technology. Have modern laser printers solved the problem mechanically?
- Nextgrid 6y ago
  Computing power is cheap enough that I'd expect modern printers to just rasterize everything ahead of time before even starting the printing mechanism so this would eliminate this problem and make the logic simpler.
templarchamp 6y ago
If you keep jamming few times, it would heat up the printer which would change the absolute clock frequency and the printer would start working again - just a thought. As a software guy, "it works locally but when it is tested in production real world, it does not", lol
dang 6y ago
If curious see also 2014 https://news.ycombinator.com/item?id=8176464 https://news.ycombinator.com/item?id=8176464 Discussed (a bit) at the time: https://news.ycombinator.com/item?id=376067 https://news.ycombinator.com/item?id=376067
jamestnz 6y ago
This reminds me of a fun story about another unusual failure mode involving some supposedly valid (but highly specific) input content: Where listening to a certain podcast in your Mazda will reliably crash the car's stereo system. "The Roman Mars Mazda Virus" https://gimletmedia.com/shows/reply-all/brh8jm https://gimletmedia.com/shows/reply-all/brh8jm
- dws 6y ago
  That one was wonderful. C programmers would stop at the mention of '%' and go "aha!", but no!
lmilcin 6y ago
I have seen a bug where the device came from clients completely wiped. Mind, that this device has been distributed to millions of people. We would see couple hundred of these came every day, with completely clean flash. The interesting part was that nowhere in the code was any sequence that could clear the flash. After very lengthy and expensive bug hunting we have determined the cause. The flash protocol had an unlock sequence that required a bunch of magic operations before the write could be performed. Somebody, at some point decided to use "unlock bypass" function to disable this magic sequence, ie. allow writes without magic bypass sequence. This was done to improve write performance when the software was upgraded in the field (with customers potentially impatiently waiting for it to complete). Unbeknownst, there was a coupling between a remote part of the circuit and the lines between MCU and the flash that caused noise to be introduced to the communication. The unlock sequence was the only thing that prevented noise to be interpreted as valid operation. The probability was still very low of this happening, but with millions of devices in the field it was enough to get about 100 failures a day.
StreamBright 6y ago
My printer jams with anything I want to print. I do not even understand how we got here. There are 21313 different software installed on my system because of this printer and it is pretty useless.
WalterBright 6y ago
That's nothing. With Star Trek computers, type in something illogical and either the computer will explode or the poor keyboardist will get electrocuted and thrown across the bridge. I attribute much of the trepidation from early computer users to typing in the wrong thing to having watched Star Trek.
notjtrig 6y ago
My printer consistently prints the large 'F' logo in first class postage upside down.
elldoubleyew 6y ago
When I have had to build out SCM systems the bane of my existence always seems to be working with printers. I understand that we now have wonderful tools like cups that abstract a lot of the worst of it away from us, however even today minor discrepancies between models/drivers/environments have always given me headaches. Especially when trying to deploy systems in facilities with physically antiquated hardware. I swear I would encounter issues similar to the one described in this article every time I had to make a new deployment. I really wish this was an industry with a little more standardization.
Cmortoc 6y ago
I work on mobile games and my favorite QA report ever was, "game crashes when banana placed on screen, does not repro with other fruit" (it was actually a boring enough bug, the banana triggered a lot of touches all at once)
- PurpleFoxy 6y ago
  Did you actually get a banana to test it?
nudgeee 6y ago
Funniest bug report i had to deal with was when i was working in a company building iOS apps, and one of our testers filed a bug with the following reproduce steps: 1. Open the app 2. Rub the phone screen on your shoulder to clean the screen 3. The app will crash Perplexed, i tried it, and couldn't reproduce it. I went to the tester to ask about it, and even though it was hard to reproduce, he managed to crash the app a couple of times. Turns out there was a bug in handling multitouch gestures which was interacting with his clothing.
swiley 6y ago
Slightly related: Has anyone had their modern HDMI monitors change color (over the whole screen) when a small portion is displaying a heavily stippled image (like a classic MacOS screenshot?)

This is why I love bugs, especially the sporadic ones--often you grow in understanding of the underlying system or code as you solve them.

segfaultbuserr 6y ago
> especially the sporadic ones Yes, but there's only a fine line between "sporadic but solvable by analysis" and "too sporadic to be reproduced or solved..." Device driver problems are often from the second category... Nevertheless, it can get interesting if the problem is reproducible like the "500 miles email" or this printer.
- marcan_42 6y ago
  Nothing is too sporadic to be reproduced, you just haven't found how to reproduce it yet :) I once solved a once-in-a-petabyte bug at Google, when I was still working there. One of our systems ran in two data centers on the same data, and the cross-check was detecting a record offset mismatch but not a data mismatch (?!); retrying the process almost always fixed it. Long story short, there was a buffer overread in Google's bespoke zlib compressor that caused it to emit slightly less efficient compressed data (one or two bytes longer) nondeterministically, even though the data was perfectly well-formed. We were detecting compressed records being at different offsets, even though the uncompressed data matched. Another Google one involved doing a postmortem on a single instance of bad data ending up on a system. I traced that back purely from logs to having been caused by a kernel panic two systems upstream, that caused disk data to not be flushed while metadata was, so on reboot the system picked up garbage from the disk sectors in the file, which turned out to be well formed data from a different file and it trickled down the layers. Also, this one has made the rounds on HN a couple times (second time I fix a golang runtime heisenbug too; first one was also fun): https://marcan.st/2017/12/debugging-an-evil-go-runtime-bug/ https://marcan.st/2017/12/debugging-an-evil-go-runtime-bug/ And I recently fixed an OBS bug that has been randomly crashing audio for streamers for at least 3 years. I found a reliable repro that involved scripting monitoring mode toggles dozens of times per second; along the way I found and fixed 3 other bugs. That one was exacerbated by some of the OBS developers having built a rhetoric that those failures were caused by user configuration mistakes, because they often went away after changes - when what was actually going on was that users were blindly trying things to solve the problem, and sometimes stumbling on a somewhat stable workaround. (That one hasn't been reviewed/merged, still in the pipes). Then there was that bug in The Homebrew Channel that I introduced after adding freetype/TTF support... That was a heap corruption that was crashing things way later (embedded baremetal code, so no memory protection or debugging tools beyond remote gdb). Finding that one ended up involving dumping memory around the corrupted heap and following pointers to what eventually turned out to be graphics queue entries, then following texture pointers and teaching myself how to read swizzled character textures from hexdumps... I eventually found I was not allocating entries for space characters in strings, but was incrementing the index on spaces anyway. Lots of fun bug hunting stories...

ovi256 6y ago

Any change of state in a system with inertia should have hysteresis! In this case, a minimum duration before restarting would have done the trick.

segfaultbuserr 6y ago
> Any change of state in a system with inertia should have hysteresis! Or noise.

mikewarot 6y ago

Back in the days of Windows 98, the HP LaserJet caused no end of headaches for me. The problem that was introduced by the compiler they used to build the print driver. To compute the paper size, margins, etc., floating point operations were used... but by then the driver had allowed the libraries that emulated the floating point processor to be unloaded, to save memory. (Due to the optimizing compiler) Any operation that involved computing the page size would crash, only if the floating point library had idled out of memory. I ended up forcing the library to be loaded at boot on all machines, and then things were stable.

sharken 6y ago
No wonder that one of the DevOps success stories revolves around HP printer drivers and firmware: https://itrevolution.com/the-amazing-devops-transformation-of-the-hp-laserjet-firmware-team-gary-gruver/ https://itrevolution.com/the-amazing-devops-transformation-o...
cesarb 6y ago
One thing which I always considered a misdesign in Windows, is its tendency to run third-party code in-process. This includes COM objects, printer drivers, and many more. We had a problem similar to yours at a previous company. The software was written in a proprietary language which had its own runtime, and that runtime expected a particular value for the floating point control word. There are some third-party DLLs, including some printer drivers and Explorer extensions, which when loaded overwrite the floating point control word with their own notion of what should be the correct value (this behavior, as far as I could determine, is introduced by the compiler used to build these third-party DLLs). The unexpected value for the floating point control word would then lead to crashes, but these crashes would not happen immediately, and they would only happen if the user had done something which led to one of these third-party DLLs being loaded, like opening/saving a file (the open/save dialog in Windows loads Explorer extensions) or printing (which loads the printer driver). The solution was to, after every point in the code which opens a file chooser or prints anything, call into a small stub DLL we wrote, which had code to reset the floating point control word to what the runtime expected.
- 6y ago

Pxtl 6y ago

It's reassuring to know that printers have always been crap.

dylan604 6y ago
My 2 least favorite things about computers: Printing and Fonts.
- jnwatson 6y ago
  Importantly, it was the two things that early Macs got right. Other than price, a 1990 Mac/LaserWriter combo is superior than a modern equivalent.
  - thih9 6y ago
    Current status of apple printer setup seems right to me too. I'm happy that I can print wirelessly from macOS or iOS, without installing any drivers or configuring anything (other than the wifi password).
    - jamiek88 6y ago
      Yeah Apple typically get printing and audio right. Hi res too. Lots of things they aren’t so strong on but they have always been good at the above.
bananamerica 6y ago
https://www.wired.com/story/why-do-printers-still-suck/ https://www.wired.com/story/why-do-printers-still-suck/
coryrc 6y ago
Printers being crap led to the Free Software movement and the GPL
- gpvos 6y ago

avisser 6y ago

My favorite bug like this was me in my buddy's 2000 VW Jetta. The car had some electrical issues, but my buddy once remarked that the windows stopped working whenever he gave me a ride. I'm 6'4" and always slide the seat back whenever I get in a car. Turns out the circuitboard controlling the windows was under the passenger seat and had come loose. Moving the seat back caused a short, "breaking" the power windows. Turns out, the windows _did_ stop working when he gave me a ride. edit: "I have him" => "he gave me"

jacquesm 6y ago
I had a Citroen ID 19B which had a pretty neat quirk: the starter relay would only get power if you switched on the headlights. I don't know if that was on purpose or not (some primitive anti-theft device) but I left it in because it was actually a neat feature.
- ComputerGuru 6y ago
  It’s a legal requirement to have headlights on any time the car is in operation (day or night) in some territories and manufacturers had different ways of enforcing it. Some unconditionally turned on the headlights for models sold in those countries, others had special variants earmarked for export to those countries. It seems Citroen found a clever way that didn’t require too much fiddling to meet the legal requirement.
  - jacquesm 6y ago
    Not in 1968, the year that car was manufactured in, and none of the other ID's that I've seen had this particular quirk, it was either done by one of the (many) previous owners or a fluke. The wiring in those cars got very brittle over time and shorts were pretty common. That car came from France to NL (I imported it). Anyway, it was a fun gimmick and it helped to dissuade people from taking it for a ride. I never did find out if it was on purpose or accidental, that dash was not one to disassemble if you didn't need to.
    - 6y ago

Evidlo 6y ago

Here's a semi-related question that maybe somebody can answer. Why is it that despite listening on TCP 9100, some printers will just print the source code when you send them a PDF or Postscript file with netcat? Is there some special PJL command to put it into Postscript mode? Are there any networked printers around nowadays that don't support Postscript?

jagged-chisel 6y ago
“...despite listening...” Yes, there’s a protocol behind that port. Just like there’s one behind port 22, 80, 443... you can’t just dump http to 443 and get a useable document in return.
- tonyedgecombe 6y ago
  Actually that is exactly how the printers listened on port 9100. You open a connection and dump the raw print file. This "protocol" was introduced by HP with their JetDirect print servers.
- jolmg 6y ago
  > you can’t just dump http to 443 and get a useable document in return. Not on 443, but on 80 you basically would. You'd just have to skip the response headers.
buckminster 6y ago
A lot of printers, by default, function like an ancient teletype. They interpret incoming data as something like ASCII and print out the text. You need to send the correct escape sequence (i.e. a sequence of bytes starting with 0x1B) to switch into a more modern mode.
jschwartzi 6y ago
A lot of Postscript printers are configured to print source code when there’s a syntax error in the file. My 2017 HP laser printer has this as an optional feature.
- 6y ago

loloquwowndueo 6y ago

Lol “OpenOffice can’t print on Tuesdays” https://bugs.launchpad.net/ubuntu/+source/cupsys/+bug/255161/comments/28 https://bugs.launchpad.net/ubuntu/+source/cupsys/+bug/255161...

11thEarlOfMar 6y ago

Tell me if this is a bug... After driving a new car to work for 2 years, the traction control started kicking in on certain curves. I take a windy road to work and had never had traction control activate. When it kicked in, it pulsed the right front brake and slowed the car quickly, likely unnerving the guy tailgating me. This happened several times, both going to and from work. The road was perfectly dry, no snow or rain. Same turns at the same speed as always. The clues are was that my rear tires had worn and I replaced them a few days before. And, I drive a Tesla Model 3. It's a rear-wheel drive version and the front tires still had good tread so I didn't replace them. Why the right front tire? What I finally brained out, which you may have quickly realized, is that the diameters of the worn front and new rear tires were then slightly different. The front had a lower diameter than the rear and therefore turned at a slightly faster rate when driving. When I took a hard bank, the outside wheels also turned faster than the inside as they do in any turn. That increased the delta in spin rate the most between the outside worn front tire and the inside new rear tire. The traction control determined that the right front must be slipping and pulsed the brake on it. I replaced the front tires and everything went back to normal.

cowsandmilk 6y ago
Why not just move the tires from the front wheels to the back?
- bbddg 6y ago
  Yeah isn't this why you should rotate your tires regularly?
  - jhoechtl 6y ago
    You get downvoted but yes, that's the reason for swapping front to back and vice versa.
- crazyjncsu 6y ago
  Just so the laymen here don’t think this is some kind of hack, this is what is referred to as “rotating your tires”, which should sound familiar. The problem is that tires are often not rotatable due to staggered setups where the front and back tires are different sizes (very common) or due to directionality of tires (less common).

famousactress 6y ago

I may have told this story before (perhaps back in 2008 if this was posted then) -- but my first gig in tech was QA for Hewlett Packard testing printer/scanner/copier units. At some point I scanned a photo of my then-girlfriend and crashed the unit. Was able to do it repeatedly, provided I scanned the image in straight/cleanly. I didn't come across any other images that reproduced the issue, only this one photo. The underlying bug was a buffer overrun of some sort, was fixed before shipping -- and the image became a part of the durable regression test suite for that department.

ComputerGuru 6y ago
Not surprised there was a bug, but I am surprised it was patched (before or after shipping). I have an “enterprise grade” HP scanner (3500 F1) with buggy Twain drivers I had to manually disassemble and patch that will crash any time you try to duplex scan a single-sided sheet of paper. The blank page detection algorithm caused a null pointer dereference if two-sided mode was selected but the second side was elided. The bug reproduced 100% of the time with all available versions of their drivers in any application using the TWAIN rather than WIA or proprietary scanner interface. It was as simple as scanning any one-sided document in duplex mode. I escalated the issue to their engineers who kept insisting the issue was with my PC until the product was EOL’d just a year or two after production. I had other hardware issues pop up and ended up getting a Fujitsu for ADF document scanning and keeping the HP for scanning photos with the flatbed. Not many stand-alone hi-res dual color ADF/flatbed scanners out there.
- ryandrake 6y ago
  If you're in software, you probably know this, but your bug was likely ignored not because it couldn't be reproduced, but because the engineers on the project were prioritizing feature work for the next version of the scanner, and did not have any support from management for fixing bugs in an already-shipping product. Sadly, driver software quality tends to max out at the level where customers won't return the hardware for a refund, and then the company abandons it and works on the next shiny new device.

thewebcount 6y ago

I hit a similar issue in OpenGL on macOS about 10 years ago. We had a fairly complex set of draw calls. It worked fine on high-end machines, and was perfectly usable, if a bit slow, on mid-range machines, but on low-end iMacs with crappy Intel graphics cards, it would cause the machine to panic and reboot. It ended up being that the draw calls filled up the smaller command buffer on the low end card, and the driver would crash. In talking with the implementors, they explained the situation, so I asked, "How do I determine that I'm filling up the command buffer?" The answer was basically, "The size of the command buffer is an implementation detail and there's no way to query OpenGL for it or for the size of the commands you're adding." Gee, thanks! We ended just breaking up the draw calls into more smaller calls and it solved the problem, but this entire bug was such an ordeal. Thankfully we're moving away from OpenGL and things are a bit more sane in Metal. Had I not had access to the OpenGL team at Apple at the time, there's no way I would have been able to figure out what was going on. I'm not sure if I would have thought to do the same work in smaller batches.

marcan_42 6y ago
That's a driver bug.
cmg 6y ago
I wonder if this was related to a kernel panic bug I ran into in Chrome around the same time! Long story short, we were using a library (Modernizr) on our web product that checked for some kind of OpenGL support that we didn't actually need, and just running that code would trigger a driver bug. Disabling that fixed it!

quercusa 6y ago

From the comments, the 500-mile email: http://www.ibiblio.org/harris/500milemail.html http://www.ibiblio.org/harris/500milemail.html

SoSoRoCoCo 6y ago

For the last 5~7 years, thousands of software-only programmers have been discovering Arduino and Raspberry Pi, and have been learning about the real-world consequences of interfacing to hardware. I help out at a local hackerspace and the excitement of someone discovering a software/hardware interface bug is contagious. Here's an example: someone was trying to make a plant watering machine and was using a FET to control the DC motor. The wire to the FET was about 15 feet long so the slope from the poor GPIO was horrendous, and the motor would basically brownout because it couldn't overcome the initial stall di/dt. The idea that not all edges are perfect 90 degrees was eye opening to him.

myself248 6y ago
Oh, dang, yeah. Hackerspace here too, and software people colliding face-first with electrical reality is an endless source of learning, amusement, and project delays. Some quality time with an oscilloscope goes a long way, but still cannot overcome all stubbornness.
robocat 6y ago
Is that because many software programmers use “genetic coding” techniques where bugs are randomly fixed until the software works? Or were you mostly dealing with newbie programmers? It’s not like software doesn’t have similar failure modes to hardware, right?
- SoSoRoCoCo 6y ago
  I don't think it was... maybe? I've seen people hack circuits they way they hack code: keep trying small tweaks until it works. Then when they connect two circuits and it doesn't work, they go back to the fundamental RTFM. It could also be the skillset: circuits are more a creative art form, logic is more... logic. (But that doesn't always hold because I've encountered cache/CAM RTL designs that were beautiful.) Failure modes are different. A transistor can go bad if you accidentally drove it too hard, so now you might need to replace every part until the circuit works. And maybe one of the xtors in your box is bad. Or maybe a wire isn't making a good contact. Just the physical aspects alone without bringing an oscilloscope into the mix ("Why is my signal so noisy? Oh, it's the fluorescent lights, or the cheap USB hub noise in 400MHz spectrum, or the wall-wart...") can be maddening. Mostly I think the people I helped out seemed to forget that "digital" only exists inside the O/S, whereas the real world is analog, and real-world debuggers are more abstract than using, say, gdb or journalctl. But then I work with MY mentors, and they have an even larger set of debug skills based on extensive component knowledge that reminds me of chess grandmasters who know all the openings. E.g, being told I'm using the wrong OpAmp because that particular device's temperature sensitivity isn't suitable for frequency-range I'm designing.

lambda 6y ago

My dad had a similar story about a bug report. This was back in the day when terminals still used paper rather than CRTs; actual teletypes. The bug report was something like "if I print a page feed on such and such terminal model, it will catch fire." The bug report form (which was paper in those days) had a few of the usual checkboxes and fields to fill in, including one for "is the problem reproducible?" That box was checked. Someone had actually gone to the effort of standing by their terminal with a fire extinguisher, running the same command and putting out the fire when it caught fire again. It turns out the problem was in some software that emulated page feeds by doing repeated line feeds. Some overflow or underflow bug caused an integer to wrap around, so instead of advancing a few lines to the next page, it would try to advance thousands or millions of lines. This particular terminal model was a new, high speed one, so it would try to advance the paper really fast. It was using those perforated strips at the sides, with the pegs that went into the holes. The pegs and holes weren't perfectly aligned, so the pegs would sometimes clip the holes a bit, causing some bits of paper to shred and get kicked up as dust. Along with a spark, I don't recall if it was from the motor or just static buildup caused by the friction of the paper, and the paper dust would ignite, causing the terminal to catch fire. I just love the image of whoever submitted that bug report, seeing the "is it reproducible" checkbox on the form, and standing by the terminal with a fire extinguisher to put it out while doing the same thing to make sure the problem was reproducible.

segfaultbuserr 6y ago
> so instead of advancing a few lines to the next page, it would try to advance thousands or millions of lines. This particular terminal model was a new, high speed one, so it would try to advance the paper really fast. [...] causing some bits of paper to shred and get kicked up as dust. Along with a spark [...] causing the terminal to catch fire. Great story! It really gives a new interpretation to the classic phase "halt and catch fire." The original interpretation was the apocalyptic myth that the CPU starts switching the bus really fast, overheats the bus drivers, and eventually ignites the electronics, which doesn't make much sense. Now we have a more realistic one. To be precise, not "halt and catch fire", but "CRLF and catch fire"? BTW, legends also say that the flammability of early printers was the origin of Unix's "lp0 on fire!" joke error message, which can still be seen in Linux for USB printers. /* * Get and print printer errors. */ static const char *usblp_messages[] = { "ok", "out of paper", "off-line", "on fire" };

satysin 6y ago

I got my start at Xerox working in their photocopier "QA" department. I say "QA" because it was more a factory setup than quality assurance office. Anyway we had test cases in folders with instructions on the front to configure the copier. Just the usual things like two-page scan to offset or collated with staples and a cover sheet, etc. Every now and then a "strange" case would be sent down of a certain job type that would cause the copier to break and hit "UM" or "unscheduled maintenance" exactly like this story. Had some interesting things pop up such as very creative ways people managed to bypass the currency detection to prevent printing (totally crap) currency copies or causing the printer to fail correctly printing the Machine Identification Code, even one scan to email job that caused a crash in the SMTP client so it would endlessly loop sending the same scan to the same email address every 5 minutes (300 second timeout) until you rebooted the machine. Fun times! This was actually how I got started with software development. I was interested in the copiers ESS (Electronic Sub System) which was a very basic Intel system running LynxOS essentially bolted to the back of the copier. I was chatting to some of the OS devs about RTOS and how it differed from a "normal" OS so learnt C with books they gave me and answered many of my questions. From there I managed to get a job up in the "real" QA department testing the printer drivers for Windows 9x, NT4, 2000 and the "soon to be released" Windows XP (or Whistler as it was known during development). I learnt how to use the Windows Debugger and about printer driver development which almost made me want to leave humanity and live in a cave somewhere in South America (: Luckily I got to move into the "nicer" world of C++/MFC and then .NET (anyone remember Managed Extensions for C++ aka C++/CLI?!). Like I said, Fun Times!

germanka 6y ago

Definitely a bug.

csours 6y ago

This brings me back to night shift production support: My infrastructure buddy came to my desk at about 1 AM and said, 'Hey, I emailed you a Word doc, could you print it for me?'. I opened it and hit print and my computer immediately went to blue screen. He said 'Yea, that happened to me too, I wondered if it was just my computer'. Nowadays, the suspicion meter is much more sensitive.

dheera 6y ago

Considering PostScript is Turing complete, it is still possible, and will always be possible, to make a PostScript file that takes 3 seconds to interpret, despite advances in CPU technology. Have modern laser printers solved the problem mechanically?

Nextgrid 6y ago
Computing power is cheap enough that I'd expect modern printers to just rasterize everything ahead of time before even starting the printing mechanism so this would eliminate this problem and make the logic simpler.

templarchamp 6y ago

If you keep jamming few times, it would heat up the printer which would change the absolute clock frequency and the printer would start working again - just a thought. As a software guy, "it works locally but when it is tested in production real world, it does not", lol

dang 6y ago

If curious see also 2014 https://news.ycombinator.com/item?id=8176464 https://news.ycombinator.com/item?id=8176464 Discussed (a bit) at the time: https://news.ycombinator.com/item?id=376067 https://news.ycombinator.com/item?id=376067

jamestnz 6y ago

This reminds me of a fun story about another unusual failure mode involving some supposedly valid (but highly specific) input content: Where listening to a certain podcast in your Mazda will reliably crash the car's stereo system. "The Roman Mars Mazda Virus" https://gimletmedia.com/shows/reply-all/brh8jm https://gimletmedia.com/shows/reply-all/brh8jm

dws 6y ago
That one was wonderful. C programmers would stop at the mention of '%' and go "aha!", but no!

lmilcin 6y ago

I have seen a bug where the device came from clients completely wiped. Mind, that this device has been distributed to millions of people. We would see couple hundred of these came every day, with completely clean flash. The interesting part was that nowhere in the code was any sequence that could clear the flash. After very lengthy and expensive bug hunting we have determined the cause. The flash protocol had an unlock sequence that required a bunch of magic operations before the write could be performed. Somebody, at some point decided to use "unlock bypass" function to disable this magic sequence, ie. allow writes without magic bypass sequence. This was done to improve write performance when the software was upgraded in the field (with customers potentially impatiently waiting for it to complete). Unbeknownst, there was a coupling between a remote part of the circuit and the lines between MCU and the flash that caused noise to be introduced to the communication. The unlock sequence was the only thing that prevented noise to be interpreted as valid operation. The probability was still very low of this happening, but with millions of devices in the field it was enough to get about 100 failures a day.

StreamBright 6y ago

My printer jams with anything I want to print. I do not even understand how we got here. There are 21313 different software installed on my system because of this printer and it is pretty useless.

WalterBright 6y ago

That's nothing. With Star Trek computers, type in something illogical and either the computer will explode or the poor keyboardist will get electrocuted and thrown across the bridge. I attribute much of the trepidation from early computer users to typing in the wrong thing to having watched Star Trek.

notjtrig 6y ago

My printer consistently prints the large 'F' logo in first class postage upside down.

elldoubleyew 6y ago

When I have had to build out SCM systems the bane of my existence always seems to be working with printers. I understand that we now have wonderful tools like cups that abstract a lot of the worst of it away from us, however even today minor discrepancies between models/drivers/environments have always given me headaches. Especially when trying to deploy systems in facilities with physically antiquated hardware. I swear I would encounter issues similar to the one described in this article every time I had to make a new deployment. I really wish this was an industry with a little more standardization.

Cmortoc 6y ago

I work on mobile games and my favorite QA report ever was, "game crashes when banana placed on screen, does not repro with other fruit" (it was actually a boring enough bug, the banana triggered a lot of touches all at once)

PurpleFoxy 6y ago
Did you actually get a banana to test it?

nudgeee 6y ago

Funniest bug report i had to deal with was when i was working in a company building iOS apps, and one of our testers filed a bug with the following reproduce steps: 1. Open the app 2. Rub the phone screen on your shoulder to clean the screen 3. The app will crash Perplexed, i tried it, and couldn't reproduce it. I went to the tester to ask about it, and even though it was hard to reproduce, he managed to crash the app a couple of times. Turns out there was a bug in handling multitouch gestures which was interacting with his clothing.

swiley 6y ago

Slightly related: Has anyone had their modern HDMI monitors change color (over the whole screen) when a small portion is displaying a heavily stippled image (like a classic MacOS screenshot?)