6 ms·

The weirdest QNX bug I've encountered (2021)

175 points by fanf2 2y ago

torginus 2y ago
Honestly this kinda shows me that no matter what degree of robustness we design into our systems (null saftety, memory safety, thread safety etc.), some types of system breaking bugs are unavoidable (such as DOSing the system by calling a system API function in an infinite loop), and are often impossible to distingushing from desired behavior.
- smaudet 2y ago
  How about no infinite loops (as a start)? Unless you are the kernel, and you can demonstrate that your loop is "safe" via some set of static analysis.
  - banish-m4 2y ago
    Achievable with formal verification and soft- and hard-realtime worst case timing validation. It's not impossible but also not easy. It requires significant engineering investment.
  - Tsarbomb 2y ago
    Are you suggesting a general solution to the halting problem?
    - jamesmunns 2y ago
      The usual solution, including in safety critical systems, is to give the judge of the halting problem a stopwatch and a gun. For example in an embedded system: a watchdog timer that you don't service during the execution of some context. If you fail to complete your task within the time, the kernel or entire system is rebooted. For example in a VM-like system, you give the code some amount of "fuel" or "budget", if it exceeds that budget, the process/tasklet/whatever is terminated by the VM. It's not a general solution to the halting problem, but a practical one.
    - fragmede 2y ago
      funnily enough, ChatGPT can examine code and can say if it will halt or not, in some small cases.
  - torginus 2y ago
    Infinite loops are not that harmful by themselves if they only spin the CPU. The scheduler can schedule other work on the core if the process' timeslice runs out, or it results in a frozen application if said infinite loop happens on an event loop thread. Neither are pleasant, but they don't compromise system integrity and as such are not substantially different from other kinds of crash bugs.
    - smaudet 2y ago
      > Infinite loops are not that harmful by themselves if they only spin the CPU Categorically that's the exception, not the norm. For always-on systems, "by themselves" is pretty critical, as they can result in DOS style bugs/attacks as illustrated in the parent article. The infinite loop is most useful e.g. for the scheduler or other event loop code, which by definition this style of code does a lot more than "only" spin the CPU. For systems capable of long-term power saving operation - the system can go dormant (either no power draw or very little). An infinite loop can be the difference between weeks of power off a battery, or days/hours.
- banish-m4 2y ago
  That's shrugging off inferior processes and substandard work. Bugs are unnecessary because there is a finite amount of code and they can all be eliminated if the correct eyeballs spend sufficient time reviewing, testing, and simplifying behavior to focus on robust reliability and correctness. Breaking changes can be allowed in dev packs with semver release notes. There are no excuses for sloppy engineering.
- KerrAvon 2y ago
  Being able to DOS the system by calling an API should be considered a bug in the system. There are mitigation strategies for such things. If you can’t (for mitigation purposes) distinguish a DOS from desired behavior, there’s something very wrong with your architecture. edit: added parenthetical
Animats 2y ago
"At this point, an intermezzo with some QNX history is in order. A bit more than a decade ago, the QNX source code was available to the public. Back then, QNX had a vibrant open source community. People would experiment with the kernel, write various useful utilities and help each other in forums. QNX even had a fully featured Desktop GUI, ran Firefox and was self-hosting, so you could develop for QNX right on QNX itself with full IDE and compiler support. It was beautiful." "Then QNX was bought, source code access was revoked and the community largely withered away. Questions were increasingly asked via private support tickets directly to QNX, locked away from the public. QNX know-how becomes harder and harder to acquire, open source software for modern QNX releases is essentially non-existent and the driver situation is a catastrophe. The QNX kernel is the most beautiful and interesting kernel I have ever had the pleasure of working with, but it lies in the shackles of corporate ownership." It's sad. QNX was originally an independent company. During that period, anyone could get a free copy of QNX for personal use. It wasn't open source, but it was available. It's POSIX-compatible, so it was a supported target for Gnu, Firefox, and Eclipse. We used QNX for our DARPA Grand Challenge vehicle in 2003-2005, and all that code was developed on desktop QNX. Then QNX was acquired by Harmon, the successor to Harmon-Kardon, which once made home audio components and pivoted to car audio. They were thinking car infotainment. Harmon didn't really know what to do with an operating system, especially since the big market was systems for industrial control and point of sale. So eventually they opened the source. Then QNX was acquired by Blackberry, the early smartphone company. They closed the source, very suddenly. They even killed off the free version for personal and educational use. So all third party open source development stopped. Blackberry eventually shipped a phone that ran QNX, but they were not powerful enough as a company to keep a third phone standard going. So Blackberry went to Android. Blackberry killed off the self-hosted desktop environment, and users now had to cross-compile from Windows. And QNX became more of a niche product than ever.
- deleted 2y ago
  [deleted]
- akira2501 2y ago
  > but they were not powerful enough as a company to keep a third phone standard going. They absolutely were, which is the tragedy of the whole thing, people absolutely loved their products and strongly preferred them to everything else on the market. Instead of recognizing the game changer that the iPhone was they slept on the market and didn't do much to bring big touch screens and rich internet applications to their platform. It was a slow and agonizing death.
arsome 2y ago
I actually ran across this issue myself, SIGQUIT'd the process, loaded it into a debugger and found the exact same problem. I can confirm the problem still exists on QNX 7.1. Fortunately we were moving off it, so I didn't think much more about it, but glad someone wrote it up.
banish-m4 2y ago
What I like about seL4 (although not a complete embedded dev platform) is formally-verification. QNX might have EAL4 in some configurations, but like most every other operating system on the planet, they haven't bothered to up their game by formally verifying it for correctness. This is a shame and entirely preventable with greater attention to testing and verification.
- kragen 2y ago
  what sel4 shows is that it's entirely preventable with a rewrite from scratch by a team of formal-methods ph.d.s over many years, if they invent a design that allows the kernel to only be a few thousand lines of code so that the titanic effort of formal verification becomes feasible, barely. it's not something you can do with a codebase of millions of lines of code or a codebase that wasn't written from scratch with formal verification in mind. yet, anyway
the_panopticon 2y ago
I recall trying to debug a crash in QNX during the mid-90's. I was impressed by the svelte OS that could load from a 3.5" floppy. The failure scenario was coincident with one of my first tasks as a BIOS engineer and it entailed adding some custom error logging in System Management Mode (SMM). Luckily for me it turned out that I had forgotten to save/restore certain general purpose registers around my SMM logic. Fun times. SMM is pretty good at 'breaking' operating systems :)
redtriumph 2y ago
https://news.ycombinator.com/item?id=29207216 https://news.ycombinator.com/item?id=29207216
tfrutuoso 2y ago
Great article, but the comments section of that blog is pure cancer. Jeez.
ragnot 2y ago
Every developer I know (myself included) that has worked with QNX has a story about some insane bug that took significant effort to uncover. At this point, I would say the only reason one should look at QNX is for cost since it is pretty cheap. The low jitter on context-switching to the highest priority thread is a nice thing but the dev process is absolute garbage.
- nubinetwork 2y ago
  When you can tell qnx to give you a root shell as an unprivileged user, ps being slow doesn't surprise me... https://www.juliandunn.net/2006/08/21/on-hacking-the-unisys-icon/ https://www.juliandunn.net/2006/08/21/on-hacking-the-unisys-...
- fargle 2y ago
  yep. it's really trash. used it 20-25 years ago when they were just introducing "neutrino" vs. the classic QNX4. the former had a good rep with auto and medical usage. - very bad port of GCC was buggy and generated bad code. it was the result of some idiot blindly haphazardly applying a ton random incoherent patches to try to get it to build instead of porting it properly. (to be clear mainline GCC at that time was fine; we re-ported it ourselves instead) - and, of course, they used their own faulty compiler to build their libraries and services ;) causing unknown carnage waiting to be discovered. - malloc broken (heavy use under multiple threads causes heap corruption). replaced with dlmalloc. - serial port driver broken. rewrote a new one. - intel network card driver crashes. replaced hardware with 3com to survive. - certain math library functions broken (iirc, fmod). replaced. and so on and on. it doesn't matter what certification your RTOS (or whatever) has. if you cannot examine the source, rebuild it, etc. (oss or private source available) - it cannot be trusted. this was one of the worse examples, but it's always like this with "proprietary" OS/toolkits.
bxparks 2y ago
I counted 417 comments on that page and scrolled through a few dozen. Every one of them was spam. That's pretty much the internet these days isn't it. Other than that, the blog post was very interesting, I learned a bit of history of QNX, and concluded that I should avoid it.
- ronsor 2y ago
  The spambots are literally having conversations with each other.
- OnionBlender 2y ago
  Is it though? I don't remember the last time I saw so many spam messages. Most sites I visit do a better job of preventing or removing spam.
nrclark 2y ago
QNX really needs to modernize if they want to survive. Their tooling ecosystem is stuck in 2008, and their kernel's performance is pretty low. IIRC, the kernel itself is also single-threaded, and can't take advantage of multiple CPUs (even if tasks can be SMP scheduled). Their moat is supposedly their ASIL certification, but I see that value shrinking more and more over time for the following reasons: 1. If your product has a software-related failure, customers won't care about all of your certifications. Only the end product. 2. I'm not convinced that the QNX kernel is less buggy than the Linux kernel. Also, most failures don't tend to be kernel related.
- foundry27 2y ago
  What kind of tooling modernization would even make a difference? At least SDP 8.0 overhauled the kernel to not be locked to a single thread anymore, which is nice IMO
- agustamir 2y ago
  > kernel's performance is pretty low Can you elaborate on this? How is it "low"
  - NoahKAndrews 2y ago
    They did, in the next sentence
- Beretta_Vexee 2y ago
  There is a very large installed base of machines running on QNX, including medical equipment, radio communications, railway switches, automotive modules, etc. Most manufacturers are upgrading in small steps and absolutely do not want to start from scratch with their software layer. Most manufacturers are upgrading in small steps and have no desire to start from scratch with their software layer. At best, they will add a touch screen, an arm processor and an Android system, but only for the interface. The critical parts will remain under QNX. The same is true for VxWorks, a large proportion of industrial PLCs, electrical networks, water and sewage systems depend on VxWorks. QNX and VxWorks are omnipresent but invisible systems.
lfkdev 2y ago
What is going on with the comment section on this post?
- ralferoo 2y ago
  I actually thought some of the comments were funny. Especially the one about the crab in the shell! No idea why they thought it was related to QNX, but an insight into the mind of spammers nonetheless.
  - exikyut 2y ago
    That particular snippet was posted four times if you ^F. It's really interesting to see the current "state of the art" in terms of the types of bots that get past the particular CAPTCHA implementation this site uses. It's a very rudimentary type of CAPTCHA, the kind that anything developed within the past 5 years would probably get past with at least 30% accuracy (logarithmically skyrocketing to >90% within the past ~2 years). So the post quality is somewhat distributed across a spectrum - on one end, dumb CAPTCHA OCR/processing <=> "slightly better than Markov chain model replication", and at the other end, clearly more sophisticated systems that more easily pass the CAPTCHA and generate more interesting posts. What's curious is that 90% of the comments are rudimentary. There are very few interesting spam posts. I'm trying to figure out what to make of this. I'm picturing some majority of utterly outdated spambot infra, still out there, scanning the Web for WordPress/XSS-level stuff, and finding success on blogs like these... and that these old bots are the only systems of their kind out there, because all the spammers collectively gave up with reCAPTCHA and CloudFlare protecting almost all meaningful concerns, with moderation following not too far behind. Kind of makes sense. But it's really depressing to compare these old clunky bots that are kind of cute (in a way) to the upgraded versions - the current-era tech, that get past moderation... and effectively pass the Turing test :'(
    - ralferoo 2y ago
      I hadn't noticed multiple copies of the crab one, but had of various others. To me the fact there are so many duplicates makes me think those texts, if not written by humans, were selected by humans. If it was truly AI-generated, I'd expect a random seed as part of its input, and unless there was very little entropy, I'm not sure it would chance upon the same exact formulations over and over. Maybe they hadn't tested the randomness aspect well in the training and it'd learned not to attach much weight to that beyond the first word or two.
deleted 2y ago
[deleted]

175 points by fanf2 2y ago

Honestly this kinda shows me that no matter what degree of robustness we design into our systems (null saftety, memory safety, thread safety etc.), some types of system breaking bugs are unavoidable (such as DOSing the system by calling a system API function in an infinite loop), and are often impossible to distingushing from desired behavior.

smaudet 2y ago
How about no infinite loops (as a start)? Unless you are the kernel, and you can demonstrate that your loop is "safe" via some set of static analysis.
- banish-m4 2y ago
  Achievable with formal verification and soft- and hard-realtime worst case timing validation. It's not impossible but also not easy. It requires significant engineering investment.
- Tsarbomb 2y ago
  Are you suggesting a general solution to the halting problem?
  - jamesmunns 2y ago
    The usual solution, including in safety critical systems, is to give the judge of the halting problem a stopwatch and a gun. For example in an embedded system: a watchdog timer that you don't service during the execution of some context. If you fail to complete your task within the time, the kernel or entire system is rebooted. For example in a VM-like system, you give the code some amount of "fuel" or "budget", if it exceeds that budget, the process/tasklet/whatever is terminated by the VM. It's not a general solution to the halting problem, but a practical one.
  - fragmede 2y ago
    funnily enough, ChatGPT can examine code and can say if it will halt or not, in some small cases.
- torginus 2y ago
  Infinite loops are not that harmful by themselves if they only spin the CPU. The scheduler can schedule other work on the core if the process' timeslice runs out, or it results in a frozen application if said infinite loop happens on an event loop thread. Neither are pleasant, but they don't compromise system integrity and as such are not substantially different from other kinds of crash bugs.
  - smaudet 2y ago
    > Infinite loops are not that harmful by themselves if they only spin the CPU Categorically that's the exception, not the norm. For always-on systems, "by themselves" is pretty critical, as they can result in DOS style bugs/attacks as illustrated in the parent article. The infinite loop is most useful e.g. for the scheduler or other event loop code, which by definition this style of code does a lot more than "only" spin the CPU. For systems capable of long-term power saving operation - the system can go dormant (either no power draw or very little). An infinite loop can be the difference between weeks of power off a battery, or days/hours.
banish-m4 2y ago
That's shrugging off inferior processes and substandard work. Bugs are unnecessary because there is a finite amount of code and they can all be eliminated if the correct eyeballs spend sufficient time reviewing, testing, and simplifying behavior to focus on robust reliability and correctness. Breaking changes can be allowed in dev packs with semver release notes. There are no excuses for sloppy engineering.
KerrAvon 2y ago
Being able to DOS the system by calling an API should be considered a bug in the system. There are mitigation strategies for such things. If you can’t (for mitigation purposes) distinguish a DOS from desired behavior, there’s something very wrong with your architecture. edit: added parenthetical

Animats 2y ago

"At this point, an intermezzo with some QNX history is in order. A bit more than a decade ago, the QNX source code was available to the public. Back then, QNX had a vibrant open source community. People would experiment with the kernel, write various useful utilities and help each other in forums. QNX even had a fully featured Desktop GUI, ran Firefox and was self-hosting, so you could develop for QNX right on QNX itself with full IDE and compiler support. It was beautiful." "Then QNX was bought, source code access was revoked and the community largely withered away. Questions were increasingly asked via private support tickets directly to QNX, locked away from the public. QNX know-how becomes harder and harder to acquire, open source software for modern QNX releases is essentially non-existent and the driver situation is a catastrophe. The QNX kernel is the most beautiful and interesting kernel I have ever had the pleasure of working with, but it lies in the shackles of corporate ownership." It's sad. QNX was originally an independent company. During that period, anyone could get a free copy of QNX for personal use. It wasn't open source, but it was available. It's POSIX-compatible, so it was a supported target for Gnu, Firefox, and Eclipse. We used QNX for our DARPA Grand Challenge vehicle in 2003-2005, and all that code was developed on desktop QNX. Then QNX was acquired by Harmon, the successor to Harmon-Kardon, which once made home audio components and pivoted to car audio. They were thinking car infotainment. Harmon didn't really know what to do with an operating system, especially since the big market was systems for industrial control and point of sale. So eventually they opened the source. Then QNX was acquired by Blackberry, the early smartphone company. They closed the source, very suddenly. They even killed off the free version for personal and educational use. So all third party open source development stopped. Blackberry eventually shipped a phone that ran QNX, but they were not powerful enough as a company to keep a third phone standard going. So Blackberry went to Android. Blackberry killed off the self-hosted desktop environment, and users now had to cross-compile from Windows. And QNX became more of a niche product than ever.

deleted 2y ago
[deleted]
akira2501 2y ago
> but they were not powerful enough as a company to keep a third phone standard going. They absolutely were, which is the tragedy of the whole thing, people absolutely loved their products and strongly preferred them to everything else on the market. Instead of recognizing the game changer that the iPhone was they slept on the market and didn't do much to bring big touch screens and rich internet applications to their platform. It was a slow and agonizing death.

arsome 2y ago

I actually ran across this issue myself, SIGQUIT'd the process, loaded it into a debugger and found the exact same problem. I can confirm the problem still exists on QNX 7.1. Fortunately we were moving off it, so I didn't think much more about it, but glad someone wrote it up.

banish-m4 2y ago

What I like about seL4 (although not a complete embedded dev platform) is formally-verification. QNX might have EAL4 in some configurations, but like most every other operating system on the planet, they haven't bothered to up their game by formally verifying it for correctness. This is a shame and entirely preventable with greater attention to testing and verification.

kragen 2y ago
what sel4 shows is that it's entirely preventable with a rewrite from scratch by a team of formal-methods ph.d.s over many years, if they invent a design that allows the kernel to only be a few thousand lines of code so that the titanic effort of formal verification becomes feasible, barely. it's not something you can do with a codebase of millions of lines of code or a codebase that wasn't written from scratch with formal verification in mind. yet, anyway

the_panopticon 2y ago

I recall trying to debug a crash in QNX during the mid-90's. I was impressed by the svelte OS that could load from a 3.5" floppy. The failure scenario was coincident with one of my first tasks as a BIOS engineer and it entailed adding some custom error logging in System Management Mode (SMM). Luckily for me it turned out that I had forgotten to save/restore certain general purpose registers around my SMM logic. Fun times. SMM is pretty good at 'breaking' operating systems :)

redtriumph 2y ago

https://news.ycombinator.com/item?id=29207216 https://news.ycombinator.com/item?id=29207216

tfrutuoso 2y ago

Great article, but the comments section of that blog is pure cancer. Jeez.

ragnot 2y ago

Every developer I know (myself included) that has worked with QNX has a story about some insane bug that took significant effort to uncover. At this point, I would say the only reason one should look at QNX is for cost since it is pretty cheap. The low jitter on context-switching to the highest priority thread is a nice thing but the dev process is absolute garbage.

nubinetwork 2y ago
When you can tell qnx to give you a root shell as an unprivileged user, ps being slow doesn't surprise me... https://www.juliandunn.net/2006/08/21/on-hacking-the-unisys-icon/ https://www.juliandunn.net/2006/08/21/on-hacking-the-unisys-...
fargle 2y ago
yep. it's really trash. used it 20-25 years ago when they were just introducing "neutrino" vs. the classic QNX4. the former had a good rep with auto and medical usage. - very bad port of GCC was buggy and generated bad code. it was the result of some idiot blindly haphazardly applying a ton random incoherent patches to try to get it to build instead of porting it properly. (to be clear mainline GCC at that time was fine; we re-ported it ourselves instead) - and, of course, they used their own faulty compiler to build their libraries and services ;) causing unknown carnage waiting to be discovered. - malloc broken (heavy use under multiple threads causes heap corruption). replaced with dlmalloc. - serial port driver broken. rewrote a new one. - intel network card driver crashes. replaced hardware with 3com to survive. - certain math library functions broken (iirc, fmod). replaced. and so on and on. it doesn't matter what certification your RTOS (or whatever) has. if you cannot examine the source, rebuild it, etc. (oss or private source available) - it cannot be trusted. this was one of the worse examples, but it's always like this with "proprietary" OS/toolkits.

bxparks 2y ago

I counted 417 comments on that page and scrolled through a few dozen. Every one of them was spam. That's pretty much the internet these days isn't it. Other than that, the blog post was very interesting, I learned a bit of history of QNX, and concluded that I should avoid it.

ronsor 2y ago
The spambots are literally having conversations with each other.
OnionBlender 2y ago
Is it though? I don't remember the last time I saw so many spam messages. Most sites I visit do a better job of preventing or removing spam.

nrclark 2y ago

QNX really needs to modernize if they want to survive. Their tooling ecosystem is stuck in 2008, and their kernel's performance is pretty low. IIRC, the kernel itself is also single-threaded, and can't take advantage of multiple CPUs (even if tasks can be SMP scheduled). Their moat is supposedly their ASIL certification, but I see that value shrinking more and more over time for the following reasons: 1. If your product has a software-related failure, customers won't care about all of your certifications. Only the end product. 2. I'm not convinced that the QNX kernel is less buggy than the Linux kernel. Also, most failures don't tend to be kernel related.

foundry27 2y ago
What kind of tooling modernization would even make a difference? At least SDP 8.0 overhauled the kernel to not be locked to a single thread anymore, which is nice IMO
agustamir 2y ago
> kernel's performance is pretty low Can you elaborate on this? How is it "low"
- NoahKAndrews 2y ago
  They did, in the next sentence
Beretta_Vexee 2y ago
There is a very large installed base of machines running on QNX, including medical equipment, radio communications, railway switches, automotive modules, etc. Most manufacturers are upgrading in small steps and absolutely do not want to start from scratch with their software layer. Most manufacturers are upgrading in small steps and have no desire to start from scratch with their software layer. At best, they will add a touch screen, an arm processor and an Android system, but only for the interface. The critical parts will remain under QNX. The same is true for VxWorks, a large proportion of industrial PLCs, electrical networks, water and sewage systems depend on VxWorks. QNX and VxWorks are omnipresent but invisible systems.

lfkdev 2y ago

What is going on with the comment section on this post?

ralferoo 2y ago
I actually thought some of the comments were funny. Especially the one about the crab in the shell! No idea why they thought it was related to QNX, but an insight into the mind of spammers nonetheless.
- exikyut 2y ago
  That particular snippet was posted four times if you ^F. It's really interesting to see the current "state of the art" in terms of the types of bots that get past the particular CAPTCHA implementation this site uses. It's a very rudimentary type of CAPTCHA, the kind that anything developed within the past 5 years would probably get past with at least 30% accuracy (logarithmically skyrocketing to >90% within the past ~2 years). So the post quality is somewhat distributed across a spectrum - on one end, dumb CAPTCHA OCR/processing <=> "slightly better than Markov chain model replication", and at the other end, clearly more sophisticated systems that more easily pass the CAPTCHA and generate more interesting posts. What's curious is that 90% of the comments are rudimentary. There are very few interesting spam posts. I'm trying to figure out what to make of this. I'm picturing some majority of utterly outdated spambot infra, still out there, scanning the Web for WordPress/XSS-level stuff, and finding success on blogs like these... and that these old bots are the only systems of their kind out there, because all the spammers collectively gave up with reCAPTCHA and CloudFlare protecting almost all meaningful concerns, with moderation following not too far behind. Kind of makes sense. But it's really depressing to compare these old clunky bots that are kind of cute (in a way) to the upgraded versions - the current-era tech, that get past moderation... and effectively pass the Turing test :'(
  - ralferoo 2y ago
    I hadn't noticed multiple copies of the crab one, but had of various others. To me the fact there are so many duplicates makes me think those texts, if not written by humans, were selected by humans. If it was truly AI-generated, I'd expect a random seed as part of its input, and unless there was very little entropy, I'm not sure it would chance upon the same exact formulations over and over. Maybe they hadn't tested the randomness aspect well in the training and it'd learned not to attach much weight to that beyond the first word or two.

deleted 2y ago

[deleted]