9 ms·

Toyota unintended acceleration and the big bowl of "spaghetti" code (2013)

35 points by SoKamil 10mo ago

gnabgib 10mo ago
Popular in 2015: (96 points, 106 comments) https://news.ycombinator.com/item?id=10437117 https://news.ycombinator.com/item?id=10437117 (152 points, 145 comments) https://news.ycombinator.com/item?id=9643204 https://news.ycombinator.com/item?id=9643204
supahfly_remix 10mo ago
Does anyone know where one could obtain the firmware for this? It might be interesting to reverse engineer.
- altairprime 10mo ago
  It’s available in various archives of the Toyota TechStream pre-2024 editions, in some sort of weird encrypted file format that can be trivially decrypted; I haven’t tried myself but the ECU I work with isn’t encrypted in-vehicle. I’ve spent five or six years in Ghidra with various hybrid Subaru-Toyota ECUs from 2013-2020 and I wonder what kind of source control practices result in the massive function spaghetti that must have produced in this SH-2A code; I can see where Toyota bolted their direct injection runloop into Subaru’s. So, yeah, if you’re curious, the firmware’s out there, if you’ve got a few years to spare and an absolutely ridiculous amount of patience (and a solid grasp of CAN bus messaging protocols, which you’ll need to identify code blocks and variables and such!) “The Car Hacker’s Handbook” may be of interest as a first step review, but honestly I just dove in with Ghidra and just .. didn’t ever stop. YMMV :)
  - supahfly_remix 10mo ago
    lol, thanks for the reply. Yes, I was thinking of looking at it, but your comment dissuaded me. Out of curiosity, did you try running ECU code through an LLM and asking questions of it, and, if so, how did that work out?
    - altairprime 10mo ago
      No.
stackghost 10mo ago
Safety Research Systems, the author of TFA, is a for-profit company whose income is based on lawsuits. Make of that what you will.
- throwaway81523 10mo ago
  Philip Koopman was an expert witness against Toyota (which is a bit questionable to me) and he has some stuff on his website about the case too. https://betterembsw.blogspot.com/search/label/Toyota%20UA https://betterembsw.blogspot.com/search/label/Toyota%20UA
LanceH 10mo ago
Ah yes, where Toyota was found guilty of not being a US company. The only thing they did in the recall was the same floor mat anchor as so many other cases. "NASA engineers found no electronic flaws in Toyota vehicles capable of producing the large throttle openings required to create dangerous high-speed unintended acceleration incidents. The two mechanical safety defects identified by NHTSA more than a year ago – “sticking” accelerator pedals and a design flaw that enabled accelerator pedals to become trapped by floor mats – remain the only known causes for these kinds of unsafe unintended acceleration incidents. Toyota has recalled nearly 8 million vehicles in the United States for these two defects." -- transportation.gov Cosmic rays and other wild theories over the simple theory of driver error. Even with a stuck throttle, the brakes will still stop a car (not to mention shifting into neutral still works).
- Denatonium 10mo ago
  Not to mention that in an emergency, you can always turn the key to kill the engine, and then put it back into pre-igntion (to unlock the steering column). You won't have power-assisted braking or power-steering, but with a bit of adrenaline-fueled strength, it is definitely preferable to being in a car that is stuck accelerating.
  - helterskelter 10mo ago
    Key?
    - chneu 10mo ago
      Long press the start/stop button.
  - laweijfmvo 10mo ago
    shift into neutral
    - ehnto 10mo ago
      It was a 2005 model, so it should have been possible. However the article isn't super clear on where exactly the software is running, and the transmission controller and engine control unit can be interlinked in various ways. Especially more modern vehicles, it would be entirely possible to write code that disallowed shifting if it was an automatic. We have no idea just how poorly orchestrated this system was and what features were affected. I don't know enough about 2005 Camry's though, so I wouldn't speculate much further than that.
fnord77 10mo ago
> Other egregious deviations from standard practice were the number of global variables in the system. (A variable is a location in memory that has a number in it. A global variable is any piece of software anywhere in the system can get to that number and read it or write it.) The academic standard is zero. Toyota had more than 10,000 global variables.
- jdlshore 10mo ago
  My understanding is that global variables are more common in embedded systems because it provides memory determinism.
  - monegator 10mo ago
    nah, by looking ad other people's firmware it's because most of my embedded colleagues are goats stuck in pre-ANSI C. You can always declare a "global" static, so it's not on the stack or heap, and access that via functions. Nothing wrong with source-file-level statics, you're bound to use them
    - deleted 10mo ago
      [deleted]
  - Glawen 10mo ago
    Yep that's the standard in embedded on bare metal without memory allocation. There is a mechanism in place to synchronise data during interrupts, so it's not really direct write. Usually also coupled with a two complement variable or similar to make sure memory is not corrupted for safety critical data.
qchris 10mo ago
Related to [1]; this topic was discussed earlier today (perhaps inspiring this submission?) in a HN thread on C++ coding standards for the F-35 JSF (search "spaghetti"). [1] https://news.ycombinator.com/item?id=46183657 https://news.ycombinator.com/item?id=46183657
PhotonHunter 10mo ago
Was there ever a recall of the ECU, and if not, why did the UA events go away? Were UA events more common at higher elevations, where there would be more cosmic ray activity? This story is like Baba Yaga, it comes out from the shadows to scare people every now and then, but Barr’s theory has the interesting property that the ECU would be cleared by the error and so there could never be evidence of the event as he postulated.
- 1970-01-01 10mo ago
  There was an ECU update but not a recall. There's thousands of these old ECUs driving cars that are still on the road. Yet somehow they never misbehaved after the mass hysteria and legal circus was over and forgotten. Hmm.
userbinator 10mo ago
I still believe that the actual cause was tin whiskers, but all the RoHS lobbying buried the evidence. http://nepp.nasa.gov/whisker/reference/tech_papers/2011-NASA-GSFC-whisker-failure-app-sensor.pdf http://nepp.nasa.gov/whisker/reference/tech_papers/2011-NASA...
- Aeglaecia 10mo ago
  im sure that any reasonably charismatic software engineer could scare the shit out of a judge/jury based on code analysis ... perusing the linked nasa document, recurrence odds of this particular failure mode do seem significantly greater than the odds of a cosmic bit flip ...
  - thebruce87m 10mo ago
    > Studies by IBM in the 1990s suggest that computers typically experience about one cosmic-ray-induced error per 256 megabytes of RAM per month. https://www.scientificamerican.com/article/solar-storms-fast-facts/ https://www.scientificamerican.com/article/solar-storms-fast... Just to give perspective on the bit flip probability. ECC ftw!
    - nsoqm 10mo ago
      Is this was true, wouldn’t any modern computer would be crashing several times a month? And please don’t tell me “oh but it is”, because it is not.
      - crote 10mo ago
        How many of those errors vould result in a full system crash, though? And how many of them are just going to cause silent and mostly-harmless data corruption? After all, was the error in the first line a typo on my side, or a single-bit upset? A while ago some researchers registered off-by-one-bit domain name typos, which due to physical key positioning were unlikely to be the result of genuine mistyping. I can't find a reference right now, but I recall them getting quite a lot of queries!
pengaru 10mo ago
"supply voltage to the electronic control system was purposely lowered and perturbed to simulate bad alternator and/or battery system. The result from the manipulation of supply voltage was rather astonishing. The control systems seemed to work even with the perturbed supply voltage but not correctly. As a matter of fact, it seemed to cause the sudden unintended acceleration repeatedly. The supply voltage to the ECU can be disturbed by minor mishap in the alternator output function and possibly by the overload of ever increasing use of electric devices in the vehicle by the driver. In any case, the current study showed the reproduction of the sudden unintended acceleration when the supply voltage changes abruptly by sudden drop of the alternator output voltage or by overload of the electric devices." https://www.sciencedirect.com/science/article/abs/pii/S0379073816303449?via%3Dihub https://www.sciencedirect.com/science/article/abs/pii/S03790...
- Glawen 10mo ago
  So they drop the voltage to mimic an engine cranking, and they are surprised that the ECU behaves like it is cranking. When engine cranks, voltage drops below minimum voltage required by ECU to keep SW running (SW resets). To counter these, ECUs keep outputs to the max. Normally I would expect an electrical loop with the crank signal though.
- M95D 10mo ago
  Couldn't read the article, only the summary, but it sounds like glitching by the description. I expect the ECU lowers 12V to 5V or 3.3V by using a buck converter which includes a filter capacitor. To glitch the CPU, the 12V would need to drop well below 5V to have any effect. I don't see how this could happen. If the battery is weak enough to drop below 9V under any conditions other than a short, that car won't even start. My suspicion is that they glitched the ECU power supply directly, not the 12V input - the summary doesn't say. My conclusion is that it's mosty (scientific) clickbait.
deleted 10mo ago
[deleted]

35 points by SoKamil 10mo ago

Popular in 2015: (96 points, 106 comments) https://news.ycombinator.com/item?id=10437117 https://news.ycombinator.com/item?id=10437117 (152 points, 145 comments) https://news.ycombinator.com/item?id=9643204 https://news.ycombinator.com/item?id=9643204

supahfly_remix 10mo ago

Does anyone know where one could obtain the firmware for this? It might be interesting to reverse engineer.

altairprime 10mo ago
It’s available in various archives of the Toyota TechStream pre-2024 editions, in some sort of weird encrypted file format that can be trivially decrypted; I haven’t tried myself but the ECU I work with isn’t encrypted in-vehicle. I’ve spent five or six years in Ghidra with various hybrid Subaru-Toyota ECUs from 2013-2020 and I wonder what kind of source control practices result in the massive function spaghetti that must have produced in this SH-2A code; I can see where Toyota bolted their direct injection runloop into Subaru’s. So, yeah, if you’re curious, the firmware’s out there, if you’ve got a few years to spare and an absolutely ridiculous amount of patience (and a solid grasp of CAN bus messaging protocols, which you’ll need to identify code blocks and variables and such!) “The Car Hacker’s Handbook” may be of interest as a first step review, but honestly I just dove in with Ghidra and just .. didn’t ever stop. YMMV :)
- supahfly_remix 10mo ago
  lol, thanks for the reply. Yes, I was thinking of looking at it, but your comment dissuaded me. Out of curiosity, did you try running ECU code through an LLM and asking questions of it, and, if so, how did that work out?
  - altairprime 10mo ago
    No.

stackghost 10mo ago

Safety Research Systems, the author of TFA, is a for-profit company whose income is based on lawsuits. Make of that what you will.

throwaway81523 10mo ago
Philip Koopman was an expert witness against Toyota (which is a bit questionable to me) and he has some stuff on his website about the case too. https://betterembsw.blogspot.com/search/label/Toyota%20UA https://betterembsw.blogspot.com/search/label/Toyota%20UA

LanceH 10mo ago

Ah yes, where Toyota was found guilty of not being a US company. The only thing they did in the recall was the same floor mat anchor as so many other cases. "NASA engineers found no electronic flaws in Toyota vehicles capable of producing the large throttle openings required to create dangerous high-speed unintended acceleration incidents. The two mechanical safety defects identified by NHTSA more than a year ago – “sticking” accelerator pedals and a design flaw that enabled accelerator pedals to become trapped by floor mats – remain the only known causes for these kinds of unsafe unintended acceleration incidents. Toyota has recalled nearly 8 million vehicles in the United States for these two defects." -- transportation.gov Cosmic rays and other wild theories over the simple theory of driver error. Even with a stuck throttle, the brakes will still stop a car (not to mention shifting into neutral still works).

Denatonium 10mo ago
Not to mention that in an emergency, you can always turn the key to kill the engine, and then put it back into pre-igntion (to unlock the steering column). You won't have power-assisted braking or power-steering, but with a bit of adrenaline-fueled strength, it is definitely preferable to being in a car that is stuck accelerating.
- helterskelter 10mo ago
  Key?
  - chneu 10mo ago
    Long press the start/stop button.
- laweijfmvo 10mo ago
  shift into neutral
  - ehnto 10mo ago
    It was a 2005 model, so it should have been possible. However the article isn't super clear on where exactly the software is running, and the transmission controller and engine control unit can be interlinked in various ways. Especially more modern vehicles, it would be entirely possible to write code that disallowed shifting if it was an automatic. We have no idea just how poorly orchestrated this system was and what features were affected. I don't know enough about 2005 Camry's though, so I wouldn't speculate much further than that.

fnord77 10mo ago

> Other egregious deviations from standard practice were the number of global variables in the system. (A variable is a location in memory that has a number in it. A global variable is any piece of software anywhere in the system can get to that number and read it or write it.) The academic standard is zero. Toyota had more than 10,000 global variables.

jdlshore 10mo ago
My understanding is that global variables are more common in embedded systems because it provides memory determinism.
- monegator 10mo ago
  nah, by looking ad other people's firmware it's because most of my embedded colleagues are goats stuck in pre-ANSI C. You can always declare a "global" static, so it's not on the stack or heap, and access that via functions. Nothing wrong with source-file-level statics, you're bound to use them
  - deleted 10mo ago
    [deleted]
- Glawen 10mo ago
  Yep that's the standard in embedded on bare metal without memory allocation. There is a mechanism in place to synchronise data during interrupts, so it's not really direct write. Usually also coupled with a two complement variable or similar to make sure memory is not corrupted for safety critical data.

qchris 10mo ago

Related to [1]; this topic was discussed earlier today (perhaps inspiring this submission?) in a HN thread on C++ coding standards for the F-35 JSF (search "spaghetti"). [1] https://news.ycombinator.com/item?id=46183657 https://news.ycombinator.com/item?id=46183657

PhotonHunter 10mo ago

Was there ever a recall of the ECU, and if not, why did the UA events go away? Were UA events more common at higher elevations, where there would be more cosmic ray activity? This story is like Baba Yaga, it comes out from the shadows to scare people every now and then, but Barr’s theory has the interesting property that the ECU would be cleared by the error and so there could never be evidence of the event as he postulated.

1970-01-01 10mo ago
There was an ECU update but not a recall. There's thousands of these old ECUs driving cars that are still on the road. Yet somehow they never misbehaved after the mass hysteria and legal circus was over and forgotten. Hmm.

userbinator 10mo ago

I still believe that the actual cause was tin whiskers, but all the RoHS lobbying buried the evidence. http://nepp.nasa.gov/whisker/reference/tech_papers/2011-NASA-GSFC-whisker-failure-app-sensor.pdf http://nepp.nasa.gov/whisker/reference/tech_papers/2011-NASA...

Aeglaecia 10mo ago
im sure that any reasonably charismatic software engineer could scare the shit out of a judge/jury based on code analysis ... perusing the linked nasa document, recurrence odds of this particular failure mode do seem significantly greater than the odds of a cosmic bit flip ...
- thebruce87m 10mo ago
  > Studies by IBM in the 1990s suggest that computers typically experience about one cosmic-ray-induced error per 256 megabytes of RAM per month. https://www.scientificamerican.com/article/solar-storms-fast-facts/ https://www.scientificamerican.com/article/solar-storms-fast... Just to give perspective on the bit flip probability. ECC ftw!
  - nsoqm 10mo ago
    Is this was true, wouldn’t any modern computer would be crashing several times a month? And please don’t tell me “oh but it is”, because it is not.
    - crote 10mo ago
      How many of those errors vould result in a full system crash, though? And how many of them are just going to cause silent and mostly-harmless data corruption? After all, was the error in the first line a typo on my side, or a single-bit upset? A while ago some researchers registered off-by-one-bit domain name typos, which due to physical key positioning were unlikely to be the result of genuine mistyping. I can't find a reference right now, but I recall them getting quite a lot of queries!

pengaru 10mo ago

"supply voltage to the electronic control system was purposely lowered and perturbed to simulate bad alternator and/or battery system. The result from the manipulation of supply voltage was rather astonishing. The control systems seemed to work even with the perturbed supply voltage but not correctly. As a matter of fact, it seemed to cause the sudden unintended acceleration repeatedly. The supply voltage to the ECU can be disturbed by minor mishap in the alternator output function and possibly by the overload of ever increasing use of electric devices in the vehicle by the driver. In any case, the current study showed the reproduction of the sudden unintended acceleration when the supply voltage changes abruptly by sudden drop of the alternator output voltage or by overload of the electric devices." https://www.sciencedirect.com/science/article/abs/pii/S0379073816303449?via%3Dihub https://www.sciencedirect.com/science/article/abs/pii/S03790...

Glawen 10mo ago
So they drop the voltage to mimic an engine cranking, and they are surprised that the ECU behaves like it is cranking. When engine cranks, voltage drops below minimum voltage required by ECU to keep SW running (SW resets). To counter these, ECUs keep outputs to the max. Normally I would expect an electrical loop with the crank signal though.
M95D 10mo ago
Couldn't read the article, only the summary, but it sounds like glitching by the description. I expect the ECU lowers 12V to 5V or 3.3V by using a buck converter which includes a filter capacitor. To glitch the CPU, the 12V would need to drop well below 5V to have any effect. I don't see how this could happen. If the battery is weak enough to drop below 9V under any conditions other than a short, that car won't even start. My suspicion is that they glitched the ECU power supply directly, not the 12V input - the summary doesn't say. My conclusion is that it's mosty (scientific) clickbait.

deleted 10mo ago

[deleted]