5 ms·

>> The Camry ETCS code was found to have 11,000 global variables. Barr described the code as “spaghetti.” Using the Cyclomatic Complexity metric, 67 functions w

>> The Camry ETCS code was found to have 11,000 global variables. Barr described the code as “spaghetti.” Using the Cyclomatic Complexity metric, 67 functions were rated untestable (meaning they scored more than 50). The throttle angle function scored more than 100 (unmaintainable). >> Toyota loosely followed the widely adopted MISRA-C coding rules but Barr’s group found 80,000 rule violations. Toyota's own internal standards make use of only 11 MISRA-C rules, and five of those were violated in the actual code. MISRA-C:1998, in effect when the code was originally written, has 93 required and 34 advisory rules. Toyota nailed six of them. How the ACTUAL FUCK did this happen!? The article makes Toyota's engineering team seem egregiously irresponsible. Is it typical for vehicle control systems to be this complicated? I would love to hear the other side of the story (from Toyota's engineers). Maybe the MISRA-C industry standard practices are ridiculous, out of touch and impractical.

brohee 11y ago
MISRA-C is practical enough that NASA's JPL used it as a basis for their coding standard (http://lars-lab.jpl.nasa.gov/JPL_Coding_Standard_C.pdf http://lars-lab.jpl.nasa.gov/JPL_Coding_Standard_C.pdf), and everyone would agree those Martian robot works pretty well...
jacquesm 11y ago
More interesting: how does it compare to the rest of the industry at that time?
- fn42 11y ago
  I'm not really qualified here (and if someone is please post) but I've been poking around with the ECU on my 2001 audi and it doesn't seem to have ECC memory either. A section of the AM29F800BB eeprom is used as random-access memory. It seems to stick a couple CRC bits in every line so maybe this is how they're doing it?
  - sitkack 11y ago
    I am starting to think that safety critical systems need to have schematics, pcb layouts, design documents and source code be registered with the gov before it touches the public. I remember when nearly everything came with schematics so it could be fixed.
astrobe_ 11y ago
> Barr’s group found 80,000 rule violations I'm a bit puzzled by this figure given that according to the article the typical LOC count of this kind of software is within that order of magnitude ("tens of thousands of lines of code").
- ynik 11y ago
  "Normal" C code that is written in a coding style that disagrees with the MISRA guidelines can have more than 1 violation per line of code, even if the quality of the codebase is otherwise quite good. For example, the static analyser I'm working on finds 32700 violations of MisraC2012 in the SQLite codebase (130 kLOC). This is because Misra is quite strict about the use of C: a simple statement like `if (ptr) f();` already violates two rules: * use explicit `ptr != NULL` comparison * always use braces with `if` Once you get around to integer arithmetic, where MISRA is quite strict about not relying on implicit conversions or numeric promotion (which is non-portable due to depending on the size of `int`), it's not rare to have 5 or more violations in a single line of code.
rwmj 11y ago
When hardware designers write code, the results are often ugly. Device drivers, firmware, embedded code, ...
- musername 11y ago
  Engineers tend to read specs and apply standards.
EliRivers 11y ago
Maybe the MISRA-C industry standard practices are ridiculous, out of touch and impractical. I currently work in the static analysis industry. Broadly speaking, the MISRA-C ruleset is, in my experience and in the experience of many customers who apply it, not impractical, clearly of benefit, and the majority of rule violations (including a number of rules violated by Toyota) easily found with static analysis tools. I even know that Toyota is a customer of (at least) one of the static analysis tool companies (obviously Toyota is a huge company, and I've no idea if the specific muppets making this clusterf had any static analysis tools; just that Toyota as a company definitely has someone buying them). It seems that they either just didn't use it, or just didn't care (or were told not to care). As an extra point of data, I sit opposite someone who is on the MISRA-C committee. He is a solid C coder who knows a great deal about the language and how to get it wrong. His day job is writing/maintaining static analysis tools for C programmes. Obviously this is no guarantee that the committee as a whole is good at maintaining the MISRA-C ruleset, but they do have at least one active, experienced and competent C coder (with lots of experience of having to actually automate detection of rule violations) at the table.
brudgers 11y ago
It's a matter of scale and economics. Every dollar cost reduction per unit is significant over the multiyear production cycle. The changes for cost reduction make the design process more complex and a Camry or any car is full of legacy features (the model designation is nearly 35 years old). On top of that car companies are massive and capital intensive with all the lack of agility that requires...retooling a production line takes many years from design to cars at the dealer. The critical decisions about programming stack had to be made in the early 1980's when the first microprocessor based control systems appeared in automobiles. Back then, automotive engineers and executives would have hardly predicted the software complexity that was on the horizon. It was state level actors who came up with Ada and Drakon...But those are money is no object and failure is not an option inspired. Toyota probably made money via their level of software quality...shutting down production for a year to get it right would have cost billions. It followed the bean counting and rode the tiger.
henk53 11y ago
>I would love to hear the other side of the story Me too, it's probably indeed bad, but there's ALWAYS another side to such stories. Unfortunately, we rarely get to hear those.
imglorp 11y ago
I'm going to go out on a huge limb here and am prepared to get shot down. I've seen this first hand as inheritor of spaghetti firmware. The limb is: don't let EE's lead a firmware group. It's a natural division of labor. The hardware guys are familiar with all the component datasheets and the bus timings and all the other low level details. If the prototype is misbehaving, they'll grab a scope and figure it out. They're probably the only guys who can. Software is an afterthought, a "free" component. Those same EE's who have performed that role get promoted to buck stoppers for hardware production. They're familiar with hardware design, production, prototyping, Design for Manufacturability, and component vendor searches. They might cross over with production engineering and that six sigma goodness. They're wrapped up in cost-per-unit to produce which is direct ROI. Software remains a "free" component; you just type it up and there it is. The culture of software design for testability, encapsulation, code quality, code review, reuse, patterns, CMM levels, etc etc is largely orthogonal to hardware culture.
- sliverstorm 11y ago
  As a hardware designer I agree that hardware people are typically not very good software engineers. I find we can reliably write small programs that will do everything they need to. But as the program grows, we are not adept at managing the explosion in complexity. However I don't believe that's because hardware culture doesn't value software or anything like that. We just have no proper education in software engineering. Speaking for myself, I grasp most fundamental software concepts. Memory structure, search algorithms, that sort of thing. I appreciate the value of abstract code qualities like testability, simplicity, reusability, etc. I just have no actual education in how to design large programs from scratch that will achieve those goals.
  - yason 11y ago
    I don't think it's about education. It's just that software isn't what hardware people mostly work with and you only get experienced in what you mostly work with. And when you're experienced, you gain some insight into how things will work. I know some basics of EE but I have no gut feeling of how to build or analyze any hardware for real. It's just something I've read and made my mind to understand but it's not something I know. Conversely, I've been writing software on the lowest to highest levels for decades and I have a hunch of how to start building something that will eventually grow really big and complex. I don't know exactly how I do it but depending on what I want to achieve I have, already at the very beginning, a quite strong sense of what might work and also what will definitely NOT work. My take on hardware is that it's mostly a black box that usually does most of what's advertised (but workarounds are regularly needed, and I guess it must be really difficult to build features into silicon and have it work 100% as designed), and these quirks had better be encapsulated in the lowest levels of the driver so that we'll get some real software building blocks sooner. This would be no basis to design hardware. I would make such a terrible mess out of it that even if I managed to design a chip and make it appear to work somewhat, it would fail spectacularly in all kinds of naive cornercases that a real EE would never have to solve because s/he would never venture to build any of them, just because s/he would know better from the start already.
chillingeffect 11y ago
I'm in no way excusing Toyota. For those who want to understand how this happens, I can shed some light. Early firmware was a replacement for analog electronics. Consider modeling the classic lunar lander. A simple digital computational loop with no branches, just math, has advantages over analog circuitry, such temperature stability, noise immunity, reproducibility, etc. From there, embedded firmware grew in the same way complex mechanical systems grew, like mills and robots. Meanwhile, non-embedded software on workstations was growing, theories were developing, algorithms and organization and business models were applied. And firmware grew and grew, but mostly like an engine grows in that abstractions are minimized, optimization happens in-place and theory is largely tribal knowledge. It's only in the last 10 years or so that firmware has metamorphosis end and is confronting other software. It looks especially conflictual (word?) to high-level devs like Ruby or what-have-you. So yeah, this is no excuse for negligence and ignorance. I just hope my perspective helps ppl evolve.
- agoetz 11y ago
  The apollo guidance computer actually used a sophisticated software interpreter for most of its code. Not just a "computational loop with no branches" http://en.wikipedia.org/wiki/Apollo_Guidance_Computer#Software http://en.wikipedia.org/wiki/Apollo_Guidance_Computer#Softwa... In addition, there is no way in hell that the control algorithms it was using could have been developed without the use of computers. State-space control theory was specifically developed to take advantage of discrete-time control systems. https://www.hq.nasa.gov/alsj/ApolloDescentGuidnce.pdf https://www.hq.nasa.gov/alsj/ApolloDescentGuidnce.pdf

brohee 11y ago

MISRA-C is practical enough that NASA's JPL used it as a basis for their coding standard (http://lars-lab.jpl.nasa.gov/JPL_Coding_Standard_C.pdf http://lars-lab.jpl.nasa.gov/JPL_Coding_Standard_C.pdf), and everyone would agree those Martian robot works pretty well...

jacquesm 11y ago

More interesting: how does it compare to the rest of the industry at that time?

fn42 11y ago
I'm not really qualified here (and if someone is please post) but I've been poking around with the ECU on my 2001 audi and it doesn't seem to have ECC memory either. A section of the AM29F800BB eeprom is used as random-access memory. It seems to stick a couple CRC bits in every line so maybe this is how they're doing it?
- sitkack 11y ago
  I am starting to think that safety critical systems need to have schematics, pcb layouts, design documents and source code be registered with the gov before it touches the public. I remember when nearly everything came with schematics so it could be fixed.

astrobe_ 11y ago

> Barr’s group found 80,000 rule violations I'm a bit puzzled by this figure given that according to the article the typical LOC count of this kind of software is within that order of magnitude ("tens of thousands of lines of code").

ynik 11y ago
"Normal" C code that is written in a coding style that disagrees with the MISRA guidelines can have more than 1 violation per line of code, even if the quality of the codebase is otherwise quite good. For example, the static analyser I'm working on finds 32700 violations of MisraC2012 in the SQLite codebase (130 kLOC). This is because Misra is quite strict about the use of C: a simple statement like `if (ptr) f();` already violates two rules: * use explicit `ptr != NULL` comparison * always use braces with `if` Once you get around to integer arithmetic, where MISRA is quite strict about not relying on implicit conversions or numeric promotion (which is non-portable due to depending on the size of `int`), it's not rare to have 5 or more violations in a single line of code.

rwmj 11y ago

When hardware designers write code, the results are often ugly. Device drivers, firmware, embedded code, ...

musername 11y ago
Engineers tend to read specs and apply standards.

EliRivers 11y ago

Maybe the MISRA-C industry standard practices are ridiculous, out of touch and impractical. I currently work in the static analysis industry. Broadly speaking, the MISRA-C ruleset is, in my experience and in the experience of many customers who apply it, not impractical, clearly of benefit, and the majority of rule violations (including a number of rules violated by Toyota) easily found with static analysis tools. I even know that Toyota is a customer of (at least) one of the static analysis tool companies (obviously Toyota is a huge company, and I've no idea if the specific muppets making this clusterf had any static analysis tools; just that Toyota as a company definitely has someone buying them). It seems that they either just didn't use it, or just didn't care (or were told not to care). As an extra point of data, I sit opposite someone who is on the MISRA-C committee. He is a solid C coder who knows a great deal about the language and how to get it wrong. His day job is writing/maintaining static analysis tools for C programmes. Obviously this is no guarantee that the committee as a whole is good at maintaining the MISRA-C ruleset, but they do have at least one active, experienced and competent C coder (with lots of experience of having to actually automate detection of rule violations) at the table.

brudgers 11y ago

It's a matter of scale and economics. Every dollar cost reduction per unit is significant over the multiyear production cycle. The changes for cost reduction make the design process more complex and a Camry or any car is full of legacy features (the model designation is nearly 35 years old). On top of that car companies are massive and capital intensive with all the lack of agility that requires...retooling a production line takes many years from design to cars at the dealer. The critical decisions about programming stack had to be made in the early 1980's when the first microprocessor based control systems appeared in automobiles. Back then, automotive engineers and executives would have hardly predicted the software complexity that was on the horizon. It was state level actors who came up with Ada and Drakon...But those are money is no object and failure is not an option inspired. Toyota probably made money via their level of software quality...shutting down production for a year to get it right would have cost billions. It followed the bean counting and rode the tiger.

henk53 11y ago

>I would love to hear the other side of the story Me too, it's probably indeed bad, but there's ALWAYS another side to such stories. Unfortunately, we rarely get to hear those.

imglorp 11y ago

I'm going to go out on a huge limb here and am prepared to get shot down. I've seen this first hand as inheritor of spaghetti firmware. The limb is: don't let EE's lead a firmware group. It's a natural division of labor. The hardware guys are familiar with all the component datasheets and the bus timings and all the other low level details. If the prototype is misbehaving, they'll grab a scope and figure it out. They're probably the only guys who can. Software is an afterthought, a "free" component. Those same EE's who have performed that role get promoted to buck stoppers for hardware production. They're familiar with hardware design, production, prototyping, Design for Manufacturability, and component vendor searches. They might cross over with production engineering and that six sigma goodness. They're wrapped up in cost-per-unit to produce which is direct ROI. Software remains a "free" component; you just type it up and there it is. The culture of software design for testability, encapsulation, code quality, code review, reuse, patterns, CMM levels, etc etc is largely orthogonal to hardware culture.

sliverstorm 11y ago
As a hardware designer I agree that hardware people are typically not very good software engineers. I find we can reliably write small programs that will do everything they need to. But as the program grows, we are not adept at managing the explosion in complexity. However I don't believe that's because hardware culture doesn't value software or anything like that. We just have no proper education in software engineering. Speaking for myself, I grasp most fundamental software concepts. Memory structure, search algorithms, that sort of thing. I appreciate the value of abstract code qualities like testability, simplicity, reusability, etc. I just have no actual education in how to design large programs from scratch that will achieve those goals.
- yason 11y ago
  I don't think it's about education. It's just that software isn't what hardware people mostly work with and you only get experienced in what you mostly work with. And when you're experienced, you gain some insight into how things will work. I know some basics of EE but I have no gut feeling of how to build or analyze any hardware for real. It's just something I've read and made my mind to understand but it's not something I know. Conversely, I've been writing software on the lowest to highest levels for decades and I have a hunch of how to start building something that will eventually grow really big and complex. I don't know exactly how I do it but depending on what I want to achieve I have, already at the very beginning, a quite strong sense of what might work and also what will definitely NOT work. My take on hardware is that it's mostly a black box that usually does most of what's advertised (but workarounds are regularly needed, and I guess it must be really difficult to build features into silicon and have it work 100% as designed), and these quirks had better be encapsulated in the lowest levels of the driver so that we'll get some real software building blocks sooner. This would be no basis to design hardware. I would make such a terrible mess out of it that even if I managed to design a chip and make it appear to work somewhat, it would fail spectacularly in all kinds of naive cornercases that a real EE would never have to solve because s/he would never venture to build any of them, just because s/he would know better from the start already.

chillingeffect 11y ago

I'm in no way excusing Toyota. For those who want to understand how this happens, I can shed some light. Early firmware was a replacement for analog electronics. Consider modeling the classic lunar lander. A simple digital computational loop with no branches, just math, has advantages over analog circuitry, such temperature stability, noise immunity, reproducibility, etc. From there, embedded firmware grew in the same way complex mechanical systems grew, like mills and robots. Meanwhile, non-embedded software on workstations was growing, theories were developing, algorithms and organization and business models were applied. And firmware grew and grew, but mostly like an engine grows in that abstractions are minimized, optimization happens in-place and theory is largely tribal knowledge. It's only in the last 10 years or so that firmware has metamorphosis end and is confronting other software. It looks especially conflictual (word?) to high-level devs like Ruby or what-have-you. So yeah, this is no excuse for negligence and ignorance. I just hope my perspective helps ppl evolve.

agoetz 11y ago
The apollo guidance computer actually used a sophisticated software interpreter for most of its code. Not just a "computational loop with no branches" http://en.wikipedia.org/wiki/Apollo_Guidance_Computer#Software http://en.wikipedia.org/wiki/Apollo_Guidance_Computer#Softwa... In addition, there is no way in hell that the control algorithms it was using could have been developed without the use of computers. State-space control theory was specifically developed to take advantage of discrete-time control systems. https://www.hq.nasa.gov/alsj/ApolloDescentGuidnce.pdf https://www.hq.nasa.gov/alsj/ApolloDescentGuidnce.pdf