16 ms·

The Quality of Embedded Software, or the Mess Has Happened

65 points by DmitryNovikov 13y ago

andrewcooke 13y ago
https://news.ycombinator.com/item?id=6636811 https://news.ycombinator.com/item?id=6636811
_pmf_ 13y ago
Ah, yes. Comfortable desktop developers theorizing about embedded software. MISRA is more about compatibility with non-standard (or pre-standard) legacy compilers from the 80s and miniscule improvements to the static analyzability of code than about software quality; following MISRA to the letter would actually decrease the maintainability of software significantly, and this is the only valid measure of code quality any practitioner should care about. > via an onboard network like CAN or FlexRay, a more complex superstructure over CAN FlexRay is a completely different bus system.
kabdib 13y ago
I know a good many bad firmware engineers. A few of my theories about why: - Many of them have a concentration in EE, not software. The software they produce is what you'd expect from an amateur. - A lot of engineers do not spend time to think about the whole stack, from decisions at the hardware level all the way to how the user interacts with the product, and how the factory produces it. The more comfortable you are working in these domains, the better decisions you will be making. Unfortunately, for many firmware engineers their responsibility stops at (say) the USB protocol they've hatched. - Firmware engineering is often seen as a lower-class of programming than desktop. (To this, I say: Ha!) I've sure seen a lot of crappy firmware, much of it from contract houses that specialize in things like BIOS and board packages. They don't seem to hire carefully. If you run across that rare gem who can write desktop apps and UI, write host drivers, write firmware, and use a scope, hang onto that person.
- tonyarkles 13y ago
  For my bachelors, I did a dual degree in EE and CS, and had a decent programming background before starting. Watching most of my EE colleagues programming was incredibly painful; it was very hack & slash until it works, and then that's it. In EE, we weren't really evaluated on the quality of our code at all, just whether or not the final product worked (as determined by a 'scope or other physical output). My limited EE experience after school (I mostly do higher-level programming these days) reflects this; this is the education they left school with, and they didn't have anyone afterwards to teach them any better. Makes me cringe so hard, knowing what some of these guys are up to now.
- jmpe 13y ago
  I know a few good firmware engineers. A reason why they, too, produce "crap" is: - Many of them come across hardware errors or suboptimal designs during a later stage in the FW development. In most cases they hack a workaround that doesn't involve a redesign to be able to move on with the next item on the list. Folks higher up the chain of command see this as an excuse to avoid a redesign: "we're not going to do a redesign for that, Mark fixed it in software". Examples: pretty much every ARM System Board out there. - Someone picked a device that was available during the design phase, but went EOL a few months later before production. It happens. - Someone found a cheaper device than originally planned but it doesn't really do what it says on the box. Typically peripheral stuff like ethernet or usb. - And here's the big one: !!Crappy Datasheet!! Some companies, some really really big IC companies produce crap datasheets. Yo get accustomed to the fact that this new Cortex-M µC will probably have an errata or updated datasheet every few months. - the redesign fixed their EMC problem to get the certificate, someone fixed the power supply. But now you get into trouble if you drive this and that device because the voltage takes a dip. But I'd like to add: I agree with your post.
- gte910h 13y ago
  Mobile programming sucked many desktop/etc capable people out of the market too --iOS guy who came from embedded linux driver land.
- svaha1728 13y ago
  I'm having the reverse experience. I moved from embedded C++ to writing desktop apps. At my former job, Bjarne's C++ coding standard was required reading. Here, anything goes. The last programmer actually convinced the team the reason his code gave different mathematical results each time it was run was because "floating point math is stochastic".
speeder 13y ago
I also saw a Toyota Corolla pull suicidal acceleration, I wonder if it is also a software bug.
- tjr 13y ago
  Software like that is where we need "software engineering", with comprehensive requirements, verification, code reviews, sign-offs, extensive testing, etc. It may also be good if such software was open source, so that anyone outside the company could also review it, if they so chose. See also: http://www.tuxradar.com/content/karen-sandler-full-interview http://www.tuxradar.com/content/karen-sandler-full-interview
  - JonFish85 13y ago
    See the thing is, you can't really do that all that easily in an embedded environment. Pretty quickly you get into an "Uncertainty Principle" situation, where it's difficult/impossible to test something without tainting the results. For all of the code reviews and testing you want to do, how are you planning on having everyone test interrupt flows, re-entrancy, transients in interrupt signals, defective/leaky grounds, watchdog timers, memory bus clock errors, mutex/semaphores, pullups that are too weak or too strong, etc? Code reviews are all well & good, to be sure, but it's not just software anymore, there are physical things to think about. "Software engineering" is usually too specifically "high level" to apply as well to embedded firmware. How do you plan on writing unit tests to run? How do you test a signal edge that seems to fall outside of a valid window once every few days? Open source the code if you want (and if it's legal: a lot of times there are NDAs signed for processors), but it does no one any good if they can't have the same hardware, made with the same components, often times compiled with proprietary compilers. It's about so much more than just code, so be careful of mis-applying great software engineering principles to a completely different situation.
    - tjr 13y ago
      I'm not very familiar with how car component software is developed, so I can't speak to that. In avionics, the software requirements are only one part of the system. There's requirements for how the hardware is to behave, and tolerances for acceptable behavior. All of these pieces are tested. The software is tested by itself in a pure software environment, and it's tested running on the specific piece of hardware that it needs to run on, and that software+hardware combination is tested in various stages of increasingly complete avionics systems, culminating in testing the entire airplane. But I was just talking about the software, where we do employ requirements, verification, testing, and "software engineering". That's only one piece of the system of course, but still a valuable piece. If the software doesn't work right by itself in an idealized environment, then it has little hope of working right when the hardware is going bonkers. And even in the idealized pure software environment, we simulate various system failures to do initial testing on how the software responds.
Mikeb85 13y ago
Not that this exonerates Toyota in any ways, but if your car is accelerating out of control, why not just take it out of gear (ie. put it into neutral)? Or have they removed that ability from newer automatic transmissions? (my last few cars all have manual transmissions)
- drbawb 13y ago
  On most new cars: the transmission is entirely computer controlled. This is the case for "semi-automated manuals" like DCTs as well as automatics. For e.g: to shift to neutral or disable a Prius at speed you have to "hold" the shifter in the neutral position [note: it doesn't seem to stay there like it would on a traditional automatic gear selector; it's spring loaded] or "hold" the ignition switch for 3-5 seconds. This is (presumably) to prevent the motor & trans from grenading itself. I haven't driven a vehicle newer than model-year 2003 recently: so in a panic situation I would expect my cars to make the "money shift" and blow up their powertrains; not "ask me for confirmation" that I want to destroy my motor. Having to wait to make a money-shift seems like a feature that looks excellent on paper, but terrible in practice. I find it difficult to buy a car which only has firmware-implemented kill switches.
  - rlanday 13y ago
    Is this common, that an automatic transmission won’t immediately shift to neutral? I thought that automatic transmissions usually let you shift to neutral without pushing the shifter button in to let you shift to neutral without blowing through it at trying to put the engine in reverse. Shifting into neutral while driving shouldn’t break anything, unless maybe if the throttle is fully open and the rev limiter isn’t working, but that’s the most important situation for being able to shift into neutral, isn’t it? I don’t see why an engineer would want to try to prevent this.
    - dmm 13y ago
      The Prius doesn't have a tradition automatic transmission with a torque converter and multiple gear ratios. The gas engine is connected to the wheels at a fixed ratio along with two motor/generators in a planetary gear system. Reverse is achieved by running the electric motor in reverse. There is no reverse gear. Shifting the neutral on the prius just means disconnecting the MGs and turning off the ICE.
deleted 13y ago
[deleted]
pkolaczk 13y ago
I remember during EE course we had an assigned project to write some software for 8051 microcontroller, in assembly. I organized my code into small "functions" and used a consistent stack-based calling convention with pushing all the resisters on the stack and restoring them back on exit, just as if it was generated by a C compiler. The teacher looked at this and said "huh, look, this is not a CS lab. We do not do things like this in embedded software. Why don't you just keep everything global and use absolute addressing instead those crazy push/pop calls? And you should be reusing registers..." Anyways, it was lot of fun.
- robomartin 13y ago
  On a resource and computing-power constrained system used for a very specific function it would not be too far off if one said something like that(with caveats). Anything past that category and he would be dead wrong. In other words, if you only have 2K or RAM to play with, a limited stack depth, a CPU clock of, say, 12MHz and a 12 clock instruction cycle you have no choice but to get very clever. Pushing and popping a bunch of registers on every function call could rob you of much needed performance and resources. In fact, on such a system that alone could cause unintended side effects and introduce potential safety issues (talking real-time systems with a requirement for deterministic response). You could also have a situation where you overflow the stack and things go from bad to worst very quickly. BTW, this, among other things, is one of the reasons I abandoned such small, 8-bit, resource-constrained architectures for most embedded projects a long time ago. The 8051 derivatives (Silicon Labs) have always been some of my favorites. I might still use them for small projects such as small control panels or simple sensor pre-processing. These days you can buy 32 bit processors running in the tens to hundreds of MHz for $10 or less. These provide you with the ability to write "proper" embedded software with all the luxuries of greatly increased resources and operating speed.
  - hfsktr 13y ago
    I like the way you explained it. The way it was written above seems like he was just told he's wrong but not any of the reasons why. Might just be my lack of knowledge in the area but it made the teacher's response more understandable.
ausjke 13y ago
I read that report a few days ago. It's indeed shocking. One way to explain it is that, the split of EE and CS in universities, most firmware engineers are EE graduates, who knows the hardware to the register level, but not as good on software, not at all. The life-threatening related products, such as cars, medical devices, rockets,etc, probably some strict certificate on software development is good, otherwise it's totally subjective. The so-called computer engineering major is a nice try, students take courses in both EE and CS fields.
diydsp 13y ago
This is not about the state of embedded software. It is about one program, the accelerator control of a Toyota. The author of the article is linkbaiting and fearmongering to get you to purchase his/her static code analyzer for C++. As embedded software becomes ever more complex, it will benefit from methodologies developed for other platforms, such as desktop applications, etc. That is all that needs to be said. Please refrain from the "EEs don't know how to program" comments. Thank you.
- pjc50 13y ago
  As embedded software becomes ever more complex This is part of the problem in itself; the Toyota software appears to be far more complex than it needs to be.
- robomartin 13y ago
  > Please refrain from the "EEs don't know how to program" comments. No, no. As an EE I can tell you that lots of EE's really suck at software development. They never really took the time to study CS to a reasonable depth. As a result you see code that is just horrid and really naive. I've seen stuff that could blow up boards Hollywood style in a microsecond. Such things as potential timing conflicts turning on two MOSFETs such that you have cross conduction and draw hundreds of amps due to the direct short from +V to GND. Now, on the other hand, "desktop" programmers are equally --if not more-- dangerous than ignorant EE's. They have no clue and no experience with hostile embedded environments. I mean, the simple startup configuration code of some modern microcontrollers is the kind of thing you'd never see in desktop or web development. Then you have a myriad of other issues: watchdog timers, drooping/unreliable power supplies, static discharge-induced reset, load dump in automotive, similar applications, or noisy signals, sensor and key-panel data, potential single bit flip situations in aerospace/radiation environments, thermal issues, etc. The list is huge. These are issues that are far better understood by an EE. And so, the best embedded software developer is probably going to be an EE with real solid training of CS principles as well as an solid understanding of defensive or safety-critical electronic and software design principles. An example that comes to mind was when we designed a high-power motor controller many years ago. This thing could handle thousands of Watts. Testing included full output short circuits, zapping it with static electricity, exposing it to RF via close proximity antennas and coils and other torture tests. We discovered that under very rare conditions the output to go to full power when power was first applied AND the power connection was, I'll say, uniquely noisy. Imagine a corroded connection to a power strip on a marine platform that is moving. When power is applied, if you look at it on an oscilloscope, it looked like the most horrible noise you'd ever seen --except this noise was actually powering the board. We had to build a custom power tester using old motor brushes and an old commutator driven by a small motor in order to simulate ugly power supply scenarios. We later replaced this with a programmable digital system that could do the same. The embedded software and the hardware was then modified to deal with these ugly conditions and much worst cases as well. The thing was nearly bulletproof. So, yeah, drop any EE in front of a compiler and you'll probably end-up with really dangerous code. An EE with the right training and lots of scars would be the best option. Someone with a bachelors or masters in CS and no EE training and embedded world experience could be scary dangerous.
pjc50 13y ago
If you like this sort of thing, read comp.risks ( http://catless.ncl.ac.uk/risks http://catless.ncl.ac.uk/risks ). On the other hand, it might give you an omnipresent sense of fear about all the computer-controlled safety critical devices around you.
RougeFemme 13y ago
I used to work in a world with lots of embedded software. My experience was that - regardless of domain or company -the "hardware" was regarded as just that - simply hardware. And it was tested as that - simply hardware - not as hardware that, oh by the way, had a helluva lot of critical embedded software. So issues that could have been caught at some level of testing weren't.
wissler 13y ago
Is there a reason why something so fundamental as throttle control/braking ought to be governed by a computer, instead of by foolproof old-fashioned hardware?
- dmm 13y ago
  The Prius like all US cars has a physical hydraulic brake system. The brake pedal is connected to the master brake cylinder. Stomping on the brake pedal in a prius, regardless of the computer state, will brake the car. The throttle is computer controlled because it allows much greater efficiency.
  - angersock 13y ago
    Anti-lock brakes still have a significant electronic component. It's not a direct link like you'd expect.
- HeyLaughingBoy 13y ago
  "Old fashioned hardware" is anything but foolproof. Linkages bind and flex and change their behavior over time, hydraulics leak and piping must run in areas that can accumulate rust (I had the not so pleasant experience of losing my main brakes due to a rusted out brake line. Driving with only an E-brake is not fun!), steel is heavy, etc. Computers do add benefit: ABS, traction control and stability control would be a nightmare to implement mechanically. There is no question that the software solution is a better idea, but the design and implementation must be done correctly.
  - vonmoltke 13y ago
    I think it is more fair to say that, being old and well-used, the engineering behind old-fashioned hardware is more mature and its failure modes are better understood. Embedded electronic systems of various types will reach that point as they are more widely deployed and used for longer periods. BTW, I have been through that exact failure scenario before myself (in an '81 Monte Carlo).
ChuckMcM 13y ago
I agree it is a bit sensationalist, and certainly disparaging of EE's, but the truth that there is more and more software between you and an unpleasant experience is absolutely true. One of the things that really annoys me is that there isn't more openness and sharing in the embedded space. Its not uncommon for me to go to a manufacturer lecture where the only compiler choices are proprietary offerings and their library functions and includes are riddled with proprietary licenses. In one case I pointed out bugs in their (re)-implmentation of some basic libc functions that were fixed in newlib years ago and pretty much everywhere else. While the people around me fumble with their Windows [1] machines I pull up a gcc cross compilation chain and begin porting the proprietary "workspace" or "project" files into a bog standard make file. I've gotten to the point where I can have their 'demo lesson' running in a fully open toolchain faster than they can get the rest of the attendees windows and custom package install issues sorted out. Of course I've already got the basic tools installed, and generally the other students are installing an older version of the same toolchain I'm using but into some incompatible set of directories that just the vendor uses. I went to Atmel, TI, and ST Micro chip seminars (looking for a processor for my educational development board) and had I installed the proprietary tools I would have ended up with three copies of GCC installed on my laptop! I've been supporting efforts like the libopencm3 desire to build an open and common ARM support library, but more of that needs to happen to build in some good practices for these embedded developers to learn from.
- nickff 13y ago
  >One of the things that really annoys me is that there isn't more openness and sharing in the embedded space. Embedded products are time and capital intensive, which means that the cost of a compiler or other tool-set is very small relative to the total cost of product development. Because of this, the developers (and their companies) do not mind paying a few thousand dollars for a new piece of software. Anything the developer creates is often only useful for competitors to one's product, so there is very little incentive to share. >I've gotten to the point where I can have their 'demo lesson' running in a fully open toolchain faster than they can get the rest of the attendees windows and custom package install issues sorted out. If you spend a year working on developing an entirely new hardware & firmware product, you see that importing new files, finding libraries, and adapting software from one IDE to another is not a very important or time consuming part of the project. When you spend a few weeks discovering that there are undocumented hardware glitches in the integrated circuit, that the errata is inaccurate, or that you own hardware behaves unexpectedly, the IDE is of little consequence. >I've been supporting efforts like the libopencm3 desire to build an open and common ARM support library, but more of that needs to happen to build in some good practices for these embedded developers to learn from. As an embedded developer, the open ARM library is of little interest to me. ARM is proprietary anyway, and building open-source projects on top of closed source systems does not seem to be a productive use of time.
wissler 13y ago
Regarding the EE bashing, there's an element of truth but overall it is an unfair characterization. I think the true source of the problem lies not in the EE degree or lack of a CS degree, but in a specific, weird distrust of abstraction by in the institutional culture of embedded systems development. This opposition (and it is opposition, not merely ignorance) isn't coming from the degree per se but from the field as it is sometimes practiced. My guess is that the reason for this culture is that long ago, there was a history of "fancy newfangled compilers" screwing up the code, engineers getting repeatedly burned, and learning not to trust such "fancy" tools. This attitude may have made some sense a long time ago, but now it's an institutional relic. So the irony is: the engineers are avoiding abstraction in order to increase reliability, but they wind up decreasing it because the code becomes impossible to manage. Anyway, this is not a defect in the way EE's are educated, it is a very specific subculture of embedded systems development, which traditionally happens to tend to hire EE's.
memracom 13y ago
This is why the world needs something like SEMAT http://semat.org/ http://semat.org/ More and more of our environment is controlled by software and when it breaks, it doesn't just shut down somebody's store but it does real damage to real objects in the real world. Esepcially electrical engineers should be following rigorous processes in building and testing software.
AndreyKarpov 13y ago
Don't be evil programmer. Use PVS-Studio Static Code Analyzer for C/C++: http://www.viva64.com/en/pvs-studio/ http://www.viva64.com/en/pvs-studio/ :)

65 points by DmitryNovikov 13y ago

https://news.ycombinator.com/item?id=6636811 https://news.ycombinator.com/item?id=6636811

_pmf_ 13y ago

Ah, yes. Comfortable desktop developers theorizing about embedded software. MISRA is more about compatibility with non-standard (or pre-standard) legacy compilers from the 80s and miniscule improvements to the static analyzability of code than about software quality; following MISRA to the letter would actually decrease the maintainability of software significantly, and this is the only valid measure of code quality any practitioner should care about. > via an onboard network like CAN or FlexRay, a more complex superstructure over CAN FlexRay is a completely different bus system.

kabdib 13y ago

I know a good many bad firmware engineers. A few of my theories about why: - Many of them have a concentration in EE, not software. The software they produce is what you'd expect from an amateur. - A lot of engineers do not spend time to think about the whole stack, from decisions at the hardware level all the way to how the user interacts with the product, and how the factory produces it. The more comfortable you are working in these domains, the better decisions you will be making. Unfortunately, for many firmware engineers their responsibility stops at (say) the USB protocol they've hatched. - Firmware engineering is often seen as a lower-class of programming than desktop. (To this, I say: Ha!) I've sure seen a lot of crappy firmware, much of it from contract houses that specialize in things like BIOS and board packages. They don't seem to hire carefully. If you run across that rare gem who can write desktop apps and UI, write host drivers, write firmware, and use a scope, hang onto that person.

tonyarkles 13y ago
For my bachelors, I did a dual degree in EE and CS, and had a decent programming background before starting. Watching most of my EE colleagues programming was incredibly painful; it was very hack & slash until it works, and then that's it. In EE, we weren't really evaluated on the quality of our code at all, just whether or not the final product worked (as determined by a 'scope or other physical output). My limited EE experience after school (I mostly do higher-level programming these days) reflects this; this is the education they left school with, and they didn't have anyone afterwards to teach them any better. Makes me cringe so hard, knowing what some of these guys are up to now.
jmpe 13y ago
I know a few good firmware engineers. A reason why they, too, produce "crap" is: - Many of them come across hardware errors or suboptimal designs during a later stage in the FW development. In most cases they hack a workaround that doesn't involve a redesign to be able to move on with the next item on the list. Folks higher up the chain of command see this as an excuse to avoid a redesign: "we're not going to do a redesign for that, Mark fixed it in software". Examples: pretty much every ARM System Board out there. - Someone picked a device that was available during the design phase, but went EOL a few months later before production. It happens. - Someone found a cheaper device than originally planned but it doesn't really do what it says on the box. Typically peripheral stuff like ethernet or usb. - And here's the big one: !!Crappy Datasheet!! Some companies, some really really big IC companies produce crap datasheets. Yo get accustomed to the fact that this new Cortex-M µC will probably have an errata or updated datasheet every few months. - the redesign fixed their EMC problem to get the certificate, someone fixed the power supply. But now you get into trouble if you drive this and that device because the voltage takes a dip. But I'd like to add: I agree with your post.
gte910h 13y ago
Mobile programming sucked many desktop/etc capable people out of the market too --iOS guy who came from embedded linux driver land.
svaha1728 13y ago
I'm having the reverse experience. I moved from embedded C++ to writing desktop apps. At my former job, Bjarne's C++ coding standard was required reading. Here, anything goes. The last programmer actually convinced the team the reason his code gave different mathematical results each time it was run was because "floating point math is stochastic".

speeder 13y ago

I also saw a Toyota Corolla pull suicidal acceleration, I wonder if it is also a software bug.

tjr 13y ago
Software like that is where we need "software engineering", with comprehensive requirements, verification, code reviews, sign-offs, extensive testing, etc. It may also be good if such software was open source, so that anyone outside the company could also review it, if they so chose. See also: http://www.tuxradar.com/content/karen-sandler-full-interview http://www.tuxradar.com/content/karen-sandler-full-interview
- JonFish85 13y ago
  See the thing is, you can't really do that all that easily in an embedded environment. Pretty quickly you get into an "Uncertainty Principle" situation, where it's difficult/impossible to test something without tainting the results. For all of the code reviews and testing you want to do, how are you planning on having everyone test interrupt flows, re-entrancy, transients in interrupt signals, defective/leaky grounds, watchdog timers, memory bus clock errors, mutex/semaphores, pullups that are too weak or too strong, etc? Code reviews are all well & good, to be sure, but it's not just software anymore, there are physical things to think about. "Software engineering" is usually too specifically "high level" to apply as well to embedded firmware. How do you plan on writing unit tests to run? How do you test a signal edge that seems to fall outside of a valid window once every few days? Open source the code if you want (and if it's legal: a lot of times there are NDAs signed for processors), but it does no one any good if they can't have the same hardware, made with the same components, often times compiled with proprietary compilers. It's about so much more than just code, so be careful of mis-applying great software engineering principles to a completely different situation.
  - tjr 13y ago
    I'm not very familiar with how car component software is developed, so I can't speak to that. In avionics, the software requirements are only one part of the system. There's requirements for how the hardware is to behave, and tolerances for acceptable behavior. All of these pieces are tested. The software is tested by itself in a pure software environment, and it's tested running on the specific piece of hardware that it needs to run on, and that software+hardware combination is tested in various stages of increasingly complete avionics systems, culminating in testing the entire airplane. But I was just talking about the software, where we do employ requirements, verification, testing, and "software engineering". That's only one piece of the system of course, but still a valuable piece. If the software doesn't work right by itself in an idealized environment, then it has little hope of working right when the hardware is going bonkers. And even in the idealized pure software environment, we simulate various system failures to do initial testing on how the software responds.

Mikeb85 13y ago

Not that this exonerates Toyota in any ways, but if your car is accelerating out of control, why not just take it out of gear (ie. put it into neutral)? Or have they removed that ability from newer automatic transmissions? (my last few cars all have manual transmissions)

drbawb 13y ago
On most new cars: the transmission is entirely computer controlled. This is the case for "semi-automated manuals" like DCTs as well as automatics. For e.g: to shift to neutral or disable a Prius at speed you have to "hold" the shifter in the neutral position [note: it doesn't seem to stay there like it would on a traditional automatic gear selector; it's spring loaded] or "hold" the ignition switch for 3-5 seconds. This is (presumably) to prevent the motor & trans from grenading itself. I haven't driven a vehicle newer than model-year 2003 recently: so in a panic situation I would expect my cars to make the "money shift" and blow up their powertrains; not "ask me for confirmation" that I want to destroy my motor. Having to wait to make a money-shift seems like a feature that looks excellent on paper, but terrible in practice. I find it difficult to buy a car which only has firmware-implemented kill switches.
- rlanday 13y ago
  Is this common, that an automatic transmission won’t immediately shift to neutral? I thought that automatic transmissions usually let you shift to neutral without pushing the shifter button in to let you shift to neutral without blowing through it at trying to put the engine in reverse. Shifting into neutral while driving shouldn’t break anything, unless maybe if the throttle is fully open and the rev limiter isn’t working, but that’s the most important situation for being able to shift into neutral, isn’t it? I don’t see why an engineer would want to try to prevent this.
  - dmm 13y ago
    The Prius doesn't have a tradition automatic transmission with a torque converter and multiple gear ratios. The gas engine is connected to the wheels at a fixed ratio along with two motor/generators in a planetary gear system. Reverse is achieved by running the electric motor in reverse. There is no reverse gear. Shifting the neutral on the prius just means disconnecting the MGs and turning off the ICE.

deleted 13y ago

[deleted]

pkolaczk 13y ago

I remember during EE course we had an assigned project to write some software for 8051 microcontroller, in assembly. I organized my code into small "functions" and used a consistent stack-based calling convention with pushing all the resisters on the stack and restoring them back on exit, just as if it was generated by a C compiler. The teacher looked at this and said "huh, look, this is not a CS lab. We do not do things like this in embedded software. Why don't you just keep everything global and use absolute addressing instead those crazy push/pop calls? And you should be reusing registers..." Anyways, it was lot of fun.

robomartin 13y ago
On a resource and computing-power constrained system used for a very specific function it would not be too far off if one said something like that(with caveats). Anything past that category and he would be dead wrong. In other words, if you only have 2K or RAM to play with, a limited stack depth, a CPU clock of, say, 12MHz and a 12 clock instruction cycle you have no choice but to get very clever. Pushing and popping a bunch of registers on every function call could rob you of much needed performance and resources. In fact, on such a system that alone could cause unintended side effects and introduce potential safety issues (talking real-time systems with a requirement for deterministic response). You could also have a situation where you overflow the stack and things go from bad to worst very quickly. BTW, this, among other things, is one of the reasons I abandoned such small, 8-bit, resource-constrained architectures for most embedded projects a long time ago. The 8051 derivatives (Silicon Labs) have always been some of my favorites. I might still use them for small projects such as small control panels or simple sensor pre-processing. These days you can buy 32 bit processors running in the tens to hundreds of MHz for $10 or less. These provide you with the ability to write "proper" embedded software with all the luxuries of greatly increased resources and operating speed.
- hfsktr 13y ago
  I like the way you explained it. The way it was written above seems like he was just told he's wrong but not any of the reasons why. Might just be my lack of knowledge in the area but it made the teacher's response more understandable.

ausjke 13y ago

I read that report a few days ago. It's indeed shocking. One way to explain it is that, the split of EE and CS in universities, most firmware engineers are EE graduates, who knows the hardware to the register level, but not as good on software, not at all. The life-threatening related products, such as cars, medical devices, rockets,etc, probably some strict certificate on software development is good, otherwise it's totally subjective. The so-called computer engineering major is a nice try, students take courses in both EE and CS fields.

diydsp 13y ago

This is not about the state of embedded software. It is about one program, the accelerator control of a Toyota. The author of the article is linkbaiting and fearmongering to get you to purchase his/her static code analyzer for C++. As embedded software becomes ever more complex, it will benefit from methodologies developed for other platforms, such as desktop applications, etc. That is all that needs to be said. Please refrain from the "EEs don't know how to program" comments. Thank you.

pjc50 13y ago
As embedded software becomes ever more complex This is part of the problem in itself; the Toyota software appears to be far more complex than it needs to be.
robomartin 13y ago
> Please refrain from the "EEs don't know how to program" comments. No, no. As an EE I can tell you that lots of EE's really suck at software development. They never really took the time to study CS to a reasonable depth. As a result you see code that is just horrid and really naive. I've seen stuff that could blow up boards Hollywood style in a microsecond. Such things as potential timing conflicts turning on two MOSFETs such that you have cross conduction and draw hundreds of amps due to the direct short from +V to GND. Now, on the other hand, "desktop" programmers are equally --if not more-- dangerous than ignorant EE's. They have no clue and no experience with hostile embedded environments. I mean, the simple startup configuration code of some modern microcontrollers is the kind of thing you'd never see in desktop or web development. Then you have a myriad of other issues: watchdog timers, drooping/unreliable power supplies, static discharge-induced reset, load dump in automotive, similar applications, or noisy signals, sensor and key-panel data, potential single bit flip situations in aerospace/radiation environments, thermal issues, etc. The list is huge. These are issues that are far better understood by an EE. And so, the best embedded software developer is probably going to be an EE with real solid training of CS principles as well as an solid understanding of defensive or safety-critical electronic and software design principles. An example that comes to mind was when we designed a high-power motor controller many years ago. This thing could handle thousands of Watts. Testing included full output short circuits, zapping it with static electricity, exposing it to RF via close proximity antennas and coils and other torture tests. We discovered that under very rare conditions the output to go to full power when power was first applied AND the power connection was, I'll say, uniquely noisy. Imagine a corroded connection to a power strip on a marine platform that is moving. When power is applied, if you look at it on an oscilloscope, it looked like the most horrible noise you'd ever seen --except this noise was actually powering the board. We had to build a custom power tester using old motor brushes and an old commutator driven by a small motor in order to simulate ugly power supply scenarios. We later replaced this with a programmable digital system that could do the same. The embedded software and the hardware was then modified to deal with these ugly conditions and much worst cases as well. The thing was nearly bulletproof. So, yeah, drop any EE in front of a compiler and you'll probably end-up with really dangerous code. An EE with the right training and lots of scars would be the best option. Someone with a bachelors or masters in CS and no EE training and embedded world experience could be scary dangerous.

pjc50 13y ago

If you like this sort of thing, read comp.risks ( http://catless.ncl.ac.uk/risks http://catless.ncl.ac.uk/risks ). On the other hand, it might give you an omnipresent sense of fear about all the computer-controlled safety critical devices around you.

RougeFemme 13y ago

I used to work in a world with lots of embedded software. My experience was that - regardless of domain or company -the "hardware" was regarded as just that - simply hardware. And it was tested as that - simply hardware - not as hardware that, oh by the way, had a helluva lot of critical embedded software. So issues that could have been caught at some level of testing weren't.

wissler 13y ago

Is there a reason why something so fundamental as throttle control/braking ought to be governed by a computer, instead of by foolproof old-fashioned hardware?

dmm 13y ago
The Prius like all US cars has a physical hydraulic brake system. The brake pedal is connected to the master brake cylinder. Stomping on the brake pedal in a prius, regardless of the computer state, will brake the car. The throttle is computer controlled because it allows much greater efficiency.
- angersock 13y ago
  Anti-lock brakes still have a significant electronic component. It's not a direct link like you'd expect.
HeyLaughingBoy 13y ago
"Old fashioned hardware" is anything but foolproof. Linkages bind and flex and change their behavior over time, hydraulics leak and piping must run in areas that can accumulate rust (I had the not so pleasant experience of losing my main brakes due to a rusted out brake line. Driving with only an E-brake is not fun!), steel is heavy, etc. Computers do add benefit: ABS, traction control and stability control would be a nightmare to implement mechanically. There is no question that the software solution is a better idea, but the design and implementation must be done correctly.
- vonmoltke 13y ago
  I think it is more fair to say that, being old and well-used, the engineering behind old-fashioned hardware is more mature and its failure modes are better understood. Embedded electronic systems of various types will reach that point as they are more widely deployed and used for longer periods. BTW, I have been through that exact failure scenario before myself (in an '81 Monte Carlo).

ChuckMcM 13y ago

I agree it is a bit sensationalist, and certainly disparaging of EE's, but the truth that there is more and more software between you and an unpleasant experience is absolutely true. One of the things that really annoys me is that there isn't more openness and sharing in the embedded space. Its not uncommon for me to go to a manufacturer lecture where the only compiler choices are proprietary offerings and their library functions and includes are riddled with proprietary licenses. In one case I pointed out bugs in their (re)-implmentation of some basic libc functions that were fixed in newlib years ago and pretty much everywhere else. While the people around me fumble with their Windows [1] machines I pull up a gcc cross compilation chain and begin porting the proprietary "workspace" or "project" files into a bog standard make file. I've gotten to the point where I can have their 'demo lesson' running in a fully open toolchain faster than they can get the rest of the attendees windows and custom package install issues sorted out. Of course I've already got the basic tools installed, and generally the other students are installing an older version of the same toolchain I'm using but into some incompatible set of directories that just the vendor uses. I went to Atmel, TI, and ST Micro chip seminars (looking for a processor for my educational development board) and had I installed the proprietary tools I would have ended up with three copies of GCC installed on my laptop! I've been supporting efforts like the libopencm3 desire to build an open and common ARM support library, but more of that needs to happen to build in some good practices for these embedded developers to learn from.

nickff 13y ago
>One of the things that really annoys me is that there isn't more openness and sharing in the embedded space. Embedded products are time and capital intensive, which means that the cost of a compiler or other tool-set is very small relative to the total cost of product development. Because of this, the developers (and their companies) do not mind paying a few thousand dollars for a new piece of software. Anything the developer creates is often only useful for competitors to one's product, so there is very little incentive to share. >I've gotten to the point where I can have their 'demo lesson' running in a fully open toolchain faster than they can get the rest of the attendees windows and custom package install issues sorted out. If you spend a year working on developing an entirely new hardware & firmware product, you see that importing new files, finding libraries, and adapting software from one IDE to another is not a very important or time consuming part of the project. When you spend a few weeks discovering that there are undocumented hardware glitches in the integrated circuit, that the errata is inaccurate, or that you own hardware behaves unexpectedly, the IDE is of little consequence. >I've been supporting efforts like the libopencm3 desire to build an open and common ARM support library, but more of that needs to happen to build in some good practices for these embedded developers to learn from. As an embedded developer, the open ARM library is of little interest to me. ARM is proprietary anyway, and building open-source projects on top of closed source systems does not seem to be a productive use of time.

wissler 13y ago

Regarding the EE bashing, there's an element of truth but overall it is an unfair characterization. I think the true source of the problem lies not in the EE degree or lack of a CS degree, but in a specific, weird distrust of abstraction by in the institutional culture of embedded systems development. This opposition (and it is opposition, not merely ignorance) isn't coming from the degree per se but from the field as it is sometimes practiced. My guess is that the reason for this culture is that long ago, there was a history of "fancy newfangled compilers" screwing up the code, engineers getting repeatedly burned, and learning not to trust such "fancy" tools. This attitude may have made some sense a long time ago, but now it's an institutional relic. So the irony is: the engineers are avoiding abstraction in order to increase reliability, but they wind up decreasing it because the code becomes impossible to manage. Anyway, this is not a defect in the way EE's are educated, it is a very specific subculture of embedded systems development, which traditionally happens to tend to hire EE's.

memracom 13y ago

This is why the world needs something like SEMAT http://semat.org/ http://semat.org/ More and more of our environment is controlled by software and when it breaks, it doesn't just shut down somebody's store but it does real damage to real objects in the real world. Esepcially electrical engineers should be following rigorous processes in building and testing software.

AndreyKarpov 13y ago

Don't be evil programmer. Use PVS-Studio Static Code Analyzer for C/C++: http://www.viva64.com/en/pvs-studio/ http://www.viva64.com/en/pvs-studio/ :)