8 ms·

Embedded Rules of Thumb (2018)

168 points by nsajko 6y ago

disposedtrolley 6y ago
With the exception of some of the rules under the Hardware section, this is some great succinct advice on software development in general :)
- jpm_sd 6y ago
  Well, they're all good rules, but nobody is going to remember them in the form of a cluttered bulleted list. Jack Ganssle is quoted repeatedly - his blog has a lot of good war stories that illustrate many of these points in much greater detail. http://www.ganssle.com/blog/index.html http://www.ganssle.com/blog/index.html
  - disposedtrolley 6y ago
    Agreed, but the intention appears to serve as a launchpad for digging into the rules yourself. Links to original sources would make this much easier of course. Thanks for the link to Ganssle’s blog. I’ll be sure to have a read today.
- projektfu 6y ago
  Almost has too much general advice. The embedded parts are easy to skim over.
PoachedSausage 6y ago
> Every sensor is a temperature sensor. Some sensors measure other things as well. Some components can become sensors under the right conditions. Example: Components becoming microphonic. At high enough frequencies digital design becomes analogue design.
- bfrog 6y ago
  So true!
- microtherion 6y ago
  > At high enough frequencies digital design becomes analogue design. I've heard this stated as "the difference between digital engineers and analog engineers is that the analog engineers KNOW when they're designing an antenna".
  - craftinator 6y ago
    > "the difference between digital engineers and analog engineers is that the analog engineers KNOW when they're designing an antenna". Hahaha this is a beautiful quote! I didn't believe this was a real concern until I ran into it in the real world.
- turbinerneiter 6y ago
  I read a great how-to once where someone described a device which blinks in the dark - and only in the dark - that uses the LED itself as light sensor.
  - cjsawyer 6y ago
    That is pure evil.
  - PoachedSausage 6y ago
    At work we replace an obsolete diode in part of a circuit in front of a high gain amplifier. The new equivalent diode came in a glass package unlike the old one. It caused a bit of head scratching to figure out why the circuit wasn't behaving as it should and also appeared to be acting as a human proximity detector.
- Cerium 6y ago
  That is how I was convinced to take a microwave engineering course. I was an EE undergrad, and had just struggled through the electrodynamics course. My professor asked me to take his microwave engineering course the following semester. I said I was not planning on it, since I had enough trouble getting through the prerequisite and I had no plans to become a microwave engineer. He asked me the following questions- Professor: "How fast is your computer at home?" - Me:"2.8GHz.", Professor: "OK, and what is considered the start of the microwave band?" - Me: "1 GHz" Professor: "So, I will see you next semester?" It turned out to be a great course, we built antennas, resonators, did simulations and compared to physical results, talked about a lot of different situations. In my career as a software engineer I have made great use of that knowledge by being able to better understand the issues that my EE counterparts are working on.
- cushychicken 6y ago
  At high enough frequencies digital design becomes analogue design. It's a great time to be a digital engineer. Projects like the PocketVNA and the NanoVNA-V2 make it feasible for you to do professional grade S parameter measurements of digital traces at home. Pair them with QucsStudio, and you have a near professional grade high speed simulation package available for less than $500. So cool. And, speaking of Rules of Thumb - Embedded Artistry's page calls out Eric Bogatin's Signal Integrity Rules of Thumb as well, which is a fantastic series of blog posts on high speed digital design and signal integrity. Worth a read if high speed digital design is your thing.
- yitchelle 6y ago
  Very true. Only depends if its thermal behavior matches your application's needs.
- gHosts 6y ago
  Had a GPS that started jabbering crap when it got cold.... showed up as system dying because it was being flooded with events, ie. Some other task died because it wasn't getting enough cpu time.
barbegal 6y ago
>Keep ISRs small I disagree. If you have interrupt priorities then you can treat your interrupt service routines as the highest priority tasks. As an example lets imagine a drone controller which has two functions it needs to perform in real time: it needs to vary the PWM signal to the motor controller and it needs to acknowledge radio packets. An interrupt is raised when the accelerometer has new data and another is raised when the radio has received a packet. You could write ISRs which just queues the accelerometer data and radio packets so they can be dealt with by a flight control task and a radio task respectively. Alternatively, you could run the whole of the flight controller inside the accelerometer ISR. This reduces copying of data and context switches. Because the accelerometer ISR now takes longer you need to use the radio ISR to acknowledge the radio packets itself. You split the radio task into two tasks: a low priority task for processing commands and the ISR task which does the acknowledgements. > Avoid blocking function calls This can be rewritten as never call functions that block on an action of a lower priority task. It leads to priority inversion in normal tasks and deadlocks in ISRs.
- nsajko 6y ago
  > I disagree. If you have interrupt priorities then you can treat your interrupt service routines as the highest priority tasks. ... I think "Keep ISRs small" is still a good guideline for most cases. Big interrupt handlers require more system-wide knowledge to verify (e.g., stack space, maximal latency, priority problems, ...). They introduce more coupling into the system, in other words. Of course, this should not be some hard rule, rather just a rule of thumb. I wouldn't like it if some policy like this prevented you from choosing the optimal solution in a case like you suggested.
  - barbegal 6y ago
    > Big interrupt handlers require more system-wide knowledge to verify I think the point I'm trying to make is that sometimes following the rule of "make your ISR small" actually makes this worse because your bigger high priority task can be interrupted or you turn interrupts off or mask interrupts and you end up with even more complexity.
bfrog 6y ago
I feel like half of what's here points to using rust or a rust like language as a good thing. Even I suppose C++ if you can use the pointer types to help yourself.
- nsajko 6y ago
  > I feel like half of what's here points to using rust or a rust like language as a good thing. Which points exactly? Also, neither the Rust standard library, nor the C++ std::shared_ptr were designed with embedded in mind. Also, Rust is kind of experimental, not really a good choice for a system that is difficult to update. Not to mention only Intel and AMD64 platforms have Rust Tier 1 support.
  - elwes5 6y ago
    One thing to keep in mind with embedded is size. Some projects you do not get to have std lib. It just does not fit. This is less and less of an issue as time goes on. But a few years ago 2k total memory (flash and RAM) was a real issue you had to conform against. Even that could seem huge for some platforms. Depending on the platform Rust or C++ may not be the first one I reached for. C is pretty compact if you strip the libs out. But usually at that point you may have to goto ASM just to make it fit. Had one project which had to be in python (platform dictated by customer). Did all the right style guide things. Classes, the works. Had to toss all of that out. As just by making classes subjected it to a 300 byte overhead per object. I had hundreds of the things. Half my memory was being used by object management. I needed that space for data. Out goes all the cool by the book things that seem right to do. Borderline a total re-write. I took those lessons to the C version. Right up until the new guy decided everything needed to be C++ and use std:lib and would not listen to me. Suddenly it did not even fit in flash and RAM even if you use both by about 5x much less any data needed. He had to spend weeks backing the changes out again. Even after re-stripping it even then the code size dictated we just could not use some platforms. Another thing to keep in mind is compiler maturity. C++ in the past 10 years has come a long way. But in these embedded platforms you may be working with a C++ tool-chain from the late 1990s (if you are lucky). Some of the toolchains shipped with these chips are in poor shape. They will never change unless you spend a lot of time bringing it up to something semi current. Getting a Rust tool chain on it? Maybe if you spent months messing around with it. Months where you could spend shipping product.
nsajko 6y ago
One of the things from the list that could be improved is this: > Algorithmic optimizations have a greater impact than micro optimizations This is kind of tautological, as micro optimizations can be an important part of algo optimization. We don't execute algorithms on formal automata, but on real machines with real architectures, limitations and opportunities. > * “Real efficiency gains come from changing the order of complexity of the algorithm, such as changing from O(N^2) to O(N*logN) complexity” Only if the constant factor that multiplies the lower-complexity expression is small enough and N is large enough.
- rightbyte 6y ago
  Ye ... this O() obsession seem to miss the "as n approach infinity" part. It is just a polynomial and the constant term or which ever can be dominating for small n. Especially relevant in embedded where searching small static arrays is a thing.
DudeInBasement 6y ago
Missing the most important Rule of Thumb. This is an embedded device. It has a _known_ purpose. Every software design decision should be based around this. This clears up any code goes here (ISR/Task), or 'Use this algo'. Literally the main point of 'Embedded'.
- gHosts 6y ago
  If your device doesn't change the way your customer does business, why bother? Every truly successful device will be used for purposes you didn't dream of when you first created it.
- froh 6y ago
  Yes and no. Yes for many embedded microcontrollers. Not so sure for larger devices with possibly several ECUs and in the field upgrades. All of the sudden the whole thing becomes a tiny data center with several nodes, and the architect needs to value some flexibility over clear cut single purpose. In automotive, embedded is transformed in this way.
roland35 6y ago
This is a great list of some basic rules of thumb! Over the years I have encountered many of these in practice. I think it is important as engineers to always try and keep an eye on the big picture. Many times it is easy to get bogged down in an interesting technical problem in embedded engineering, and we can forget how that problem fits in with the overall state of the project. One example was I found that I could not send more than 1 USB bulk packet per millisecond using Texas Instruments' USB library, even though in theory you could send 19 per millisecond. I wasted a few days to try and fix that, but it was a waste of time because speeding up USB only sped up firmware updates which were not a big deal! There are many examples like this unfortunately. I think the biggest thing missing from this list is how important it is to thoughtfully design the hardware to make the firmware easier to design. This requires a less agile approach, or more flexible prototype hardware until requirements are better understood. Just one example is using separate SPI/I2C busses for separate peripherals, even though technically they can be shared. This helps you not have to worry at all about collisions or managing Direct Memory Interfaces (DMA).
- bradstewart 6y ago
  Definitely agree with thoughtful hardware design. Simple things like good placement of test points to attach scopes on relevant signals is extremely helpful.
  - roland35 6y ago
    Yes test points are important for absolutely everything on the first iteration of a new hardware design, you just have no idea which ones you need before starting the project! I think it is also important to have a big enough test point to actually use too! There have been many times I have been trying to hold a probe to a tiny little exposed circle pad, just wishing it was something big enough to clip a lead to.
- astrobe_ 6y ago
  I don't think solving the USB issue was a total waste of time. Knowing what is going on is always valuable. The "My USB throughput is not what it should be... Oh, well. Must be the USB cable..." attitude is something that can bite you later, because it can be the symptom of a bug (e.g. packets lost because of a buggy ISR) that's just waiting the worst moment to show its ugly face.
cushychicken 6y ago
Embedded Artistry's blog is worth a sub if you are in the embedded world, and aren't reading already.
uterm 6y ago
Thanks for posting this! As an electronics enthusiast, this really sheds a lot of light on different aspects of hardware/software development that I hadn't been exposed to before. I feel like there is a lot of tribal knowledge that people in the business or going to school for EE stuff are privy to that isn't always out there for the rest of us, so kudos for aggregating some of that. along the same line, does anyone have further articles on hardware topics, like sourcing parts for commercial products? It's another part of electrical engineering and bringing products to market that I haven't seen much of online but would love to learn more about.
- nsajko 6y ago
  Not sure if this is what you want, but it's a short read anyway; search for Second Sources: * http://www.ganssle.com/tem/tem288.html http://www.ganssle.com/tem/tem288.html * http://www.ganssle.com/tem/tem289.html http://www.ganssle.com/tem/tem289.html
- HeyLaughingBoy 6y ago
  > sourcing parts for commercial products? That's a topic onto itself. Is there something specific you're thinking about?
  - uterm 6y ago
    For sure, I totally realize those are huge topics, and many people have full time jobs just focusing one one aspect of that. I guess what I'm really looking for is a good overview of standard operating procedures and rules of thumb for purchasing parts and inventory management. I've seen a lot of really cool hobby projects turned small run commercial products in the music/synthesizer world and I guess I'm curious how you would go about sourcing parts for something like that in way that's close to how it's done professionally. Also, more specifically what sort of tools do people typically use for managing inventory? How about comparing prices/vendors? I imagine something more sophisticated than a simple spreadsheet would be in order and I have to figure it's different from company to company, but are their any widely accepted industry standard tool?
gHosts 6y ago
> Programs without documentation have no value. Counterpoint. Your users won't read the documentation. Get used to it. Odds on your developers won't either. Take care when you do read the documentation, it's probably out of date and / or wrong.

168 points by nsajko 6y ago

With the exception of some of the rules under the Hardware section, this is some great succinct advice on software development in general :)

jpm_sd 6y ago
Well, they're all good rules, but nobody is going to remember them in the form of a cluttered bulleted list. Jack Ganssle is quoted repeatedly - his blog has a lot of good war stories that illustrate many of these points in much greater detail. http://www.ganssle.com/blog/index.html http://www.ganssle.com/blog/index.html
- disposedtrolley 6y ago
  Agreed, but the intention appears to serve as a launchpad for digging into the rules yourself. Links to original sources would make this much easier of course. Thanks for the link to Ganssle’s blog. I’ll be sure to have a read today.
projektfu 6y ago
Almost has too much general advice. The embedded parts are easy to skim over.

PoachedSausage 6y ago

> Every sensor is a temperature sensor. Some sensors measure other things as well. Some components can become sensors under the right conditions. Example: Components becoming microphonic. At high enough frequencies digital design becomes analogue design.

bfrog 6y ago
So true!
microtherion 6y ago
> At high enough frequencies digital design becomes analogue design. I've heard this stated as "the difference between digital engineers and analog engineers is that the analog engineers KNOW when they're designing an antenna".
- craftinator 6y ago
  > "the difference between digital engineers and analog engineers is that the analog engineers KNOW when they're designing an antenna". Hahaha this is a beautiful quote! I didn't believe this was a real concern until I ran into it in the real world.
turbinerneiter 6y ago
I read a great how-to once where someone described a device which blinks in the dark - and only in the dark - that uses the LED itself as light sensor.
- cjsawyer 6y ago
  That is pure evil.
- PoachedSausage 6y ago
  At work we replace an obsolete diode in part of a circuit in front of a high gain amplifier. The new equivalent diode came in a glass package unlike the old one. It caused a bit of head scratching to figure out why the circuit wasn't behaving as it should and also appeared to be acting as a human proximity detector.
Cerium 6y ago
That is how I was convinced to take a microwave engineering course. I was an EE undergrad, and had just struggled through the electrodynamics course. My professor asked me to take his microwave engineering course the following semester. I said I was not planning on it, since I had enough trouble getting through the prerequisite and I had no plans to become a microwave engineer. He asked me the following questions- Professor: "How fast is your computer at home?" - Me:"2.8GHz.", Professor: "OK, and what is considered the start of the microwave band?" - Me: "1 GHz" Professor: "So, I will see you next semester?" It turned out to be a great course, we built antennas, resonators, did simulations and compared to physical results, talked about a lot of different situations. In my career as a software engineer I have made great use of that knowledge by being able to better understand the issues that my EE counterparts are working on.
cushychicken 6y ago
At high enough frequencies digital design becomes analogue design. It's a great time to be a digital engineer. Projects like the PocketVNA and the NanoVNA-V2 make it feasible for you to do professional grade S parameter measurements of digital traces at home. Pair them with QucsStudio, and you have a near professional grade high speed simulation package available for less than $500. So cool. And, speaking of Rules of Thumb - Embedded Artistry's page calls out Eric Bogatin's Signal Integrity Rules of Thumb as well, which is a fantastic series of blog posts on high speed digital design and signal integrity. Worth a read if high speed digital design is your thing.
yitchelle 6y ago
Very true. Only depends if its thermal behavior matches your application's needs.
gHosts 6y ago
Had a GPS that started jabbering crap when it got cold.... showed up as system dying because it was being flooded with events, ie. Some other task died because it wasn't getting enough cpu time.

barbegal 6y ago

>Keep ISRs small I disagree. If you have interrupt priorities then you can treat your interrupt service routines as the highest priority tasks. As an example lets imagine a drone controller which has two functions it needs to perform in real time: it needs to vary the PWM signal to the motor controller and it needs to acknowledge radio packets. An interrupt is raised when the accelerometer has new data and another is raised when the radio has received a packet. You could write ISRs which just queues the accelerometer data and radio packets so they can be dealt with by a flight control task and a radio task respectively. Alternatively, you could run the whole of the flight controller inside the accelerometer ISR. This reduces copying of data and context switches. Because the accelerometer ISR now takes longer you need to use the radio ISR to acknowledge the radio packets itself. You split the radio task into two tasks: a low priority task for processing commands and the ISR task which does the acknowledgements. > Avoid blocking function calls This can be rewritten as never call functions that block on an action of a lower priority task. It leads to priority inversion in normal tasks and deadlocks in ISRs.

nsajko 6y ago
> I disagree. If you have interrupt priorities then you can treat your interrupt service routines as the highest priority tasks. ... I think "Keep ISRs small" is still a good guideline for most cases. Big interrupt handlers require more system-wide knowledge to verify (e.g., stack space, maximal latency, priority problems, ...). They introduce more coupling into the system, in other words. Of course, this should not be some hard rule, rather just a rule of thumb. I wouldn't like it if some policy like this prevented you from choosing the optimal solution in a case like you suggested.
- barbegal 6y ago
  > Big interrupt handlers require more system-wide knowledge to verify I think the point I'm trying to make is that sometimes following the rule of "make your ISR small" actually makes this worse because your bigger high priority task can be interrupted or you turn interrupts off or mask interrupts and you end up with even more complexity.

bfrog 6y ago

I feel like half of what's here points to using rust or a rust like language as a good thing. Even I suppose C++ if you can use the pointer types to help yourself.

nsajko 6y ago
> I feel like half of what's here points to using rust or a rust like language as a good thing. Which points exactly? Also, neither the Rust standard library, nor the C++ std::shared_ptr were designed with embedded in mind. Also, Rust is kind of experimental, not really a good choice for a system that is difficult to update. Not to mention only Intel and AMD64 platforms have Rust Tier 1 support.
- elwes5 6y ago
  One thing to keep in mind with embedded is size. Some projects you do not get to have std lib. It just does not fit. This is less and less of an issue as time goes on. But a few years ago 2k total memory (flash and RAM) was a real issue you had to conform against. Even that could seem huge for some platforms. Depending on the platform Rust or C++ may not be the first one I reached for. C is pretty compact if you strip the libs out. But usually at that point you may have to goto ASM just to make it fit. Had one project which had to be in python (platform dictated by customer). Did all the right style guide things. Classes, the works. Had to toss all of that out. As just by making classes subjected it to a 300 byte overhead per object. I had hundreds of the things. Half my memory was being used by object management. I needed that space for data. Out goes all the cool by the book things that seem right to do. Borderline a total re-write. I took those lessons to the C version. Right up until the new guy decided everything needed to be C++ and use std:lib and would not listen to me. Suddenly it did not even fit in flash and RAM even if you use both by about 5x much less any data needed. He had to spend weeks backing the changes out again. Even after re-stripping it even then the code size dictated we just could not use some platforms. Another thing to keep in mind is compiler maturity. C++ in the past 10 years has come a long way. But in these embedded platforms you may be working with a C++ tool-chain from the late 1990s (if you are lucky). Some of the toolchains shipped with these chips are in poor shape. They will never change unless you spend a lot of time bringing it up to something semi current. Getting a Rust tool chain on it? Maybe if you spent months messing around with it. Months where you could spend shipping product.

nsajko 6y ago

One of the things from the list that could be improved is this: > Algorithmic optimizations have a greater impact than micro optimizations This is kind of tautological, as micro optimizations can be an important part of algo optimization. We don't execute algorithms on formal automata, but on real machines with real architectures, limitations and opportunities. > * “Real efficiency gains come from changing the order of complexity of the algorithm, such as changing from O(N^2) to O(N*logN) complexity” Only if the constant factor that multiplies the lower-complexity expression is small enough and N is large enough.

rightbyte 6y ago
Ye ... this O() obsession seem to miss the "as n approach infinity" part. It is just a polynomial and the constant term or which ever can be dominating for small n. Especially relevant in embedded where searching small static arrays is a thing.

DudeInBasement 6y ago

Missing the most important Rule of Thumb. This is an embedded device. It has a _known_ purpose. Every software design decision should be based around this. This clears up any code goes here (ISR/Task), or 'Use this algo'. Literally the main point of 'Embedded'.

gHosts 6y ago
If your device doesn't change the way your customer does business, why bother? Every truly successful device will be used for purposes you didn't dream of when you first created it.
froh 6y ago
Yes and no. Yes for many embedded microcontrollers. Not so sure for larger devices with possibly several ECUs and in the field upgrades. All of the sudden the whole thing becomes a tiny data center with several nodes, and the architect needs to value some flexibility over clear cut single purpose. In automotive, embedded is transformed in this way.

roland35 6y ago

This is a great list of some basic rules of thumb! Over the years I have encountered many of these in practice. I think it is important as engineers to always try and keep an eye on the big picture. Many times it is easy to get bogged down in an interesting technical problem in embedded engineering, and we can forget how that problem fits in with the overall state of the project. One example was I found that I could not send more than 1 USB bulk packet per millisecond using Texas Instruments' USB library, even though in theory you could send 19 per millisecond. I wasted a few days to try and fix that, but it was a waste of time because speeding up USB only sped up firmware updates which were not a big deal! There are many examples like this unfortunately. I think the biggest thing missing from this list is how important it is to thoughtfully design the hardware to make the firmware easier to design. This requires a less agile approach, or more flexible prototype hardware until requirements are better understood. Just one example is using separate SPI/I2C busses for separate peripherals, even though technically they can be shared. This helps you not have to worry at all about collisions or managing Direct Memory Interfaces (DMA).

bradstewart 6y ago
Definitely agree with thoughtful hardware design. Simple things like good placement of test points to attach scopes on relevant signals is extremely helpful.
- roland35 6y ago
  Yes test points are important for absolutely everything on the first iteration of a new hardware design, you just have no idea which ones you need before starting the project! I think it is also important to have a big enough test point to actually use too! There have been many times I have been trying to hold a probe to a tiny little exposed circle pad, just wishing it was something big enough to clip a lead to.
astrobe_ 6y ago
I don't think solving the USB issue was a total waste of time. Knowing what is going on is always valuable. The "My USB throughput is not what it should be... Oh, well. Must be the USB cable..." attitude is something that can bite you later, because it can be the symptom of a bug (e.g. packets lost because of a buggy ISR) that's just waiting the worst moment to show its ugly face.

cushychicken 6y ago

Embedded Artistry's blog is worth a sub if you are in the embedded world, and aren't reading already.

uterm 6y ago

Thanks for posting this! As an electronics enthusiast, this really sheds a lot of light on different aspects of hardware/software development that I hadn't been exposed to before. I feel like there is a lot of tribal knowledge that people in the business or going to school for EE stuff are privy to that isn't always out there for the rest of us, so kudos for aggregating some of that. along the same line, does anyone have further articles on hardware topics, like sourcing parts for commercial products? It's another part of electrical engineering and bringing products to market that I haven't seen much of online but would love to learn more about.

nsajko 6y ago
Not sure if this is what you want, but it's a short read anyway; search for Second Sources: * http://www.ganssle.com/tem/tem288.html http://www.ganssle.com/tem/tem288.html * http://www.ganssle.com/tem/tem289.html http://www.ganssle.com/tem/tem289.html
HeyLaughingBoy 6y ago
> sourcing parts for commercial products? That's a topic onto itself. Is there something specific you're thinking about?
- uterm 6y ago
  For sure, I totally realize those are huge topics, and many people have full time jobs just focusing one one aspect of that. I guess what I'm really looking for is a good overview of standard operating procedures and rules of thumb for purchasing parts and inventory management. I've seen a lot of really cool hobby projects turned small run commercial products in the music/synthesizer world and I guess I'm curious how you would go about sourcing parts for something like that in way that's close to how it's done professionally. Also, more specifically what sort of tools do people typically use for managing inventory? How about comparing prices/vendors? I imagine something more sophisticated than a simple spreadsheet would be in order and I have to figure it's different from company to company, but are their any widely accepted industry standard tool?

gHosts 6y ago

> Programs without documentation have no value. Counterpoint. Your users won't read the documentation. Get used to it. Odds on your developers won't either. Take care when you do read the documentation, it's probably out of date and / or wrong.