8 ms·

The embedded world is very slow to change, so you can read about "modern approaches" (i.e. approaches used today) in any book about embedded programming written

The embedded world is very slow to change, so you can read about "modern approaches" (i.e. approaches used today) in any book about embedded programming written in the last 30 years. I currently work on spacecraft flight software and the only real advance on this project over something like the space shuttle that I can point to is that we're trying out some continuous integration on this project. We would like to use a lot of modern C++ features, but the compiler for our flight hardware platform is GCC 4.1 (upgrading to GCC 4.3 soon if we're lucky).

rowanG077 7y ago
I find it interesting that such critical code is written in C. Why not use something with a lot more (easily)statically provable properties. Like Rust or Agda?
- MiroF 7y ago
  Because safety critical fields are also slow-moving.
- retrac 7y ago
  I think the answer was right there in their comment. "The compiler for our flight hardware platform is GCC 4.1 (upgrading to GCC 4.3 soon if we're lucky)". Often, the only high-level language available for an embedded platform is a standard C compiler. If you're lucky.
- NobodyNada 7y ago
  Using a newer language carries a lot of risks and challenges for embedded programs: - There’s a high risk of bugs in the compiler/standard library in languages with lots of features - Usually, the manufacturer of an embedded platform provides a C compiler. Porting a new compiler can be a LOT of work, and the resulting port can often be very buggy - Even if you can get a compiler to work, many newer languages rely on a complicated runtime/standard library, which is a deal-breaker when your complete program has to fit in a few kilobytes of ROM
- amw-zero 7y ago
  You’ll find that for very serious, industrial applications, a conservative mindset prevails. C may not be trendy at the moment, but it powers the computing world. Its shortcomings are also extremely well known and also statically analyzable. Also, think about when flight software started being written. Was Rust an option? And once it came out, do you expect that programmers who are responsible for millions of people’s lives to drop their decades of tested code and development practices to make what is a bet on what is still a new language? What I find interesting is this mindset. My conservativeness on a project is directly proportional to its importance / criticality, and I can’t think of anything more important or critical than software that runs on a commercial airplane. C is a small, very well understood language. Of course it gives you nothing in terms of automatic memory safety, but that is one tradeoff in the list of hundreds of other dimensions. When building “important” things it’s important to think about tradeoffs, identify your biases, and make a choice that’s best for the project and the people that the choice will affect. If you told me that the moment anyone dies as a result of my software I would have to be killed, I would make sure to use the most tried-and-true tools available to me.
  - AtlasBarfed 7y ago
    You're advocating throwing baby out with bathwater. Rust interops with C seamlessly, doesn't it? You don't have to throw out good code to use a better language or framework. C may be statically analyzable to some degree, but if Rust's multithreading is truly provable, then new code can be Rust and of course still use the tried and true C libraries. Disclaimer: I still haven't actually learned any Rust, so my logic is CIO-level of potential ignorance.
    - jschwartzi 7y ago
      The issue is that you’re trading a problem space that is very well understood for one that isn’t. Making a safe program in C is all about being explicit about resource allocation and controlling resources. So we tend to require that habit in development. It’s socialized. The only thing you’d be doing is using technology to replace the socialization. And you’d be adding new problems from Rust that don’t exist in the C world. It’s tempting in a lot of cases to read the data sheet and determine that the product is good enough. But there are a lot of engineering and organizational challenges that aren’t written in the marketing documents. Those challenges have to be searched for and social and technological tools must be developed to solve those challenges. As an exercise in use of technology it looks easy but there’s an entire human and organizational side to it that gets lost in discussions on HN.
- bdavis__ 7y ago
  Ada is used a fair amount in high $ projects. Toolchains are expensive, and the C compiler is provided for free from the chip / board vendor.
- hechang1997 7y ago
  Isn't Ada already used in areospace industry?
harryf 7y ago
I wonder if the same is true of Space X?
- monocasa 7y ago
  Yeah. AFAIK they use FreeRTOS for the real deeply embedded stuff which would look very familiar to this discussion.
AlotOfReading 7y ago
Having worked on embedded systems for a decade at this point, the fact that we allow vendors to get away with providing ancient compilers and runtimes is shameful. We know that these old toolchains have thousands of documented bugs, many critical. We know how to produce code with better verification, but just don't push for the tools to do it.
- Baeocystin 7y ago
  Isn't the key part that these older systems have documented bugs? Or, to put it another way, if there's a wasp in the room (and there always is), I'd want to know where it is.
  - AlotOfReading 7y ago
    That doesn't end up being the case for a number of reasons. Firstly, no one is actually able to account of all of these known issues a priori. I don't like calling things impossible, but writing safe C that avoids any compiler bugs is probably best labeled as that. Secondly, vendors make modifications during their release process, which introduces new (and fun!) bugs. You're not really avoiding hidden wasps, just labeling some of them. If you simply moved to a newer compiler, you wouldn't have to avoid them, they'd mostly be gone (or at worst, labeled).
    - Baeocystin 7y ago
      Are the newer compilers truly that much better? I've been working in tech since the 90's, and I can't say that for the tools I've used I've noticed any great improvement in overall quality- bugs get swatted, new ones created in what feels like a constant measure. I am assuming that many optimizations are turned off regardless, due to wanting to keep the resulting assembly as predictable as possible, but I do not work in the embedded space, so this is perhaps a naive question.
  - imtringued 7y ago
    I think the idea is that you don't want a whole wasp nest. Just a bunch of stray wasps.
bargle0 7y ago
How do you do CI/CD for embedded systems?
- deleted 7y ago
  [deleted]
- jakeinspace 7y ago
  CI during the first phases of development in my experience is now often done with modern tooling (gitlab CI, Jenkins), compiling and running tests on a regular Linux x86 build server. Later phases switch over to some sort of emulated test harness, with interrupts coming from simulated flight hardware. Obviously the further along in the development process, the more expensive and slow it is to run tests. Maybe some software groups (SpaceX?) have a process that allows for tight test loops all the way to actual hardware in the loop tests.
- hyldmo 7y ago
  I can't speak for what the rest of the industry does, but some chip manufacturers provide decent emulators, so you can run some tests there. We have also done some hardware tests where we connect our hardware to a raspberry pi or similar and run our CI there. It doesn't replace real-world testing, but it does get us some of the way there.
cblum 7y ago
If you don’t mind me asking, how could one get into this field if they’re already an experienced software engineer in the more “vanilla” stuff (web services, etc.)?

rowanG077 7y ago

I find it interesting that such critical code is written in C. Why not use something with a lot more (easily)statically provable properties. Like Rust or Agda?

MiroF 7y ago
Because safety critical fields are also slow-moving.
retrac 7y ago
I think the answer was right there in their comment. "The compiler for our flight hardware platform is GCC 4.1 (upgrading to GCC 4.3 soon if we're lucky)". Often, the only high-level language available for an embedded platform is a standard C compiler. If you're lucky.
NobodyNada 7y ago
Using a newer language carries a lot of risks and challenges for embedded programs: - There’s a high risk of bugs in the compiler/standard library in languages with lots of features - Usually, the manufacturer of an embedded platform provides a C compiler. Porting a new compiler can be a LOT of work, and the resulting port can often be very buggy - Even if you can get a compiler to work, many newer languages rely on a complicated runtime/standard library, which is a deal-breaker when your complete program has to fit in a few kilobytes of ROM
amw-zero 7y ago
You’ll find that for very serious, industrial applications, a conservative mindset prevails. C may not be trendy at the moment, but it powers the computing world. Its shortcomings are also extremely well known and also statically analyzable. Also, think about when flight software started being written. Was Rust an option? And once it came out, do you expect that programmers who are responsible for millions of people’s lives to drop their decades of tested code and development practices to make what is a bet on what is still a new language? What I find interesting is this mindset. My conservativeness on a project is directly proportional to its importance / criticality, and I can’t think of anything more important or critical than software that runs on a commercial airplane. C is a small, very well understood language. Of course it gives you nothing in terms of automatic memory safety, but that is one tradeoff in the list of hundreds of other dimensions. When building “important” things it’s important to think about tradeoffs, identify your biases, and make a choice that’s best for the project and the people that the choice will affect. If you told me that the moment anyone dies as a result of my software I would have to be killed, I would make sure to use the most tried-and-true tools available to me.
- AtlasBarfed 7y ago
  You're advocating throwing baby out with bathwater. Rust interops with C seamlessly, doesn't it? You don't have to throw out good code to use a better language or framework. C may be statically analyzable to some degree, but if Rust's multithreading is truly provable, then new code can be Rust and of course still use the tried and true C libraries. Disclaimer: I still haven't actually learned any Rust, so my logic is CIO-level of potential ignorance.
  - jschwartzi 7y ago
    The issue is that you’re trading a problem space that is very well understood for one that isn’t. Making a safe program in C is all about being explicit about resource allocation and controlling resources. So we tend to require that habit in development. It’s socialized. The only thing you’d be doing is using technology to replace the socialization. And you’d be adding new problems from Rust that don’t exist in the C world. It’s tempting in a lot of cases to read the data sheet and determine that the product is good enough. But there are a lot of engineering and organizational challenges that aren’t written in the marketing documents. Those challenges have to be searched for and social and technological tools must be developed to solve those challenges. As an exercise in use of technology it looks easy but there’s an entire human and organizational side to it that gets lost in discussions on HN.
bdavis__ 7y ago
Ada is used a fair amount in high $ projects. Toolchains are expensive, and the C compiler is provided for free from the chip / board vendor.
hechang1997 7y ago
Isn't Ada already used in areospace industry?

harryf 7y ago

I wonder if the same is true of Space X?

monocasa 7y ago
Yeah. AFAIK they use FreeRTOS for the real deeply embedded stuff which would look very familiar to this discussion.

AlotOfReading 7y ago

Having worked on embedded systems for a decade at this point, the fact that we allow vendors to get away with providing ancient compilers and runtimes is shameful. We know that these old toolchains have thousands of documented bugs, many critical. We know how to produce code with better verification, but just don't push for the tools to do it.

Baeocystin 7y ago
Isn't the key part that these older systems have documented bugs? Or, to put it another way, if there's a wasp in the room (and there always is), I'd want to know where it is.
- AlotOfReading 7y ago
  That doesn't end up being the case for a number of reasons. Firstly, no one is actually able to account of all of these known issues a priori. I don't like calling things impossible, but writing safe C that avoids any compiler bugs is probably best labeled as that. Secondly, vendors make modifications during their release process, which introduces new (and fun!) bugs. You're not really avoiding hidden wasps, just labeling some of them. If you simply moved to a newer compiler, you wouldn't have to avoid them, they'd mostly be gone (or at worst, labeled).
  - Baeocystin 7y ago
    Are the newer compilers truly that much better? I've been working in tech since the 90's, and I can't say that for the tools I've used I've noticed any great improvement in overall quality- bugs get swatted, new ones created in what feels like a constant measure. I am assuming that many optimizations are turned off regardless, due to wanting to keep the resulting assembly as predictable as possible, but I do not work in the embedded space, so this is perhaps a naive question.
- imtringued 7y ago
  I think the idea is that you don't want a whole wasp nest. Just a bunch of stray wasps.

bargle0 7y ago

How do you do CI/CD for embedded systems?

deleted 7y ago
[deleted]
jakeinspace 7y ago
CI during the first phases of development in my experience is now often done with modern tooling (gitlab CI, Jenkins), compiling and running tests on a regular Linux x86 build server. Later phases switch over to some sort of emulated test harness, with interrupts coming from simulated flight hardware. Obviously the further along in the development process, the more expensive and slow it is to run tests. Maybe some software groups (SpaceX?) have a process that allows for tight test loops all the way to actual hardware in the loop tests.
hyldmo 7y ago
I can't speak for what the rest of the industry does, but some chip manufacturers provide decent emulators, so you can run some tests there. We have also done some hardware tests where we connect our hardware to a raspberry pi or similar and run our CI there. It doesn't replace real-world testing, but it does get us some of the way there.

cblum 7y ago

If you don’t mind me asking, how could one get into this field if they’re already an experienced software engineer in the more “vanilla” stuff (web services, etc.)?