7 ms·

Embedded-systems programming is not taught, and no one is willing to pay for training. The result is that development is outsourced to entities that claim, fal

Embedded-systems programming is not taught, and no one is willing to pay for training. The result is that development is outsourced to entities that claim, falsely, to have the knowledge. Eventually the consequences of the fact that they do not have the knowledge surface in an undeniable manner, and the only way to cover is to make a great show of a fresh start. (This affects all industries, not just automotive, but right now that is where the spotlight shines.)

tehjoker 1y ago
I'm not sure this is exactly the problem. It sounds like turning the car into a platform with changeable parts has caused both organizational and technical problems. To be fair, im still not sold that this is an advancement except maybe in simplifying the number of components. I'd prefer the car to work without "updates" and DLC. Why does my car need a firewall??
- cosmicgadget 1y ago
  It needs two! One to keep engine fires out of the passenger compartment[1] and one to keep unauthorized users or code out of your infotainment and control systems. [1] https://wikipedia.org/wiki/Firewall_(engine) https://wikipedia.org/wiki/Firewall_(engine)
  - DonHopkins 1y ago
    All cars and should be equipped with two firewall extinguishers, one for the network and one for the passenger compartment.
- AlotOfReading 1y ago
  It's not practical to produce a car that never needs updates. That would be a bug-free system, which is impossible. Since they're going to ship updates anyway, a lot of focus is on minimizing the cost and hence OTA. For what it's worth, I work in this industry and the general rule of thumb is that every increase in validation from QM (standard quality) up to the various levels of safety critical code has up to 10x the cost per line of code of the previous level.
  - School-Cotton 1y ago
    > It's not practical to produce a car that never needs updates Exactly that was done for decades.
    - deleted 1y ago
      [deleted]
    - miohtama 1y ago
      But those cars are no longer competitive. There is only a marginal buyer group who wants to drive these "bricks", which would also unlikely pass the requirements set for new cars.
    - bronson 1y ago
      Until 1994, the year of the first software-only recall, maybe. Things have changed. Heck, manufacturers were issuing service bulletins to fix the fuel maps in their cars in the 1980s.
    - AlotOfReading 1y ago
      It was not. Recalls have included software updates (sometimes via component replacement) since ECUs became common in the 1980s. Reverse engineering the binaries and flashing updated parameters is actually how ECU tuning used to be done.
    - deleted 1y ago
      [deleted]
  - tehjoker 1y ago
    If you get updates at the dealership, you don’t need a network firewall.
  - rjsw 1y ago
    Updating the software in the computers that control the car has traditionally been combined with providing diagnostic support for it through the dealerships, not done OTA. Having an OBDII connector has been mandated in vehicles for a long time, you plug something into it that lets you either listen to CAN bus traffic or reprogram an individual Electronic Control Unit (ECU). Now that all vehicles have entertainment systems connected to the internet, I guess it is tempting to use that to reprogram ECUs, I haven't been working in this area recently though. The first use case of connecting entertainment systems to a vehicle bus that I can remember was to read some engine settings and turn up the volume on the radio at higher speeds.
tonetegeatinst 1y ago
This is all the more frustrating as I'm in the security side if IT, and have been trying to teach myself C and assembly for embedded development and understanding how malware and vulnerability exist in this ecosystem and how I can help address these issues.
- maldev 1y ago
  You can find router firmware sourcecode online and find pretty egregious vulnerabilities if you're really trying to learn. Alot of embedded stuff is outsourced and doesn't want to waste the computing power for stuff like stack canaries. I recall the following from making a tool for a dlink? router? //Reads a file name foo ReadFilePath() { // Get file name // TICKET 21321: Fixed crash by increasing buffer size char FilePath[100]; ReadFileName(&FilePath); } It sticks out to me, since the crash was clearly from a buffer overflow, and they had this documented in the source code that increasing the buffer size fixes it. What they didn't realize was that the bug would still happen and you could get a buffer overflow from this and do whatever you wanted. This is the level of programmer you're dealing with who's writing embedded software in an overseas sweatshop. And the talent isn't even there domestically since they're severely underpaid compared to someone writing simple javascript.
  - FrankWilhoit 1y ago
    The people who actually can do it are not underpaid. These days they are brought in to do cleanup. They can name their price and pick their assignments.
    - HideousKojima 1y ago
      >The people who actually can do it are not underpaid. The pipelines to create more such people are sorely lacking though
      - jmb99 1y ago
        Only if you’re looking for top dollar when you graduate. Which unfortunately a lot of people are. It makes sense, most people pick this field for the salary and not out of passion (like the vast majority of professions). But if you take a couple C/assembly/systems electives, look for internships at hardware companies, build a couple of toy projects on the side, and graduate with even a modicum of embedded experience, there will be companies that will hire you, pretty much guaranteed. You won’t be making 250k out of the gate, but you should still be making a more-than-livable salary (and frequently in a lower cost of living area than, say, the Bay), and if you pick companies correctly, you can be working with and learning from some truly genius engineers. The pipeline’s there, it’s just not attractive (read: $$$$$$$) enough to pull in most people in the industry.
tcmart14 1y ago
I took an embedded course in university where we programmed the AVR AtMega 328p on the Arduino UNO not using the Arduino libraries and compiler. Make files and setting up an environment. But yea, a single class probably isn't sufficient and also I image a lot of embedded companies have a preference to hire someone already familiar with the chip they are targeting and the toolchain for the stack. I also see a lot of asking for experience with RTOS, which in my class, we didn't use an RTOS.
- nickff 1y ago
  RTOS-based development varies significantly from RTOS to RTOS, so I’m not sure how much it’d help to learn to use one. On the other hand, most fundamental OS knowledge is fully transferable to RTOS, so that would be helpful for embedded developers to understand.
  - ghaff 1y ago
    Yes, there’s a ton of specificity. Could probably say that about kernel dev too. But there is a ton of things people do that’s a lot more generalized. Of course I’ve used very little of specific things I got tested on in my day to day over the years.
- FrankWilhoit 1y ago
  Programming embedded devices is not the same thing as "embedded-systems programming". The latter means, first and foremost, that the software is not allowed to crash, ever, for any reason, else it is people's lives. I did some initial requirements work on a system to monitor continuous-web papermaking machinery; the line had to be stopped, physically and completely, within 100ms if anything went wrong, because an uncontained web of paper can literally cut people in half. They wanted, in order to be able to hire, to use one of the embedded flavors of a well-known consumer-grade OS, and I had to prove to them that there was no way to make any of them safe, at any cost. And they knew their hardware, because they had built it themselves. The absolute last resort is a watchdog timer that hits the reset button if N milliseconds go by without the software telling it it's okay. This is what you have to implement if you are dealing with buggy and undocumented hardware -- as, all too often, you are. Sometimes you can get some doco for $ and an NDA, but then in order to get the real doco it is much more $$$ and a much tighter NDA, and the existence of that option is not even divulged until after things have already gone very far south. If it were only a matter of reading the top-level doco for this or that chip, there would be no issue.
blueflow 1y ago
It is safe to say that Computer Engineering has a problem with enabling knowledge transfer.
- jauntywundrkind 1y ago
  Yes! But it's also obvious that the industry doesn't have a prayer to ever reform. Stuck between proprietary and NDA's chips everywhere, using proprietary and NDA's toolchains and development kits, to product proprietary DRM'ed products. This is an industry that is about as far from the light of science & enlightenment as it is possible to get, ensnared as deeply in the entangling anti-human anti-science Intellectual Property qualgmire-hell as can be got. Oh sure plenty of science goes it! It's fantastically interesting & technical! But aside from some Application Notes write-ups trying desperately to help move the practice along, move it out of jank, knowledge goes in, but it doesn't ever come out! There's such a lack of peershios with which to practice science, to report your findings to, to replicate works on. The software world talks about its patterns and practices. The biggest industries on the planet are building software like wild AND are mad into open source. But... computer engineering is the shadowland, where no talk nor victories that happen there are allowed to be shared, where nothing escapes confinement. What a fucking plagued awful land of people unable to ever do the right thing, unable to bring their work out of the dark & into real civilization.
  - nyarlathotep_ 1y ago
    I can find 1000s of posts or blogs or whatever on every React nuance, Rust thing, LLM trend, or whatever, but nothing even describing what "real" embedded programming looks like in any fashion (I'm not counting blinking an Arduino LED here). What does writing ABS module software look like? I'd actually love to know--it's not an area where you can "vibe code" your way to a 'working' product.
    - gmueckl 1y ago
      Can't speak for cars specifically, but other software built to comparable standards requires a super rigorous process including detailed specs, risk analysis and mitigation, detailed design, comprehensive automated and manual testing against the soec including test coverage, tons of documentation on everything and finally a third party certification on everything. The process is so far removed from typical web and business slop that it's an entirely different world of its own.
odiroot 1y ago
I've been taught 8051 programming at my university. But I'm an older Gen Y, this could be going away for all I know.
- dmoy 1y ago
  Embedded programming is definitely still taught... in EE.
kevin_thibedeau 1y ago
Automotive has the problem of overwrought frameworks and no-code tooling that make it hard to fix problems and make improvements. Once the original devs are burned out or laid off the codebase rots and gets handed off to maintenance devs who barely know how anything works. I'm waiting for a recall fix for the underpowered Sync 2.5 system to correct a backup camera problem. I'm not looking forward to worsening of all the current bugs with USB audio file playback that cause the UI to hang or not show a fully rendered display.
bluedino 1y ago
Everything is just outsourced to the lowest bidder anyway
tonyhart7 1y ago
ok but why tho??? I have a lot of interest in embedded system can someone tell me if there are any course that taught this??
- jeffrallen 1y ago
  I learned embedded in the school of hard knocks.
  - tonyhart7 1y ago
    Yeah but I want to make my time effecient, because failure which cost hardware can be expensive
bitwize 1y ago
Companies are not willing to pay what the people who know embedded deserve. $150,000, $200,000 and up for a JavaScript webshit "engineer", $100,000 max if you work in embedded, unless you have a super specialist knowledge maintaining software on NASA's remaining PDP-11s or whatever that they can't afford to lose.
- jmb99 1y ago
  Fortunately that is incorrect. I mentioned in another comment, but I’m well over $100k USD equivalent in salary alone as an embedded engineer, working in a relatively low cost of living area in Canada, graduated 3 years ago. Working for a “regular” company. Maybe things just really suck for embedded in the states? But since my last year of university I’ve been inundated with recruiters for embedded positions, and I’ve never had a problem finding work. ~75th percentile in salary alone for software engineers in my area, ~55th-60th for Canada. I make more than every JS developer I know who graduated with me, except for the ones who moved to Seattle, Vancouver, or the Bay.
  - bitwize 1y ago
    That's good to hear. Maybe Canada's got its head screwed on straight when it comes to assigning value to software roles.
sarchertech 1y ago
My CS degree concentration is embedded systems. I love embedded programming, but it would probably cost me $200k a year to do it versus the backend distributed systems stuff I do now.
- jmb99 1y ago
  Admittedly I don’t know your salary or market, but it is possible to make decent money in embedded. Connections & market timing are both vital though, in my experience, as well as being actually good at your job. I’m in Canada so numbers are way different, but salary-wise I’m in the ~75th percentile software engineers in my area, my title is embedded engineer, and I’m fairly junior (3 years out of university, ~6 years full-time experience). I’m working with some other embedded people who are in the 95th percentile for software engineers in the country. The main problem is there are very few high-paying embedded jobs; conversely though, there seem to be even fewer highly-skilled embedded engineers looking for work. I recently interviewed at a company paying 50th-85th (based on experience) percentile trying to hire pretty much any competent embedded engineer, and their problem isn’t insufficient salary, it’s just a lack of applicants or any skill level. From what I’ve heard, the same seems to be true pretty much everywhere. Now sure, if you’re looking for 500k+ jobs, embedded isn’t the area to be in, unfortunately. But I prefer low-stress, fun-environment embedded jobs, and don’t mind trading off salary for that. Different strokes.
  - Zanfa 1y ago
    Problems like this always come down to salary. I love embedded (hardware in general, really) and would absolutely love to do it, but during my entire career, the salaries for embedded have been so much lower than you get for slinging JS/web shit. And now with 15 years in, the gap is even worse. At this point, when I wanted to get back into hardware, it made more financial sense to outfit my home office with all the measuring instruments, debuggers, tools and other equipment necessary for embedded work and do it as a hobby. If I had the space, I could even get full-size CNC machines and still come out ahead cash wise. It’s insane. It’s no wonder they can’t find experienced embedded devs, when it makes no financial sense to stick with it over a decade.

tehjoker 1y ago

I'm not sure this is exactly the problem. It sounds like turning the car into a platform with changeable parts has caused both organizational and technical problems. To be fair, im still not sold that this is an advancement except maybe in simplifying the number of components. I'd prefer the car to work without "updates" and DLC. Why does my car need a firewall??

cosmicgadget 1y ago
It needs two! One to keep engine fires out of the passenger compartment[1] and one to keep unauthorized users or code out of your infotainment and control systems. [1] https://wikipedia.org/wiki/Firewall_(engine) https://wikipedia.org/wiki/Firewall_(engine)
- DonHopkins 1y ago
  All cars and should be equipped with two firewall extinguishers, one for the network and one for the passenger compartment.
AlotOfReading 1y ago
It's not practical to produce a car that never needs updates. That would be a bug-free system, which is impossible. Since they're going to ship updates anyway, a lot of focus is on minimizing the cost and hence OTA. For what it's worth, I work in this industry and the general rule of thumb is that every increase in validation from QM (standard quality) up to the various levels of safety critical code has up to 10x the cost per line of code of the previous level.
- School-Cotton 1y ago
  > It's not practical to produce a car that never needs updates Exactly that was done for decades.
  - deleted 1y ago
    [deleted]
  - miohtama 1y ago
    But those cars are no longer competitive. There is only a marginal buyer group who wants to drive these "bricks", which would also unlikely pass the requirements set for new cars.
  - bronson 1y ago
    Until 1994, the year of the first software-only recall, maybe. Things have changed. Heck, manufacturers were issuing service bulletins to fix the fuel maps in their cars in the 1980s.
  - AlotOfReading 1y ago
    It was not. Recalls have included software updates (sometimes via component replacement) since ECUs became common in the 1980s. Reverse engineering the binaries and flashing updated parameters is actually how ECU tuning used to be done.
  - deleted 1y ago
    [deleted]
- tehjoker 1y ago
  If you get updates at the dealership, you don’t need a network firewall.
- rjsw 1y ago
  Updating the software in the computers that control the car has traditionally been combined with providing diagnostic support for it through the dealerships, not done OTA. Having an OBDII connector has been mandated in vehicles for a long time, you plug something into it that lets you either listen to CAN bus traffic or reprogram an individual Electronic Control Unit (ECU). Now that all vehicles have entertainment systems connected to the internet, I guess it is tempting to use that to reprogram ECUs, I haven't been working in this area recently though. The first use case of connecting entertainment systems to a vehicle bus that I can remember was to read some engine settings and turn up the volume on the radio at higher speeds.

tonetegeatinst 1y ago

This is all the more frustrating as I'm in the security side if IT, and have been trying to teach myself C and assembly for embedded development and understanding how malware and vulnerability exist in this ecosystem and how I can help address these issues.

maldev 1y ago
You can find router firmware sourcecode online and find pretty egregious vulnerabilities if you're really trying to learn. Alot of embedded stuff is outsourced and doesn't want to waste the computing power for stuff like stack canaries. I recall the following from making a tool for a dlink? router? //Reads a file name foo ReadFilePath() { // Get file name // TICKET 21321: Fixed crash by increasing buffer size char FilePath[100]; ReadFileName(&FilePath); } It sticks out to me, since the crash was clearly from a buffer overflow, and they had this documented in the source code that increasing the buffer size fixes it. What they didn't realize was that the bug would still happen and you could get a buffer overflow from this and do whatever you wanted. This is the level of programmer you're dealing with who's writing embedded software in an overseas sweatshop. And the talent isn't even there domestically since they're severely underpaid compared to someone writing simple javascript.
- FrankWilhoit 1y ago
  The people who actually can do it are not underpaid. These days they are brought in to do cleanup. They can name their price and pick their assignments.
  - HideousKojima 1y ago
    >The people who actually can do it are not underpaid. The pipelines to create more such people are sorely lacking though
    - jmb99 1y ago
      Only if you’re looking for top dollar when you graduate. Which unfortunately a lot of people are. It makes sense, most people pick this field for the salary and not out of passion (like the vast majority of professions). But if you take a couple C/assembly/systems electives, look for internships at hardware companies, build a couple of toy projects on the side, and graduate with even a modicum of embedded experience, there will be companies that will hire you, pretty much guaranteed. You won’t be making 250k out of the gate, but you should still be making a more-than-livable salary (and frequently in a lower cost of living area than, say, the Bay), and if you pick companies correctly, you can be working with and learning from some truly genius engineers. The pipeline’s there, it’s just not attractive (read: $$$$$$$) enough to pull in most people in the industry.

tcmart14 1y ago

I took an embedded course in university where we programmed the AVR AtMega 328p on the Arduino UNO not using the Arduino libraries and compiler. Make files and setting up an environment. But yea, a single class probably isn't sufficient and also I image a lot of embedded companies have a preference to hire someone already familiar with the chip they are targeting and the toolchain for the stack. I also see a lot of asking for experience with RTOS, which in my class, we didn't use an RTOS.

nickff 1y ago
RTOS-based development varies significantly from RTOS to RTOS, so I’m not sure how much it’d help to learn to use one. On the other hand, most fundamental OS knowledge is fully transferable to RTOS, so that would be helpful for embedded developers to understand.
- ghaff 1y ago
  Yes, there’s a ton of specificity. Could probably say that about kernel dev too. But there is a ton of things people do that’s a lot more generalized. Of course I’ve used very little of specific things I got tested on in my day to day over the years.
FrankWilhoit 1y ago
Programming embedded devices is not the same thing as "embedded-systems programming". The latter means, first and foremost, that the software is not allowed to crash, ever, for any reason, else it is people's lives. I did some initial requirements work on a system to monitor continuous-web papermaking machinery; the line had to be stopped, physically and completely, within 100ms if anything went wrong, because an uncontained web of paper can literally cut people in half. They wanted, in order to be able to hire, to use one of the embedded flavors of a well-known consumer-grade OS, and I had to prove to them that there was no way to make any of them safe, at any cost. And they knew their hardware, because they had built it themselves. The absolute last resort is a watchdog timer that hits the reset button if N milliseconds go by without the software telling it it's okay. This is what you have to implement if you are dealing with buggy and undocumented hardware -- as, all too often, you are. Sometimes you can get some doco for $ and an NDA, but then in order to get the real doco it is much more $$$ and a much tighter NDA, and the existence of that option is not even divulged until after things have already gone very far south. If it were only a matter of reading the top-level doco for this or that chip, there would be no issue.

blueflow 1y ago

It is safe to say that Computer Engineering has a problem with enabling knowledge transfer.

jauntywundrkind 1y ago
Yes! But it's also obvious that the industry doesn't have a prayer to ever reform. Stuck between proprietary and NDA's chips everywhere, using proprietary and NDA's toolchains and development kits, to product proprietary DRM'ed products. This is an industry that is about as far from the light of science & enlightenment as it is possible to get, ensnared as deeply in the entangling anti-human anti-science Intellectual Property qualgmire-hell as can be got. Oh sure plenty of science goes it! It's fantastically interesting & technical! But aside from some Application Notes write-ups trying desperately to help move the practice along, move it out of jank, knowledge goes in, but it doesn't ever come out! There's such a lack of peershios with which to practice science, to report your findings to, to replicate works on. The software world talks about its patterns and practices. The biggest industries on the planet are building software like wild AND are mad into open source. But... computer engineering is the shadowland, where no talk nor victories that happen there are allowed to be shared, where nothing escapes confinement. What a fucking plagued awful land of people unable to ever do the right thing, unable to bring their work out of the dark & into real civilization.
- nyarlathotep_ 1y ago
  I can find 1000s of posts or blogs or whatever on every React nuance, Rust thing, LLM trend, or whatever, but nothing even describing what "real" embedded programming looks like in any fashion (I'm not counting blinking an Arduino LED here). What does writing ABS module software look like? I'd actually love to know--it's not an area where you can "vibe code" your way to a 'working' product.
  - gmueckl 1y ago
    Can't speak for cars specifically, but other software built to comparable standards requires a super rigorous process including detailed specs, risk analysis and mitigation, detailed design, comprehensive automated and manual testing against the soec including test coverage, tons of documentation on everything and finally a third party certification on everything. The process is so far removed from typical web and business slop that it's an entirely different world of its own.

odiroot 1y ago

I've been taught 8051 programming at my university. But I'm an older Gen Y, this could be going away for all I know.

dmoy 1y ago
Embedded programming is definitely still taught... in EE.

kevin_thibedeau 1y ago

Automotive has the problem of overwrought frameworks and no-code tooling that make it hard to fix problems and make improvements. Once the original devs are burned out or laid off the codebase rots and gets handed off to maintenance devs who barely know how anything works. I'm waiting for a recall fix for the underpowered Sync 2.5 system to correct a backup camera problem. I'm not looking forward to worsening of all the current bugs with USB audio file playback that cause the UI to hang or not show a fully rendered display.

bluedino 1y ago

Everything is just outsourced to the lowest bidder anyway

tonyhart7 1y ago

ok but why tho??? I have a lot of interest in embedded system can someone tell me if there are any course that taught this??

jeffrallen 1y ago
I learned embedded in the school of hard knocks.
- tonyhart7 1y ago
  Yeah but I want to make my time effecient, because failure which cost hardware can be expensive

bitwize 1y ago

Companies are not willing to pay what the people who know embedded deserve. $150,000, $200,000 and up for a JavaScript webshit "engineer", $100,000 max if you work in embedded, unless you have a super specialist knowledge maintaining software on NASA's remaining PDP-11s or whatever that they can't afford to lose.

jmb99 1y ago
Fortunately that is incorrect. I mentioned in another comment, but I’m well over $100k USD equivalent in salary alone as an embedded engineer, working in a relatively low cost of living area in Canada, graduated 3 years ago. Working for a “regular” company. Maybe things just really suck for embedded in the states? But since my last year of university I’ve been inundated with recruiters for embedded positions, and I’ve never had a problem finding work. ~75th percentile in salary alone for software engineers in my area, ~55th-60th for Canada. I make more than every JS developer I know who graduated with me, except for the ones who moved to Seattle, Vancouver, or the Bay.
- bitwize 1y ago
  That's good to hear. Maybe Canada's got its head screwed on straight when it comes to assigning value to software roles.

sarchertech 1y ago

My CS degree concentration is embedded systems. I love embedded programming, but it would probably cost me $200k a year to do it versus the backend distributed systems stuff I do now.

jmb99 1y ago
Admittedly I don’t know your salary or market, but it is possible to make decent money in embedded. Connections & market timing are both vital though, in my experience, as well as being actually good at your job. I’m in Canada so numbers are way different, but salary-wise I’m in the ~75th percentile software engineers in my area, my title is embedded engineer, and I’m fairly junior (3 years out of university, ~6 years full-time experience). I’m working with some other embedded people who are in the 95th percentile for software engineers in the country. The main problem is there are very few high-paying embedded jobs; conversely though, there seem to be even fewer highly-skilled embedded engineers looking for work. I recently interviewed at a company paying 50th-85th (based on experience) percentile trying to hire pretty much any competent embedded engineer, and their problem isn’t insufficient salary, it’s just a lack of applicants or any skill level. From what I’ve heard, the same seems to be true pretty much everywhere. Now sure, if you’re looking for 500k+ jobs, embedded isn’t the area to be in, unfortunately. But I prefer low-stress, fun-environment embedded jobs, and don’t mind trading off salary for that. Different strokes.
- Zanfa 1y ago
  Problems like this always come down to salary. I love embedded (hardware in general, really) and would absolutely love to do it, but during my entire career, the salaries for embedded have been so much lower than you get for slinging JS/web shit. And now with 15 years in, the gap is even worse. At this point, when I wanted to get back into hardware, it made more financial sense to outfit my home office with all the measuring instruments, debuggers, tools and other equipment necessary for embedded work and do it as a hobby. If I had the space, I could even get full-size CNC machines and still come out ahead cash wise. It’s insane. It’s no wonder they can’t find experienced embedded devs, when it makes no financial sense to stick with it over a decade.