7 ms·

> Employees may be leaving the embedded space Embedded opportunities have been slowly shrinking for years. For whatever combination of reasons, a lot of employ

> Employees may be leaving the embedded space Embedded opportunities have been slowly shrinking for years. For whatever combination of reasons, a lot of employers think that embedded work is easy or otherwise doesn’t require a large budget. It’s increasingly bizarre to get a well-designed IoT device with a very polished mobile app and web UI, then struggle with hardware factory resets and firmware upgrades because the embedded side of the product didn’t get the same level of attention. It’s like embedded somehow became an afterthought in the industry. Perhaps because it’s the only part of the system that doesn’t have a highly polished UI layered on top of it? Over the past decade I’ve witnessed multiple companies over focus on anything that goes well into slide decks (UX mock-ups, animations, etc.) or generates vanity metrics (number of backend servers, requests per second to the cloud) while ignoring anything that doesn’t have a visual pop to it (embedded firmware)

larve 4y ago
One thing I came to see working in both web and embedded for two decades now: a lot of embedded developers often miss the “product” side of what they are building. This probably doesn’t explain the lower pay, but it might be a reason why embedded overall doesn’t get the recognition it deserves: the embedded engineers don’t know how to communicate their value / provide more value to the business. This is becoming increasingly important as you well note, where devices are all connected, and things like setup and updating and connectivity are crucial. Designing not only a robust, but a user-friendly firmware update process is actually a lot more work than just building a bootloader: you need to communicate to the user, in realtime, what is going on. Cancelling an action needs to be immediate and provide feedback on the process of the cancelling. Error handling needs to provide useful information, and probably a special UX. These do need to be factored into the embedded software right from the start, because they significantly increase the complexity, and it’s extremely easy for management to miss how crucial that part is. I keep a few horrible chinese consumer electronics devices on hand (webcam, mp3 player, mobile phone) to show what I mean. The only difference between an ipod touch and a noname mp3 player with touchscreen is… the software. Having to press 3 inaccessible buttons, connect a USB volume named “NO NAME”, have it hang for 2 minutes when unmounting, then show a black screen for 3 more minutes, before showing … that it didn’t update, vs a smoothly progressing update progress bar showing the steps, the devices showing up in my online dashboard as soon as it reboots, that’s what my value as an embedded engineer is.
bsder 4y ago
> It’s increasingly bizarre to get a well-designed IoT device with a very polished mobile app and web UI, then struggle with hardware factory resets and firmware upgrades because the embedded side of the product didn’t get the same level of attention. Why? The issue is that you have to actually ... you know ... PLAN when you have an embedded device. You can churn the app and the web and the backend infinitely so there is no penalty for doing so. If you take that attitude and apply it to embedded you wind up with an expensive pile of scrap.
thewebcount 4y ago
There was a time in the late 90s/early 2000s where this happened to driver development on the (Classic) Mac. Companies would make some USB device and get a reasonable driver made for Windows (I assume - I wasn't using Windows at the time). Then they would say, "Well, MacOS is 10% the market of Windows, so we'll pay 1/10th for someone to develop a driver for this." But it turned out that USB worked completely differently on the Mac from how it did on Windows, so none of the Windows code was relevant at all for the Mac devs. They would either get what they paid for (which was terrible for users) or they would not get a Mac driver. This is around the time I stopped buying any device that required installing a driver. Many of these devices didn't really need one because they were regular USB-spec devices (keyboards, scanners, etc.) To this day, I will not install a driver for a fucking mouse. Why would that be required?
Jtsummers 4y ago
"Programming is just typing" was a typical management refrain when I was in the embedded field (more properly, now I'm adjacent to it). It was frustrating. Computer scientists and programmers aren't Real(tm) Engineers so they don't deserve as much money. They can't be in charge because you can't have engineers answering to non-engineers (ie, very few leads let alone managers coming from the software side). Which leads to a culture that's overly hardware centric with insufficient leadership/management understanding of what software actually entails. Also, "This doesn't meet the current requirements, fix it with software!" The best one was when the case wasn't waterproof... How the fuck is software supposed to fix that? They literally expected the software team to work magic. A lot of pushback got that requirement kicked back over to the mechanical engineering team to address, but it took months. Moronic.
- AtlasBarfed 4y ago
  It might also be that the management chains in embedded are likely former engineers or EE people. In Web you get a lot of management interfaces where the boss can't do their job: that's the perfect recipe for high pay. In embedded and other non-software engineering but engineering firms, the management is typically engineers that CAN do their subordinates jobs, they just don't want to.
  - Jtsummers 4y ago
    That's what I was getting at with this: > They can't be in charge because you can't have engineers answering to non-engineers (ie, very few leads let alone managers coming from the software side). Which leads to a culture that's overly hardware centric with insufficient leadership/management understanding of what software actually entails. In hardware-centric orgs, software developers are a small step above technicians in their pecking order, sometimes below. The pecking order itself is annoying enough, but when you switch from designing your own ASICs to buying COTS dev boards and primarily only adding software to it, you're not really a hardware company anymore. But it'll take another generation for them to realize it, or a severe crunch if someone comes along and realizes that they can pay embedded devs $200k+ and eat the lunch of half these companies.
ryandrake 4y ago
Many hardware companies still see software as just another line item on the BOM: Like a screw or a gasket. It's something you build cheaply or buy from a supplier and sprinkle it on the product somewhere on the assembly line. These hardware companies have no concept of typical measures of software quality, building an ecosystem, release management, sometimes even no concept of source control. They tell an overworked embedded software engineer: "Go build something that barely meets these requirements, ship it, and then throw the scrap away." like metal shavings from their CNC machine.
- commandlinefan 4y ago
  > typical measures of software quality To play devil's advocate - are there any (useful) measures of software quality? Even this place is mostly programmers and we can't even agree whether we should be writing unit tests or not.
  - marmetio 4y ago
    Sort of. There are accurate measures with verifiable predictive power. But useful depends on cost/benefit, which in turn depends on ability to implement and market forces. There's a company that looked at reducing critical defects from a sort of actuarial perspective. They have a few decades of cross-industry data. I've used their model, and it works. If you don't need a numerical result, you can just read the white paper about what's most important [1]. So to partially answer your question: unit testing reduces defects, but reducing defects might not be worth the costs to you. And defects might not be the only thing that matters. There are other measures of goodness, like maintainability, which complicates the answer. You'd have to collect your own data for that. [1] https://missionreadysoftware.com/articles/ https://missionreadysoftware.com/articles/
    - goalieca 4y ago
      I’d say for micro services and large distributed system, you do need a pyramid of testing with most covered at the unit level. The system is just too large and continuously changing as all the different versions of services release.
eschneider 4y ago
Yeah, I sorta specialize in that whole IoT firmware update/fleet monitoring/make sure everything at the edge runs smoothly end of things, and if you find a company that realizes that this is something that MUST work smoothly if they're going to scale, then it's a very sweet place to be. Even better, that sorta work combines low-level C++ with lots of back-end web service work, so you're never 'just' a C++ programmer.

larve 4y ago

One thing I came to see working in both web and embedded for two decades now: a lot of embedded developers often miss the “product” side of what they are building. This probably doesn’t explain the lower pay, but it might be a reason why embedded overall doesn’t get the recognition it deserves: the embedded engineers don’t know how to communicate their value / provide more value to the business. This is becoming increasingly important as you well note, where devices are all connected, and things like setup and updating and connectivity are crucial. Designing not only a robust, but a user-friendly firmware update process is actually a lot more work than just building a bootloader: you need to communicate to the user, in realtime, what is going on. Cancelling an action needs to be immediate and provide feedback on the process of the cancelling. Error handling needs to provide useful information, and probably a special UX. These do need to be factored into the embedded software right from the start, because they significantly increase the complexity, and it’s extremely easy for management to miss how crucial that part is. I keep a few horrible chinese consumer electronics devices on hand (webcam, mp3 player, mobile phone) to show what I mean. The only difference between an ipod touch and a noname mp3 player with touchscreen is… the software. Having to press 3 inaccessible buttons, connect a USB volume named “NO NAME”, have it hang for 2 minutes when unmounting, then show a black screen for 3 more minutes, before showing … that it didn’t update, vs a smoothly progressing update progress bar showing the steps, the devices showing up in my online dashboard as soon as it reboots, that’s what my value as an embedded engineer is.

bsder 4y ago

> It’s increasingly bizarre to get a well-designed IoT device with a very polished mobile app and web UI, then struggle with hardware factory resets and firmware upgrades because the embedded side of the product didn’t get the same level of attention. Why? The issue is that you have to actually ... you know ... PLAN when you have an embedded device. You can churn the app and the web and the backend infinitely so there is no penalty for doing so. If you take that attitude and apply it to embedded you wind up with an expensive pile of scrap.

thewebcount 4y ago

There was a time in the late 90s/early 2000s where this happened to driver development on the (Classic) Mac. Companies would make some USB device and get a reasonable driver made for Windows (I assume - I wasn't using Windows at the time). Then they would say, "Well, MacOS is 10% the market of Windows, so we'll pay 1/10th for someone to develop a driver for this." But it turned out that USB worked completely differently on the Mac from how it did on Windows, so none of the Windows code was relevant at all for the Mac devs. They would either get what they paid for (which was terrible for users) or they would not get a Mac driver. This is around the time I stopped buying any device that required installing a driver. Many of these devices didn't really need one because they were regular USB-spec devices (keyboards, scanners, etc.) To this day, I will not install a driver for a fucking mouse. Why would that be required?

Jtsummers 4y ago

"Programming is just typing" was a typical management refrain when I was in the embedded field (more properly, now I'm adjacent to it). It was frustrating. Computer scientists and programmers aren't Real(tm) Engineers so they don't deserve as much money. They can't be in charge because you can't have engineers answering to non-engineers (ie, very few leads let alone managers coming from the software side). Which leads to a culture that's overly hardware centric with insufficient leadership/management understanding of what software actually entails. Also, "This doesn't meet the current requirements, fix it with software!" The best one was when the case wasn't waterproof... How the fuck is software supposed to fix that? They literally expected the software team to work magic. A lot of pushback got that requirement kicked back over to the mechanical engineering team to address, but it took months. Moronic.

AtlasBarfed 4y ago
It might also be that the management chains in embedded are likely former engineers or EE people. In Web you get a lot of management interfaces where the boss can't do their job: that's the perfect recipe for high pay. In embedded and other non-software engineering but engineering firms, the management is typically engineers that CAN do their subordinates jobs, they just don't want to.
- Jtsummers 4y ago
  That's what I was getting at with this: > They can't be in charge because you can't have engineers answering to non-engineers (ie, very few leads let alone managers coming from the software side). Which leads to a culture that's overly hardware centric with insufficient leadership/management understanding of what software actually entails. In hardware-centric orgs, software developers are a small step above technicians in their pecking order, sometimes below. The pecking order itself is annoying enough, but when you switch from designing your own ASICs to buying COTS dev boards and primarily only adding software to it, you're not really a hardware company anymore. But it'll take another generation for them to realize it, or a severe crunch if someone comes along and realizes that they can pay embedded devs $200k+ and eat the lunch of half these companies.

ryandrake 4y ago

Many hardware companies still see software as just another line item on the BOM: Like a screw or a gasket. It's something you build cheaply or buy from a supplier and sprinkle it on the product somewhere on the assembly line. These hardware companies have no concept of typical measures of software quality, building an ecosystem, release management, sometimes even no concept of source control. They tell an overworked embedded software engineer: "Go build something that barely meets these requirements, ship it, and then throw the scrap away." like metal shavings from their CNC machine.

commandlinefan 4y ago
> typical measures of software quality To play devil's advocate - are there any (useful) measures of software quality? Even this place is mostly programmers and we can't even agree whether we should be writing unit tests or not.
- marmetio 4y ago
  Sort of. There are accurate measures with verifiable predictive power. But useful depends on cost/benefit, which in turn depends on ability to implement and market forces. There's a company that looked at reducing critical defects from a sort of actuarial perspective. They have a few decades of cross-industry data. I've used their model, and it works. If you don't need a numerical result, you can just read the white paper about what's most important [1]. So to partially answer your question: unit testing reduces defects, but reducing defects might not be worth the costs to you. And defects might not be the only thing that matters. There are other measures of goodness, like maintainability, which complicates the answer. You'd have to collect your own data for that. [1] https://missionreadysoftware.com/articles/ https://missionreadysoftware.com/articles/
  - goalieca 4y ago
    I’d say for micro services and large distributed system, you do need a pyramid of testing with most covered at the unit level. The system is just too large and continuously changing as all the different versions of services release.

eschneider 4y ago

Yeah, I sorta specialize in that whole IoT firmware update/fleet monitoring/make sure everything at the edge runs smoothly end of things, and if you find a company that realizes that this is something that MUST work smoothly if they're going to scale, then it's a very sweet place to be. Even better, that sorta work combines low-level C++ with lots of back-end web service work, so you're never 'just' a C++ programmer.