6 ms·

for me, its that the hardware doesn't always work or at least not as you expect. if you write workstation or server code, 99.99999% of your problems will be sof

by dmh2000 10y ago

for me, its that the hardware doesn't always work or at least not as you expect. if you write workstation or server code, 99.99999% of your problems will be software bugs. But on an embedded system, especially something custom, you can have weird, intermittent hardware behavior that takes a lot of work to pin down. and sometimes you can't fix it so you workaround. It's both rewarding to get this stuff to work while at the same time extremely frustrating. I've worked embedded systems for years but every year I tell my colleagues that I'm switching to IT so if my hardware doesn't work I can just throw it away and buy a new workstation.

elevensies 10y ago
This has been my experience too on the limited number of embedded projects I've done. I once had an intermittent issue with the device locking up when going into sleep mode that took me 60 hours to get to the exact cause, which was in the CPLD, where isuues with a similar level of complexity I'd usually be able to pin down in a single day in my expertise area of server software. Also, recently having received an Arduino project to work on, it was a bit of a garden path to get to the point where I could get it running under GDB. Now I make sure I always have two working pieces of hardware, and I verify the hardware functionality to the extent possible before trying to implement software features on top of it.
oakwhiz 10y ago
Funny because I'm in IT and sometimes I feel like switching to embedded systems for almost exactly the same reason.
probablybanned 10y ago
"Oh, so this feature is just totally broken, then? Cool." Always read the errata, folks.
- ianhowson 10y ago
  "Also, we're not fixing the bugs, because customers have already built products that work around the bugs. Fixing the bugs would break their products." Case in point: Microchip ENC28J60 (http://www.microchip.com/wwwproducts/en/en022889 http://www.microchip.com/wwwproducts/en/en022889). Less than half of the advertised features actually work.
  - vvanders 10y ago
    Or the cost of the tape-out outweighs the number of customers using the feature.
    - ianhowson 10y ago
      Yup. The times when I've worked on small-quantity ASICs, tapeout cost and engineer availability have been the main reason that bugs weren't fixed. The ENC28J60 did at least four more silicon revs after the errata were released, so... yay Microchip?
  - bsder 10y ago
    Yeah, don't use their CAN interface IC's either. Exact same complaint. In general, I now avoid Microchip like the plague. Which means I now avoid Atmel, Micrel, SMSC, etc. <cries>
    - pryelluw 10y ago
      What chips do you use then?
      - bsder 10y ago
        Generally I will go for a built-in CAN controller on a microcontroller. Those generally work fine. Brands? A lot of TI and ST.
HeyLaughingBoy 10y ago
"Fun" problems from my embedded systems career in no particular order: 1)Writing device drivers for a chip that I can't get to work. Has to be my code, this is a basic function of the chip; why am I so stupid??!!? Finally call the vendor: oh, yeah we know about that bug, the next revision fixes it. Why isn't everyone else screaming about this problem? Well, you guys are only the second company to sample this chip. The first one found the bug! 2) System has to autoload a tray when operator closes the door. Works fine right up until it doesn't. Turns out some of the (visually opaque) trays are transparent at the infrared wavelength of the sensor that detects them. 3) (on a call with Field Service). Intermittent problem that seems to be software: a moving carriage is reporting errors, but only during certain moves. Field tech has replaced every relevant module and problem still happens. On a hunch I have him disable the safety interlocks and open all the doors to watch what happens. Problem is that there is a weak spot in an cable in an unrelated assembly that causes it to momentarily collide with the carriage, making the carriage fail to get to its destination. By the time they both stop moving, all hardware tests fine, so no one suspected the other assembly. 4) I love telling this one: hardware caught on fire because a jammed assembly caused a motor driver to overheat. Tester filed a bug against the software because it didn't report an error to the user. The CPU was fkn toasted!
- pryelluw 10y ago
  More stories, please.

by dmh2000 10y ago

elevensies 10y ago

This has been my experience too on the limited number of embedded projects I've done. I once had an intermittent issue with the device locking up when going into sleep mode that took me 60 hours to get to the exact cause, which was in the CPLD, where isuues with a similar level of complexity I'd usually be able to pin down in a single day in my expertise area of server software. Also, recently having received an Arduino project to work on, it was a bit of a garden path to get to the point where I could get it running under GDB. Now I make sure I always have two working pieces of hardware, and I verify the hardware functionality to the extent possible before trying to implement software features on top of it.

oakwhiz 10y ago

Funny because I'm in IT and sometimes I feel like switching to embedded systems for almost exactly the same reason.

probablybanned 10y ago

"Oh, so this feature is just totally broken, then? Cool." Always read the errata, folks.

ianhowson 10y ago
"Also, we're not fixing the bugs, because customers have already built products that work around the bugs. Fixing the bugs would break their products." Case in point: Microchip ENC28J60 (http://www.microchip.com/wwwproducts/en/en022889 http://www.microchip.com/wwwproducts/en/en022889). Less than half of the advertised features actually work.
- vvanders 10y ago
  Or the cost of the tape-out outweighs the number of customers using the feature.
  - ianhowson 10y ago
    Yup. The times when I've worked on small-quantity ASICs, tapeout cost and engineer availability have been the main reason that bugs weren't fixed. The ENC28J60 did at least four more silicon revs after the errata were released, so... yay Microchip?
- bsder 10y ago
  Yeah, don't use their CAN interface IC's either. Exact same complaint. In general, I now avoid Microchip like the plague. Which means I now avoid Atmel, Micrel, SMSC, etc. <cries>
  - pryelluw 10y ago
    What chips do you use then?
    - bsder 10y ago
      Generally I will go for a built-in CAN controller on a microcontroller. Those generally work fine. Brands? A lot of TI and ST.

HeyLaughingBoy 10y ago

"Fun" problems from my embedded systems career in no particular order: 1)Writing device drivers for a chip that I can't get to work. Has to be my code, this is a basic function of the chip; why am I so stupid??!!? Finally call the vendor: oh, yeah we know about that bug, the next revision fixes it. Why isn't everyone else screaming about this problem? Well, you guys are only the second company to sample this chip. The first one found the bug! 2) System has to autoload a tray when operator closes the door. Works fine right up until it doesn't. Turns out some of the (visually opaque) trays are transparent at the infrared wavelength of the sensor that detects them. 3) (on a call with Field Service). Intermittent problem that seems to be software: a moving carriage is reporting errors, but only during certain moves. Field tech has replaced every relevant module and problem still happens. On a hunch I have him disable the safety interlocks and open all the doors to watch what happens. Problem is that there is a weak spot in an cable in an unrelated assembly that causes it to momentarily collide with the carriage, making the carriage fail to get to its destination. By the time they both stop moving, all hardware tests fine, so no one suspected the other assembly. 4) I love telling this one: hardware caught on fire because a jammed assembly caused a motor driver to overheat. Tester filed a bug against the software because it didn't report an error to the user. The CPU was fkn toasted!

pryelluw 10y ago
More stories, please.