3 ms·

>I would gladly use 10k lines of code to save one moving part or one switch This creates a whole new reliability problem. Software reliability is notorious for

by bumby 4y ago

>I would gladly use 10k lines of code to save one moving part or one switch This creates a whole new reliability problem. Software reliability is notorious for its interface failures and there can be many more of those modes (where hardware typically has a few wear-out style failures). I think about my relatively high-end TV that seems to always struggle with software updates without any glitches. It seems like we've just been lulled into accepting software glitches as the price of having software control everything. Luckily, most of the software is not safety critical. I think this is one of the dangers of relying on software to close reliability gaps. Yes, you can change it quickly and cheaply and it's versatile, but it's often very hard to identify all the failure modes.

eternityforest 4y ago
True embedded software is a lot different. Smart TVs are desktop or phone class computers, and even then, the glitches are usually "A power cycle fixes it" or "One feature that wouldn't exist at all without software stops working" type of thing. For non safety critical stuff it's much better to crash every week than to wear out in 5 years.
- bumby 4y ago
  >glitches are usually "A power cycle fixes it" Maybe I'm becoming a curmudgeon or maybe it's my bias as a former reliability engineer, but the fact that we've normalized this drives me crazy. Imagine your frustration if your car needed to be "power cycled" on a regular basis just to keep it moving forward, yet we've become accustomed to that in our day-to-day interactions with electronics. It's the bane of my existence that I have to continuously power cycle a device to do something seemingly simply like change a tv streaming service or scan a document. Interface failures are rampant.
  - lloeki 4y ago
    > your car needed to be "power cycled" on a regular basis just to keep it moving forward you'd be surprised how often watchdogs get their GIGO conditions triggered and things perform a cleanup reboot while you drive
    - bumby 4y ago
      I have no issue if it automatically recovers to a safe or known state. My issue is when this doesn't happen and the user has to perform the recovery via a power cycle. From my perspective, this means the system has failed and is, by definition, not reliable. We just seem to accept low reliability in consumer software.
  - eternityforest 4y ago
    It's definitely not ideal, and I obviously try to do better than that in any product I work on. But I've seen so many things fail due to bad switches, connectors, and other simple parts, to me it's a great tradeoff to occasionally reboot something as opposed to having it totally fail at some random time within a few years. I wouldn't trust my life to it, but I'll trust my watch party to it. Plus, a consumer now only needs to power cycle or factory reboot, so untrained people can keep stuff going. I'd much rather they need reboots than need skilled work like they used to, that just wouldn't be practical in a digital first world. Oddly enough having failures be so normalized makes them less problematic, because the victim doesn't get many complaints. People who have to wait just assume that's how it is, nobody will tell you you shouldn't have used a computer for that Before all these complex systems we used pen and paper and things subject to human error, and we had to deal with that quite a bit. Computers suck compared to what they could be but they are amazing compared to a lot of older stuff.
    - bumby 4y ago
      We can design hardware to last, but I think we often don’t for economic reasons. My larger point is that using software just trades one well-understood failure mode for a group of unknown, hard to test failure modes, usually because it’s cheaper. I think my biggest complaint comes as a result of the data economy. You can still find “dumb” tvs but they are usually a lot more expensive than their smart variants, I assume because they cannot monetize your viewing habits. My SO convinced me to get rid of my old laser, USB interface printer. That one always worked but was old. My new WIFI enabled one requires logging into an app to use, and the USB interface maybe works 10% of the time. The app and hardware constantly require power cycling before it works. Many times it never does. One thing becomes apparent in quality engineering: often, adding more components/interfaces undermines reliability because there are often exponentially more failure modes. It feels like we’re being trained to no longer expect reliability, and the beneficiaries are not the consumers but the companies that create unnecessary interfaces to monetize data.
      - eternityforest 4y ago
        I think that definitely applies when you have updatable software or software that interacts with something updatable, but less so for the 8 bit embedded MCU stuff typically used to replace hardware. If you, say, use a microcontroller to soft start a motor instead of making it bigger, or use a capsense pad instead of a switch(Yes they can be obnoxious, but at least they last!), you are unlikely to get into a permanent bricked state because you're not writing to flash at all. Your failure modes might increase in number as software complexity increases, but for the most part they stay ephemeral. I'm not sure exactly why experiences are so varied on things like smart TVs and WiFi printers. I have the "Smart" version of almost everything, and it usually works very well, but some people seem to have generally bad experiences. I think some of it might be because they only test on fairly generic consumer setups with the latest stock Android, no firewall, no VPN, Bluetooth never turned off, no custom DNS, gmail for the email account, etc, and more tech savvy people tend to have a few things that modern tech doesn't like. It always seems like people who tinker have problems other people don't. I also notice iOS users seem to have trouble with random gadgets and peripherals not being too happy, but a my iPhone using friends aren't a big sample size. If I was designing a product, I'd definitely be more careful, but as a user at home, I tend to avoid anything heavily customized, so I don't usually see as many of the bugs.
  - mgkimsal 4y ago
    "Let me reboot my tv". "Just reboot your phone". These are relatively common phrases now that would have baffled anyone 30 years ago.
Damogran6 4y ago
Got a nespresso Machine last year...worked great....hooked up to Wifi and the phone app and had it firmware update and it turned it into a brick. They sent me a replacement that works great and is entirely ignorant of the outside world (and will stay that way.)
- eternityforest 4y ago
  Are you on a network with some kind of firewalling or tracker blocking? I wonder if they are programmed to stop working if they can't phone home after exiting dumb mode? Or if there's an activation step they never planned to need recovery from failure of?
  - Damogran6 4y ago
    I don’t think it was intentional, talking to their tech support, there was no way to restore it to function. I suspect a bricked firmware update.
    - bombela 4y ago
      What's so sad is that usually bricking is a pure software bug. That overwrites whatever code boots the device far enough so thatnl it can update itself. If you have the hardware programmer you can most often update the flash directly. I don't expect everybody to have all the hardware programmers on hand. But I would gladly go to a repair shop to get a quick firmware flash to unbrik some device. Of course this is probably not worth it for manufacturers. It's so infuriating because most device even have a relatively standard programming port already (though you need to open open the device to access it)
      - eternityforest 4y ago
        This is a case where Arduino is so far beyond any professional solution it's just insane. USB update, no problem. If products would all just use a Wemos D1 or something on their board, none of this would be an issue. But sadly software defined products need(or think they need) proprietary firmware and Arduino isn't set up for strong DRM.
mschuster91 4y ago
> It seems like we've just been lulled into accepting software glitches as the price of having software control everything. Luckily, most of the software is not safety critical. And the thing is, somehow every vendor seems to skip out on ensuring a working out-of-band upgrade process - despite U-Boot bringing everything you need with it to provide recovery via whatever means you want. Ethernet, USB, a serial debug port... hell, I did A/B partition support and bootloader checks with a watchdog years ago, hacked together in an afternoon but able to first attempt booting from the fresh firmware, if that failed three times it would attempt the B partition containing the old image, and if that failed it would attempt booting from a recovery partition that was intelligent enough to bring up the base board with enough networking to be able to bring up a web server and accept a signed firmware image. Or, if that's too complex, a fucking button that the user has to hold and it will boot into the recovery image directly. It's not rocket science, it's not even much effort, all you need is a fucking week at least and a couple of cents for a switch. Pure penny pinching, and I hate companies that produce hardware that can turn to e-waste because of it.
- bombela 4y ago
  It even makes the development of the product easier for the programmers. Maybe when they don't need to fight the flashing process they can focus on writing better code. One can still hope.

by bumby 4y ago

eternityforest 4y ago

True embedded software is a lot different. Smart TVs are desktop or phone class computers, and even then, the glitches are usually "A power cycle fixes it" or "One feature that wouldn't exist at all without software stops working" type of thing. For non safety critical stuff it's much better to crash every week than to wear out in 5 years.

bumby 4y ago
>glitches are usually "A power cycle fixes it" Maybe I'm becoming a curmudgeon or maybe it's my bias as a former reliability engineer, but the fact that we've normalized this drives me crazy. Imagine your frustration if your car needed to be "power cycled" on a regular basis just to keep it moving forward, yet we've become accustomed to that in our day-to-day interactions with electronics. It's the bane of my existence that I have to continuously power cycle a device to do something seemingly simply like change a tv streaming service or scan a document. Interface failures are rampant.
- lloeki 4y ago
  > your car needed to be "power cycled" on a regular basis just to keep it moving forward you'd be surprised how often watchdogs get their GIGO conditions triggered and things perform a cleanup reboot while you drive
  - bumby 4y ago
    I have no issue if it automatically recovers to a safe or known state. My issue is when this doesn't happen and the user has to perform the recovery via a power cycle. From my perspective, this means the system has failed and is, by definition, not reliable. We just seem to accept low reliability in consumer software.
- eternityforest 4y ago
  It's definitely not ideal, and I obviously try to do better than that in any product I work on. But I've seen so many things fail due to bad switches, connectors, and other simple parts, to me it's a great tradeoff to occasionally reboot something as opposed to having it totally fail at some random time within a few years. I wouldn't trust my life to it, but I'll trust my watch party to it. Plus, a consumer now only needs to power cycle or factory reboot, so untrained people can keep stuff going. I'd much rather they need reboots than need skilled work like they used to, that just wouldn't be practical in a digital first world. Oddly enough having failures be so normalized makes them less problematic, because the victim doesn't get many complaints. People who have to wait just assume that's how it is, nobody will tell you you shouldn't have used a computer for that Before all these complex systems we used pen and paper and things subject to human error, and we had to deal with that quite a bit. Computers suck compared to what they could be but they are amazing compared to a lot of older stuff.
  - bumby 4y ago
    We can design hardware to last, but I think we often don’t for economic reasons. My larger point is that using software just trades one well-understood failure mode for a group of unknown, hard to test failure modes, usually because it’s cheaper. I think my biggest complaint comes as a result of the data economy. You can still find “dumb” tvs but they are usually a lot more expensive than their smart variants, I assume because they cannot monetize your viewing habits. My SO convinced me to get rid of my old laser, USB interface printer. That one always worked but was old. My new WIFI enabled one requires logging into an app to use, and the USB interface maybe works 10% of the time. The app and hardware constantly require power cycling before it works. Many times it never does. One thing becomes apparent in quality engineering: often, adding more components/interfaces undermines reliability because there are often exponentially more failure modes. It feels like we’re being trained to no longer expect reliability, and the beneficiaries are not the consumers but the companies that create unnecessary interfaces to monetize data.
    - eternityforest 4y ago
      I think that definitely applies when you have updatable software or software that interacts with something updatable, but less so for the 8 bit embedded MCU stuff typically used to replace hardware. If you, say, use a microcontroller to soft start a motor instead of making it bigger, or use a capsense pad instead of a switch(Yes they can be obnoxious, but at least they last!), you are unlikely to get into a permanent bricked state because you're not writing to flash at all. Your failure modes might increase in number as software complexity increases, but for the most part they stay ephemeral. I'm not sure exactly why experiences are so varied on things like smart TVs and WiFi printers. I have the "Smart" version of almost everything, and it usually works very well, but some people seem to have generally bad experiences. I think some of it might be because they only test on fairly generic consumer setups with the latest stock Android, no firewall, no VPN, Bluetooth never turned off, no custom DNS, gmail for the email account, etc, and more tech savvy people tend to have a few things that modern tech doesn't like. It always seems like people who tinker have problems other people don't. I also notice iOS users seem to have trouble with random gadgets and peripherals not being too happy, but a my iPhone using friends aren't a big sample size. If I was designing a product, I'd definitely be more careful, but as a user at home, I tend to avoid anything heavily customized, so I don't usually see as many of the bugs.
- mgkimsal 4y ago
  "Let me reboot my tv". "Just reboot your phone". These are relatively common phrases now that would have baffled anyone 30 years ago.

Damogran6 4y ago

Got a nespresso Machine last year...worked great....hooked up to Wifi and the phone app and had it firmware update and it turned it into a brick. They sent me a replacement that works great and is entirely ignorant of the outside world (and will stay that way.)

eternityforest 4y ago
Are you on a network with some kind of firewalling or tracker blocking? I wonder if they are programmed to stop working if they can't phone home after exiting dumb mode? Or if there's an activation step they never planned to need recovery from failure of?
- Damogran6 4y ago
  I don’t think it was intentional, talking to their tech support, there was no way to restore it to function. I suspect a bricked firmware update.
  - bombela 4y ago
    What's so sad is that usually bricking is a pure software bug. That overwrites whatever code boots the device far enough so thatnl it can update itself. If you have the hardware programmer you can most often update the flash directly. I don't expect everybody to have all the hardware programmers on hand. But I would gladly go to a repair shop to get a quick firmware flash to unbrik some device. Of course this is probably not worth it for manufacturers. It's so infuriating because most device even have a relatively standard programming port already (though you need to open open the device to access it)
    - eternityforest 4y ago
      This is a case where Arduino is so far beyond any professional solution it's just insane. USB update, no problem. If products would all just use a Wemos D1 or something on their board, none of this would be an issue. But sadly software defined products need(or think they need) proprietary firmware and Arduino isn't set up for strong DRM.

mschuster91 4y ago

> It seems like we've just been lulled into accepting software glitches as the price of having software control everything. Luckily, most of the software is not safety critical. And the thing is, somehow every vendor seems to skip out on ensuring a working out-of-band upgrade process - despite U-Boot bringing everything you need with it to provide recovery via whatever means you want. Ethernet, USB, a serial debug port... hell, I did A/B partition support and bootloader checks with a watchdog years ago, hacked together in an afternoon but able to first attempt booting from the fresh firmware, if that failed three times it would attempt the B partition containing the old image, and if that failed it would attempt booting from a recovery partition that was intelligent enough to bring up the base board with enough networking to be able to bring up a web server and accept a signed firmware image. Or, if that's too complex, a fucking button that the user has to hold and it will boot into the recovery image directly. It's not rocket science, it's not even much effort, all you need is a fucking week at least and a couple of cents for a switch. Pure penny pinching, and I hate companies that produce hardware that can turn to e-waste because of it.

bombela 4y ago
It even makes the development of the product easier for the programmers. Maybe when they don't need to fight the flashing process they can focus on writing better code. One can still hope.