8 ms·

As a firmware engineer, I'm one of the people who actually writes the code that goes inside the IoT devices. I'm very interested in what the FCC might be able t

As a firmware engineer, I'm one of the people who actually writes the code that goes inside the IoT devices. I'm very interested in what the FCC might be able to do here. How does the FCC define a security flaw? Would updates only be distributed when there is a flaw that needs fixing? Remote update mechanisms can themselves present security problems in some domains. Thus, some devices should only be updatable if the owner has physical access to the device. Will the manufacturer be liable for damages caused by attacks on vulnerable devices that were not sufficiently updated by their owners? IoT is making its way into defense and enterprise environments where reliability is a matter of national security. An update nearly always results in some downtime for the device, even if it's just a couple seconds. Sometimes, it may be in the best interest of a device's owner to defer an update indefinitely, until that device's continuous operation is no longer mission-critical. Even if the owner can't control exactly what is in an update, they absolutely MUST be able to control when an update occurs.

baby_souffle 3y ago
> Even if the owner can't control exactly what is in an update, they absolutely MUST be able to control when an update occurs. +1 for this at the consumer level. My oven may have a critical update, but - for right now - *nothing* is more critical than finishing dinner. I'll let the update apply the day after thanksgiving when I'm doing the dishes. There are a few connected appliances brands that do this well: updates are broken down into two classes - "critical/security" updates and "all the other updates" - and I get to pick from "do nothing, notify, notify and download, notify, schedule/auto-apply" for both channels. Unfortunately this is not common because it takes planning and more than the bare minimum to pull off.
- klausa 3y ago
  I think this is oversimplifying things. Is finishing the dinner more important than applying a patch that fixes actively exploited bug that locks your oven into cleaning mode and burns everything inside into ash over the next three hours? Or something that disables the safety checks and lets the oven overheat and burn your house down? (Granted, the latter shouldn't physically be possible because it should have physical temperature killswitches etc.) People always (understandably so!) consider software updates to be annoyances; but especially when you give an example like _an oven_, the potential for _catastrophic_ failures is too great.
  - MarcoPerazaFCC 3y ago
    One my favorite IoT botnet scenarios is an attacker taking control of thousands of ovens/air conditions/other high-wattage devices and using them to cause power outages. https://www.usenix.org/system/files/conference/usenixsecurity18/sec18-soltan.pdf https://www.usenix.org/system/files/conference/usenixsecurit... I wonder how the impulse to connect everything to the internet will be remembered.
    - ezzaf 3y ago
      The flip side of that is you can use control of all those high wattage devices to prevent power outages by shifting load to times when more energy is available. Hopefully, that's how the impulse will be remembered.
      - necovek 3y ago
        Or by the rise of salmonela infections...
  - deleted 3y ago
    [deleted]
  - hot_gril 3y ago
    These seem like very rare scenarios. If we're concerned about dire threats like this, the manufacturer needs a way to remotely send devices into an internet-disconnected "safe mode" before anyone's even talking about updates.
    - klausa 3y ago
      How do you, as a user, determine whether your oven is in "safe mode" that the manufacturer has toggled, or in "safe mode" that makes the UI look the same as the real one, but is actually a malicious code that waits until 3AM to start the cleaning mode? I agree that these scenarios are (relatively) far fetched _now_; but if we expect the future to include connecting appliances like that to the internet, then solving problems like those is table-stakes stuff. I understand that enforced software updates are annoying; but for high-stakes environments like this they strike me as a lesser of the two evils.
- hot_gril 3y ago
  I'm wondering why you'd have a smart oven in the first place. Seems like all risk and no reward.
  - MrDrMcCoy 3y ago
    Probably for some feature or astetic that's unrelated to the smart features, but is only available on the smart oven.
    - galangalalgol 3y ago
      This os definitely most of it. Also, someone figured out they could do something with the parameters of the convection oven and get close to an air frier. Got the mode a year later. Lastly, when the meat thermometer hits temp, I get a notification. That is actually useful.
  - stevehawk 3y ago
    The ability to pre-heat my oven without standing in front of it. That's really the big win. But also to be able to tell if spouse or children left it on.
    - hot_gril 3y ago
      But I can walk to my oven and turn it on, which wasn't a real problem even when I lived in a huge house. What am I missing?
      - spiderice 3y ago
        We really can’t force you to see the obvious if you’re set on not understanding something.
        hot_gril 3y ago
        That's alright, I'm good. Smart ovens are pretty rare, so it seems like a lot of people don't get this.
- fragmede 3y ago
  The more critical thing than finishing Thanksgiving dinner for your oven is not burning your house down.
MichaelZuo 3y ago
This is a good point, some IoT devices really can't be designed to be physically serviceable, while still remaining reasonably compact, e.g. those that need very high levels of water resistance, especially saltwater resistance. And adding any remote update mechanism at all would more then likely decrease overall security. So there actually should be a counter mandate too, for devices that are impractical to design to be physically serviceable, while meeting certain size/weight/etc. requirements. To make sure remote update mechanisms, of any kind, are never implemented, unless the manufacturer can guarantee that the update mechanism itself doesn't introduce new flaws.
- fragmede 3y ago
  It sounds like you're looking for a carve out so you don't have to upgrade your devices to have a modern microcontroller that supports remote updates and are using saltwater as a scary thing so no one challenges you on it. You can conformal coat a ESP32 with a sensor and battery and a wireless charger, and get remote updating. If hobbyists are doing that without commercial backing, what industry experts like you have access to must be even better.
  - MichaelZuo 3y ago
    You appear to be under a lot of mistaken assumptions. And in any case, the default ideal is to have an option to update or reprogram devices in-person. It's never an ideal, that I've ever seen expressed on HN, to have a remote actor capable of doing so, unless in a totally air gapped environment.
    - fragmede 3y ago
      I'm interested in learning! what are those mistaken assumptions?
  - beambot 3y ago
    Wouldn't a Starlink satellite qualify as an IOT edge device? How do you propose a user-servicable physical switch on a device in LEO? Not all devices have easy or cost-effective physical access -- that's why IOT is particularly effective at bridging the digital-physical divide.
alerighi 3y ago
> Remote update mechanisms can themselves present security problems in some domains. Not really if done right to be fair. It's just a matter of implementing a signature verification of the firmware updates that are installed on the device. > IoT is making its way into defense and enterprise environments where reliability is a matter of national security. If it's a matter of national security surely you don't use IoT devices connected to the public internet. At least the devices are in some private network, where the traffic is under your control. So if you don't do security updates it may be acceptable under that circumstances.
- arp242 3y ago
  > It's just a matter of implementing a signature verification of the firmware updates that are installed on the device. In principle: yes. In practice: signing keys seem to get leaked all the time. It's not a mechanism I would blindly trust in security sensitive domains.
  - lnxg33k1 3y ago
    We could start by asking iot companies to be ISO 27002, stop saving passwords in excel files or in s3 bucket publicly accessible could help Security is not an update, security is a cultural trait of an entity or an individual
- TeMPOraL 3y ago
  > If it's a matter of national security surely you don't use IoT devices connected to the public internet. Of course they do. That's the flip side of PaaS and reverse-NIH syndrome, the "opex > capex" thinking: "Industry 4.0" is built on web tech, with all the practices and assumptions baked in. Your critical infrastructure is, or is about to, be running JavaScript on a docker-compose cluster, and expecting to be piecemal-updated daily. And then, there's also "shadow IT" - going behind the back of IT and using COTS SaaS to work around red tape is still... going around IT and giving untracked third-party vendors access to organizational information and operations. "Making its way" doesn't only mean "introduced by design" - those vulnerabilities just creep in.
MarcoPerazaFCC 3y ago
Thanks for this thoughtful feedback. I encourage you to file an official comment, especially regarding end-user control of update timing. Maybe my response here https://news.ycombinator.com/item?id=37394935 https://news.ycombinator.com/item?id=37394935 addresses some of your other concerns? We'd love to hear your thoughts.
- ryukoposting 3y ago
  Thank you for the reply! I have submitted my official comment.
MetaWhirledPeas 3y ago
Good comments. > The FCC recently issued a Notice of Proposed Rulemaking [2] for a cybersecurity labeling program for connected devices. This sounds like it is intended for consumer products, and it also sounds optional. I would hope that users with a legitimate reason to do so (defense, enterprise) would have the capacity to not participate and forego the label.
- ryukoposting 3y ago
  The line between "consumer product" and "enterprise/defense product" can be blurry. For example, event security teams may use their personal smartphones to communicate with medical staff. A lot of IoT companies (especially the startups) focus on the customers with the deepest pockets (enterprise and defense). If big-ticket customers demand this label, it generates a great deal of incentive for IoT companies to just say "to hell with it, we want that label on everything we make." In any case, the words "national security" are usually a good way to get the attention of a three letter ;)
WalterBright 3y ago
> Remote update mechanisms can themselves present security problems in some domains. I've proposed many times that the device have a physical write-enable switch on it, not a software switch. That way, a malware infestation won't survive a reboot, and your backup hard drives won't get compromised. I'm amazed that nobody does this. (Hard drives used to have a write-enable switch on them.)
- landemva 3y ago
  Why doesn't MS do similar on all of their OS code by placing it on a read-only filesystem? Only allow updates when reboot into an update mode.
  - WalterBright 3y ago
    I have zero faith in software read-only modes.
  - Too 3y ago
    Android does this. OS is read only. Updates happen on second partition and takes effect by swapping partition on reboot. Integrity is verified by secure boot.
    - landemva 3y ago
      Interesting. How is it verified? Maybe keep Merkle tree hash of eveything in TPM?
- structural 3y ago
  Many flash chips have write enable pins at the hardware level. So this support really does still exist in theory! However, these pins mostly aren't used for any kind of switch in basically every PCB design I've seen. Either it's "always enabled" so the storage can always be written, or it's a chip that's programmed from the factory and always disabled to prevent any kind of user update.
  - 3y ago
01100011 3y ago
It would help if there were reference designs and testsuites for common embedded tasks like remote updates, configuration storage, input validation, etc. I don't do much embedded work these days, but I remember a lot of shoddy, ersatz designs based on random OSS components. I remember a small mfg of hardware devices used widely in payment processing tell me they were secure because "they used SSL" despite not taking care of physical or system security in a coherent, effective way. Honestly, companies need to have some liability or other incentives to even care about security a lot of the time.
bboygravity 3y ago
I also write firmware for IoT. IMO: one of the most important incentives towards secure IoT is: don't make it IoT unless it benefits the user greatly (instead of benefitting the manufacturer). Don't let companies connect everything they want to the internet irresponsibly and if they must: force them to make it really really secure and long lasting (10+ years). Especially when there's high risk of causing physical damage to things and / or identity theft and / or privacy issues. Which en masse translate to national security issues. Force them to sell devices that keep functioning even if the manufacturer (servers) cease to exist. Example: introduce a mandatory manufacturer IoT insurance that pays out in case the device ceases to function (securely) within x years (even in case of manufacturer bankruptcy). Random chaotic brainfarts.

baby_souffle 3y ago

> Even if the owner can't control exactly what is in an update, they absolutely MUST be able to control when an update occurs. +1 for this at the consumer level. My oven may have a critical update, but - for right now - *nothing* is more critical than finishing dinner. I'll let the update apply the day after thanksgiving when I'm doing the dishes. There are a few connected appliances brands that do this well: updates are broken down into two classes - "critical/security" updates and "all the other updates" - and I get to pick from "do nothing, notify, notify and download, notify, schedule/auto-apply" for both channels. Unfortunately this is not common because it takes planning and more than the bare minimum to pull off.

klausa 3y ago
I think this is oversimplifying things. Is finishing the dinner more important than applying a patch that fixes actively exploited bug that locks your oven into cleaning mode and burns everything inside into ash over the next three hours? Or something that disables the safety checks and lets the oven overheat and burn your house down? (Granted, the latter shouldn't physically be possible because it should have physical temperature killswitches etc.) People always (understandably so!) consider software updates to be annoyances; but especially when you give an example like _an oven_, the potential for _catastrophic_ failures is too great.
- MarcoPerazaFCC 3y ago
  One my favorite IoT botnet scenarios is an attacker taking control of thousands of ovens/air conditions/other high-wattage devices and using them to cause power outages. https://www.usenix.org/system/files/conference/usenixsecurity18/sec18-soltan.pdf https://www.usenix.org/system/files/conference/usenixsecurit... I wonder how the impulse to connect everything to the internet will be remembered.
  - ezzaf 3y ago
    The flip side of that is you can use control of all those high wattage devices to prevent power outages by shifting load to times when more energy is available. Hopefully, that's how the impulse will be remembered.
    - necovek 3y ago
      Or by the rise of salmonela infections...
- deleted 3y ago
  [deleted]
- hot_gril 3y ago
  These seem like very rare scenarios. If we're concerned about dire threats like this, the manufacturer needs a way to remotely send devices into an internet-disconnected "safe mode" before anyone's even talking about updates.
  - klausa 3y ago
    How do you, as a user, determine whether your oven is in "safe mode" that the manufacturer has toggled, or in "safe mode" that makes the UI look the same as the real one, but is actually a malicious code that waits until 3AM to start the cleaning mode? I agree that these scenarios are (relatively) far fetched _now_; but if we expect the future to include connecting appliances like that to the internet, then solving problems like those is table-stakes stuff. I understand that enforced software updates are annoying; but for high-stakes environments like this they strike me as a lesser of the two evils.
hot_gril 3y ago
I'm wondering why you'd have a smart oven in the first place. Seems like all risk and no reward.
- MrDrMcCoy 3y ago
  Probably for some feature or astetic that's unrelated to the smart features, but is only available on the smart oven.
  - galangalalgol 3y ago
    This os definitely most of it. Also, someone figured out they could do something with the parameters of the convection oven and get close to an air frier. Got the mode a year later. Lastly, when the meat thermometer hits temp, I get a notification. That is actually useful.
- stevehawk 3y ago
  The ability to pre-heat my oven without standing in front of it. That's really the big win. But also to be able to tell if spouse or children left it on.
  - hot_gril 3y ago
    But I can walk to my oven and turn it on, which wasn't a real problem even when I lived in a huge house. What am I missing?
    - spiderice 3y ago
      We really can’t force you to see the obvious if you’re set on not understanding something.
      - hot_gril 3y ago
        That's alright, I'm good. Smart ovens are pretty rare, so it seems like a lot of people don't get this.
fragmede 3y ago
The more critical thing than finishing Thanksgiving dinner for your oven is not burning your house down.

MichaelZuo 3y ago

This is a good point, some IoT devices really can't be designed to be physically serviceable, while still remaining reasonably compact, e.g. those that need very high levels of water resistance, especially saltwater resistance. And adding any remote update mechanism at all would more then likely decrease overall security. So there actually should be a counter mandate too, for devices that are impractical to design to be physically serviceable, while meeting certain size/weight/etc. requirements. To make sure remote update mechanisms, of any kind, are never implemented, unless the manufacturer can guarantee that the update mechanism itself doesn't introduce new flaws.

fragmede 3y ago
It sounds like you're looking for a carve out so you don't have to upgrade your devices to have a modern microcontroller that supports remote updates and are using saltwater as a scary thing so no one challenges you on it. You can conformal coat a ESP32 with a sensor and battery and a wireless charger, and get remote updating. If hobbyists are doing that without commercial backing, what industry experts like you have access to must be even better.
- MichaelZuo 3y ago
  You appear to be under a lot of mistaken assumptions. And in any case, the default ideal is to have an option to update or reprogram devices in-person. It's never an ideal, that I've ever seen expressed on HN, to have a remote actor capable of doing so, unless in a totally air gapped environment.
  - fragmede 3y ago
    I'm interested in learning! what are those mistaken assumptions?
- beambot 3y ago
  Wouldn't a Starlink satellite qualify as an IOT edge device? How do you propose a user-servicable physical switch on a device in LEO? Not all devices have easy or cost-effective physical access -- that's why IOT is particularly effective at bridging the digital-physical divide.

alerighi 3y ago

> Remote update mechanisms can themselves present security problems in some domains. Not really if done right to be fair. It's just a matter of implementing a signature verification of the firmware updates that are installed on the device. > IoT is making its way into defense and enterprise environments where reliability is a matter of national security. If it's a matter of national security surely you don't use IoT devices connected to the public internet. At least the devices are in some private network, where the traffic is under your control. So if you don't do security updates it may be acceptable under that circumstances.

arp242 3y ago
> It's just a matter of implementing a signature verification of the firmware updates that are installed on the device. In principle: yes. In practice: signing keys seem to get leaked all the time. It's not a mechanism I would blindly trust in security sensitive domains.
- lnxg33k1 3y ago
  We could start by asking iot companies to be ISO 27002, stop saving passwords in excel files or in s3 bucket publicly accessible could help Security is not an update, security is a cultural trait of an entity or an individual
TeMPOraL 3y ago
> If it's a matter of national security surely you don't use IoT devices connected to the public internet. Of course they do. That's the flip side of PaaS and reverse-NIH syndrome, the "opex > capex" thinking: "Industry 4.0" is built on web tech, with all the practices and assumptions baked in. Your critical infrastructure is, or is about to, be running JavaScript on a docker-compose cluster, and expecting to be piecemal-updated daily. And then, there's also "shadow IT" - going behind the back of IT and using COTS SaaS to work around red tape is still... going around IT and giving untracked third-party vendors access to organizational information and operations. "Making its way" doesn't only mean "introduced by design" - those vulnerabilities just creep in.

MarcoPerazaFCC 3y ago

Thanks for this thoughtful feedback. I encourage you to file an official comment, especially regarding end-user control of update timing. Maybe my response here https://news.ycombinator.com/item?id=37394935 https://news.ycombinator.com/item?id=37394935 addresses some of your other concerns? We'd love to hear your thoughts.

ryukoposting 3y ago
Thank you for the reply! I have submitted my official comment.

MetaWhirledPeas 3y ago

Good comments. > The FCC recently issued a Notice of Proposed Rulemaking [2] for a cybersecurity labeling program for connected devices. This sounds like it is intended for consumer products, and it also sounds optional. I would hope that users with a legitimate reason to do so (defense, enterprise) would have the capacity to not participate and forego the label.

ryukoposting 3y ago
The line between "consumer product" and "enterprise/defense product" can be blurry. For example, event security teams may use their personal smartphones to communicate with medical staff. A lot of IoT companies (especially the startups) focus on the customers with the deepest pockets (enterprise and defense). If big-ticket customers demand this label, it generates a great deal of incentive for IoT companies to just say "to hell with it, we want that label on everything we make." In any case, the words "national security" are usually a good way to get the attention of a three letter ;)

WalterBright 3y ago

> Remote update mechanisms can themselves present security problems in some domains. I've proposed many times that the device have a physical write-enable switch on it, not a software switch. That way, a malware infestation won't survive a reboot, and your backup hard drives won't get compromised. I'm amazed that nobody does this. (Hard drives used to have a write-enable switch on them.)

landemva 3y ago
Why doesn't MS do similar on all of their OS code by placing it on a read-only filesystem? Only allow updates when reboot into an update mode.
- WalterBright 3y ago
  I have zero faith in software read-only modes.
- Too 3y ago
  Android does this. OS is read only. Updates happen on second partition and takes effect by swapping partition on reboot. Integrity is verified by secure boot.
  - landemva 3y ago
    Interesting. How is it verified? Maybe keep Merkle tree hash of eveything in TPM?
structural 3y ago
Many flash chips have write enable pins at the hardware level. So this support really does still exist in theory! However, these pins mostly aren't used for any kind of switch in basically every PCB design I've seen. Either it's "always enabled" so the storage can always be written, or it's a chip that's programmed from the factory and always disabled to prevent any kind of user update.
- 3y ago

01100011 3y ago

It would help if there were reference designs and testsuites for common embedded tasks like remote updates, configuration storage, input validation, etc. I don't do much embedded work these days, but I remember a lot of shoddy, ersatz designs based on random OSS components. I remember a small mfg of hardware devices used widely in payment processing tell me they were secure because "they used SSL" despite not taking care of physical or system security in a coherent, effective way. Honestly, companies need to have some liability or other incentives to even care about security a lot of the time.

bboygravity 3y ago

I also write firmware for IoT. IMO: one of the most important incentives towards secure IoT is: don't make it IoT unless it benefits the user greatly (instead of benefitting the manufacturer). Don't let companies connect everything they want to the internet irresponsibly and if they must: force them to make it really really secure and long lasting (10+ years). Especially when there's high risk of causing physical damage to things and / or identity theft and / or privacy issues. Which en masse translate to national security issues. Force them to sell devices that keep functioning even if the manufacturer (servers) cease to exist. Example: introduce a mandatory manufacturer IoT insurance that pays out in case the device ceases to function (securely) within x years (even in case of manufacturer bankruptcy). Random chaotic brainfarts.