6 ms·

Devil's advocate. As someone who has developed a Linux based appliance with over 100k live units across the globe, it seems insane to NOT have access to the thi

Devil's advocate. As someone who has developed a Linux based appliance with over 100k live units across the globe, it seems insane to NOT have access to the thing you're selling and that you have to maintain. If your thing breaks or gets bricked by an update, you will call support and expect them to fix it. You don't want to send in your device or have a support technician come to your house to fix it. So yes, to the conspiracy theorists it may look like a secret backdoor -- it sorta is. But in many cases I bet it's just a safety net for developers and support to fix things. I speak for myself and my own experience working for $oldjob. Other companies or countries may of course use this differently. And of course companies get sold and such so you'll never know.

01HNNWZ0MV43FF 2y ago
The problem is it's gonna be a golden key system where everyone who's worked there for the last ten years has a copy
- binwiederhier 2y ago
  That is correct. But it is possible to design a system with short lived auth tokens/keys and frequent key rotation. I designed such a system at $oldjob for remote access (see [1]). Obviously there is always a risk, and there are always syseng/ops people with access. That is correct. [1] https://blog.heckel.io/2019/11/19/providing-remote-access-to-devices-via-ssh-tunnels/ https://blog.heckel.io/2019/11/19/providing-remote-access-to...
  - number6 2y ago
    Nice write up and a lot of gotchas you encountered
flaminHotSpeedo 2y ago
That's a fair argument, but it doesn't appear that that updates are high on sleep number's priority list: > The hub includes Python 2.7.18. While extremely old (keep in mind the Hub appears to have been last updated in 2018) If we give them the benefit of the doubt, perhaps they intended to to keep it up to date but ultimately compaines need to either be transparent about their remote access and manage it responsibly, which includes keeping the system patched, or give up access
- binwiederhier 2y ago
  > Python 2.7.18 I am not defending them for not keeping their stuff up-to-date, but it is very common practice for embedded systems to be hopelessly outdated. I've done what OP describes with IPMI/BMC systems for $mainboardmanufacturer1 and $mainboardmanufacturer2 (both really big name brands), and their BMC systems were equally outdated. It was almost comical, but really sad at the same time. Moral of the story is to firewall things off really well, I suppose. At $oldjob, I designed an upgrade mechanism to do A/B image updates so things were always up to date, or at 2-3 weeks out of date. See [1]. For small embedded systems that do not have enough space/bandwidth, this may not be feasible though. [1] https://blog.heckel.io/2019/09/18/image-based-upgrades-upgrading-software-and-os-of-80k-servers-every-two-weeks/ https://blog.heckel.io/2019/09/18/image-based-upgrades-upgra...
  - flaminHotSpeedo 2y ago
    I know I'm preaching to the choir, but I do prefer not having to treat my bed as a hostile device on my network
    - bitmasher9 2y ago
      Either treat your devices as potentially hostile or simply don’t connect your bed to a network.
    - hughesjj 2y ago
      I mean it's not like sleep number can tell when you're having sex or sleeping or anything.... Er, I mean okay, well at least they're budget friendly.... Well at least they don't have an ssh server constantly running in the background or something
    - nucleardog 2y ago
      Even if it didn’t have the intentional backdoor… you probably should be treating it as hostile anyway. Even where not intentionally hostile, not intentionally privacy invading, not trying to fetch updates so it can show you more ads, not… most of this stuff is so hopelessly out-of-date and full of security vulnerabilities it’s only not hostile out of luck. I don’t connect anything to WiFi unless absolutely necessary. And by that I don’t mean “the device demands it” (I just won’t buy the damn thing) but “it’s a core part of the functionality I’m asking of it”. I’ll prefer zwave/zigbee, Bluetooth, or something else wherever possible when communication is required. (If I were forced to use this bed and it had no manual controls I would definitely have used Bluetooth, avoiding this whole issue.) And even for the devices that do get a WiFi connection… they run entirely isolated, on a separate SSID and VLAN from my normal devices and traffic, and with a whitelist for what traffic is allowed. As far as I’m concerned the only difference between this bed and the other devices is that we know about the issues with this bed. We have no reason to believe that the other devices are any better, and in fact a pretty large body of evidence suggesting that they’re probably not.
Aurornis 2y ago
> As someone who has developed a Linux based appliance with over 100k live units across the globe, it seems insane to NOT have access to the thing you're selling and that you have to maintain. I’ve developed Linux devices selling that many units (and more) and I’m baffled that anyone would think this is a viable way to handle things at this scale. Units like this should have a firmly read-only Linux firmware that can only be changed by signed updates. The only data you would actually get or modify is the diagnostic data or the contents of the settings. Both of those can be sent through mechanisms that shouldn’t require SSH access. The correct way to handle this is with a debug info feature. Put something in the app that will zip up logs and configuration files and send them in for support, with the user’s explicit permission obviously. If you can’t figure it out from logs, you can use their config files to clone the situation on a device in the office. The bigger issue is: Who are you going to task with SSHing into customer devices? With 100K or more people filing support requests, it would be insane to have engineers handling those requests with anything having to do with SSH. It would be equally insane to hand off access to customer support people and give them the keys to SSH into customer devices.
- deleted 2y ago
  [deleted]
- binwiederhier 2y ago
  I agree that that is the gold standard. Having an immutable Linux that is well tested on your own hardware and upgraded like that. At the time I inherited a system that had 30-50k units deployed and was updated via Debian/APT. Older units were running Ubuntu 10.04 (it was 2016) and were hopelessly outdated. We managed to pull every single device to Ubuntu 16.04 and designed a fully automated image based update mechanism for them (I've linked it in other posts). We tried for read only base systems, but it was too tricky, so images stayed read-write, with migration of configs across upgrades. At the time, customers even had access via SSH (similar to NAS devices these days). I think what you are describing works for well defined hardware with a medium complexity software stack, or at least something that is limited in terms of epipheral device usage. The appliance I was managing was heavily using raided disk, ZFS, loops, dmsetup, and many other Linux tools that we have all seen fail in horrible ways. Not having SSH access, and not being able to diagnose lockups or hanging progress (D state issues) in a live system would have severely crippled us in being able to fix these issues. Many of them I'm sure we would not have been able to. We had failing disks, slow disks, failing RAM, hanging loop devices, corrupt loop devices, hanging ZFS, hanging ZFS, hanging ZFS, many of its bugs we fixed upstream, and and and... On top of that, we had a "bring your own device" product that literally allowed people to use whatever hardware they want. That makes the read only firmware thing ever trickier. As said in the beginning, I agree with you in principle, but there are many cases in which it's not as black and white. And I can fully understand the rationale of providing remote access. Side note: I would have never expected to be down voted on HN for expressing an opinion in a respectful manner about a subject that I have knowledge about, just because it is the "unpopular" opinion. On Reddit, I'd expect to be downvoted for something folks don't like, but on HN in thought the button is just for use against trolling and such.
toast0 2y ago
If you sold it, you should not have remote access to it. Auto-update is de facto isomorphic with remote access capability but that doesn't mean you should have a remote shell. At most, maaaaybe a way for the customer to enable a shell for developer support. Otherwise, a/b setup to avoid remote bricking, DFU or whatever current standard for customer driven unbricking in exceptional cases. But really, test all the forward and reverse update cases and keep a handful of samples of all shipped hardware so you can make sure everything actually works, and you can figure out how to fix it when you mess it up. Always test upgrades starting from factory fresh with all the versions you ever shipped from the factory. (I've run into products where several updates in, version X would work or not based on the original version from the factory forever ago because of original config or something that didn't get migrated properly but never caused problems until recently).
- fooqux 2y ago
  If you have the ability to update firmware, you have the ability to add remote access whenever you like. You're already trusting the vendor either way. That said, this current situation of an always-on SSH connection/backdoor is just begging to be exploited by an irate employee, curious intern, or worms. It's impossible to know what sort of safeguards the vendor has in place, if any. Putting a lock on a nuke is good, but not building the nuke at all is better.
jtriangle 2y ago
And I'm sure every one of those 100k devices has a unique ssh key right? Surely you can see the problem.
- binwiederhier 2y ago
  Not just that's they have - per session ssh keys that are valid for only 6 hours - all ssh sessions are audit logged and have to go through jump servers tied to tech roles - all sessions fully monitored via "script" and can be replayed You can also see a write-up here: https://news.ycombinator.com/item?id=40840040 https://news.ycombinator.com/item?id=40840040
protocolture 2y ago
It does seem insane. But the support engineer having local network access after remoting in without the customers willing consent also seems insane. Its obviously there so they can fix these devices, but shortcuts made for engineers are such a common security risk. Ideally you would have a backdoor on the device thats open only to the local network. User runs an app on their PC, provides willing consent for someone to complete a support task by providing an OTC to the engineer. App goes and discovers the device, and hosts the session for the engineer. If the user cant perform such a task they can probably buy a device with one button on it that will, or pay for a callout or return.

01HNNWZ0MV43FF 2y ago

The problem is it's gonna be a golden key system where everyone who's worked there for the last ten years has a copy

binwiederhier 2y ago
That is correct. But it is possible to design a system with short lived auth tokens/keys and frequent key rotation. I designed such a system at $oldjob for remote access (see [1]). Obviously there is always a risk, and there are always syseng/ops people with access. That is correct. [1] https://blog.heckel.io/2019/11/19/providing-remote-access-to-devices-via-ssh-tunnels/ https://blog.heckel.io/2019/11/19/providing-remote-access-to...
- number6 2y ago
  Nice write up and a lot of gotchas you encountered

flaminHotSpeedo 2y ago

That's a fair argument, but it doesn't appear that that updates are high on sleep number's priority list: > The hub includes Python 2.7.18. While extremely old (keep in mind the Hub appears to have been last updated in 2018) If we give them the benefit of the doubt, perhaps they intended to to keep it up to date but ultimately compaines need to either be transparent about their remote access and manage it responsibly, which includes keeping the system patched, or give up access

binwiederhier 2y ago
> Python 2.7.18 I am not defending them for not keeping their stuff up-to-date, but it is very common practice for embedded systems to be hopelessly outdated. I've done what OP describes with IPMI/BMC systems for $mainboardmanufacturer1 and $mainboardmanufacturer2 (both really big name brands), and their BMC systems were equally outdated. It was almost comical, but really sad at the same time. Moral of the story is to firewall things off really well, I suppose. At $oldjob, I designed an upgrade mechanism to do A/B image updates so things were always up to date, or at 2-3 weeks out of date. See [1]. For small embedded systems that do not have enough space/bandwidth, this may not be feasible though. [1] https://blog.heckel.io/2019/09/18/image-based-upgrades-upgrading-software-and-os-of-80k-servers-every-two-weeks/ https://blog.heckel.io/2019/09/18/image-based-upgrades-upgra...
- flaminHotSpeedo 2y ago
  I know I'm preaching to the choir, but I do prefer not having to treat my bed as a hostile device on my network
  - bitmasher9 2y ago
    Either treat your devices as potentially hostile or simply don’t connect your bed to a network.
  - hughesjj 2y ago
    I mean it's not like sleep number can tell when you're having sex or sleeping or anything.... Er, I mean okay, well at least they're budget friendly.... Well at least they don't have an ssh server constantly running in the background or something
  - nucleardog 2y ago
    Even if it didn’t have the intentional backdoor… you probably should be treating it as hostile anyway. Even where not intentionally hostile, not intentionally privacy invading, not trying to fetch updates so it can show you more ads, not… most of this stuff is so hopelessly out-of-date and full of security vulnerabilities it’s only not hostile out of luck. I don’t connect anything to WiFi unless absolutely necessary. And by that I don’t mean “the device demands it” (I just won’t buy the damn thing) but “it’s a core part of the functionality I’m asking of it”. I’ll prefer zwave/zigbee, Bluetooth, or something else wherever possible when communication is required. (If I were forced to use this bed and it had no manual controls I would definitely have used Bluetooth, avoiding this whole issue.) And even for the devices that do get a WiFi connection… they run entirely isolated, on a separate SSID and VLAN from my normal devices and traffic, and with a whitelist for what traffic is allowed. As far as I’m concerned the only difference between this bed and the other devices is that we know about the issues with this bed. We have no reason to believe that the other devices are any better, and in fact a pretty large body of evidence suggesting that they’re probably not.

Aurornis 2y ago

> As someone who has developed a Linux based appliance with over 100k live units across the globe, it seems insane to NOT have access to the thing you're selling and that you have to maintain. I’ve developed Linux devices selling that many units (and more) and I’m baffled that anyone would think this is a viable way to handle things at this scale. Units like this should have a firmly read-only Linux firmware that can only be changed by signed updates. The only data you would actually get or modify is the diagnostic data or the contents of the settings. Both of those can be sent through mechanisms that shouldn’t require SSH access. The correct way to handle this is with a debug info feature. Put something in the app that will zip up logs and configuration files and send them in for support, with the user’s explicit permission obviously. If you can’t figure it out from logs, you can use their config files to clone the situation on a device in the office. The bigger issue is: Who are you going to task with SSHing into customer devices? With 100K or more people filing support requests, it would be insane to have engineers handling those requests with anything having to do with SSH. It would be equally insane to hand off access to customer support people and give them the keys to SSH into customer devices.

deleted 2y ago
[deleted]
binwiederhier 2y ago
I agree that that is the gold standard. Having an immutable Linux that is well tested on your own hardware and upgraded like that. At the time I inherited a system that had 30-50k units deployed and was updated via Debian/APT. Older units were running Ubuntu 10.04 (it was 2016) and were hopelessly outdated. We managed to pull every single device to Ubuntu 16.04 and designed a fully automated image based update mechanism for them (I've linked it in other posts). We tried for read only base systems, but it was too tricky, so images stayed read-write, with migration of configs across upgrades. At the time, customers even had access via SSH (similar to NAS devices these days). I think what you are describing works for well defined hardware with a medium complexity software stack, or at least something that is limited in terms of epipheral device usage. The appliance I was managing was heavily using raided disk, ZFS, loops, dmsetup, and many other Linux tools that we have all seen fail in horrible ways. Not having SSH access, and not being able to diagnose lockups or hanging progress (D state issues) in a live system would have severely crippled us in being able to fix these issues. Many of them I'm sure we would not have been able to. We had failing disks, slow disks, failing RAM, hanging loop devices, corrupt loop devices, hanging ZFS, hanging ZFS, hanging ZFS, many of its bugs we fixed upstream, and and and... On top of that, we had a "bring your own device" product that literally allowed people to use whatever hardware they want. That makes the read only firmware thing ever trickier. As said in the beginning, I agree with you in principle, but there are many cases in which it's not as black and white. And I can fully understand the rationale of providing remote access. Side note: I would have never expected to be down voted on HN for expressing an opinion in a respectful manner about a subject that I have knowledge about, just because it is the "unpopular" opinion. On Reddit, I'd expect to be downvoted for something folks don't like, but on HN in thought the button is just for use against trolling and such.

toast0 2y ago

If you sold it, you should not have remote access to it. Auto-update is de facto isomorphic with remote access capability but that doesn't mean you should have a remote shell. At most, maaaaybe a way for the customer to enable a shell for developer support. Otherwise, a/b setup to avoid remote bricking, DFU or whatever current standard for customer driven unbricking in exceptional cases. But really, test all the forward and reverse update cases and keep a handful of samples of all shipped hardware so you can make sure everything actually works, and you can figure out how to fix it when you mess it up. Always test upgrades starting from factory fresh with all the versions you ever shipped from the factory. (I've run into products where several updates in, version X would work or not based on the original version from the factory forever ago because of original config or something that didn't get migrated properly but never caused problems until recently).

fooqux 2y ago
If you have the ability to update firmware, you have the ability to add remote access whenever you like. You're already trusting the vendor either way. That said, this current situation of an always-on SSH connection/backdoor is just begging to be exploited by an irate employee, curious intern, or worms. It's impossible to know what sort of safeguards the vendor has in place, if any. Putting a lock on a nuke is good, but not building the nuke at all is better.

jtriangle 2y ago

And I'm sure every one of those 100k devices has a unique ssh key right? Surely you can see the problem.

binwiederhier 2y ago
Not just that's they have - per session ssh keys that are valid for only 6 hours - all ssh sessions are audit logged and have to go through jump servers tied to tech roles - all sessions fully monitored via "script" and can be replayed You can also see a write-up here: https://news.ycombinator.com/item?id=40840040 https://news.ycombinator.com/item?id=40840040

protocolture 2y ago

It does seem insane. But the support engineer having local network access after remoting in without the customers willing consent also seems insane. Its obviously there so they can fix these devices, but shortcuts made for engineers are such a common security risk. Ideally you would have a backdoor on the device thats open only to the local network. User runs an app on their PC, provides willing consent for someone to complete a support task by providing an OTC to the engineer. App goes and discovers the device, and hosts the session for the engineer. If the user cant perform such a task they can probably buy a device with one button on it that will, or pay for a callout or return.