11 ms·

I used to work at a small company. We had a few remote embedded devices that did work and sent data back to the mothership over the internet. Their firmware cou

by MBCook 3y ago

I used to work at a small company. We had a few remote embedded devices that did work and sent data back to the mothership over the internet. Their firmware could be remotely updated, but we were always very careful. Well one day a mistake was finally made. Some of the devices went into a sort of loop. They’d start running the important process, something would go wrong, and they’d just retry every few minutes. We caught the issue almost instantly since we were watching the deploy, and were able to stop updates before any other devices picked it up. But those that already got it were down. We could ask the devices to send us the output of a command remotely, but it was too limited to be able to send back an error log. We didn’t have time to send back like 255 characters at a time or whatever, we needed to get it fixed ASAP. And that’s when the genius stupid hack was suggested. While we couldn’t send up a full log file, we could certainly send up something the length of a URL. So what if we sent down a little command to stick the log on PasteBin and send up the URL to us? Worked like a charm. We could identify what was going wrong and fix it in just a few minutes. After some quick (but VERY thorough) testing everything was fixed.

kr0bat 3y ago
How you could have enough control over the machine to reroute the error log to (what I assume was) a Pastebin api, while also lacking access to any of the files on the machine? In my mind you'd be required to ssh into the machine to upload, and if you're ssh'd in, why not just cat the log?
- MBCook 3y ago
  Good question! We couldn’t SSH in, which is too bad this would all be trivial. We had no direct access to the boxes, they were often behind firewalls. In fact that was the suggested placement for security reasons. They weren’t full servers, just little embedded things. We had a little HTTP API that it was always talking to. It would call the API to send data back to us or just check in regularly, and we would return to it a little bit of status stuff like the current time to keep clocks in sync, and a list of which “commands” they need to run. Mostly the commands were things like “your calibration data is out of data, pull an update“ or “a firmware update is available“. But one of them let us run arbitrary shell commands. The system was very limited. I wasn’t a developer directly on the project but I think it was just our custom software plus busy box and a handful of other things our normal shell scripts used. I assume it had been added after some previous incident. I believe the basic idea was that during troubleshooting you could tell a box to return the output of “cat /etc/resolv.conf” or something else that we hadn’t preplanned for without having to send someone into the field. But since it was only for small things like that it couldn’t return a full file. Luckily one of the commands was either curl or wget. So we could send down “curl -whatever /log/path https://pastebin/upload” https://pastebin/upload” or whatever it was. I don’t remember if we signed up for a pastebin account so we knew where it would show up or if we had it return URL to us in the output of the curl command. This suggestion was literally a joke. We were all beating our heads against the wall trying to help and someone just said “why don’t we just stick it on pastebin“ out of frustration, and the developer on the project realized we had what we needed to do that and it would work.
- lelanthran 3y ago
  > In my mind you'd be required to ssh into the machine to upload, and if you're ssh'd in, why not just cat the log? Ssh on remote IoT class devices is works. The problem is rarely ssh, the problem is always some form of key management plus NATs in-between. If you've got a few thousand devices in the field, public key management can become a a real pain, especially when you want to revoke keys.
- iimblack 3y ago
  I’ve worked at a company where our remote access was over a super slow modem line but the machine did have access to the internet.
- zimpenfish 3y ago
  I was doing some proxy soak testing for a company once where we had to run the tests from the server room but there was no non-proxy connectivity from that room to where we were monitoring the tests. Simple solution: output the progress to Dropbox, watch the same file upstairs. Bit of delay, sure, but better than having no idea how things are going until the 30-60min test is done (and no, we weren't allowed to sit in the server room watching it.)
stephenr 3y ago
Your company had remote embedded devices but didn't keep one "locally" for debugging issues?
- bongodongobob 3y ago
  There's a zillion things that you can't necessarily test for locally. When you have a fleet of IoT devices deployed in other people's environments, there's literally no way to test everything. Your question comes down to "Why didn't you just deploy bug free code using perfect processes? That's what I always do." I mean, cmon.
  - stephenr 3y ago
    I never said they should test for everything. OP's description suggests it was an error that's common to all the deployed instances that received the update, rather than some specific combination of environment and that deployment. It would have allowed them to run the same deployment locally and use physical access (serial, a display, sd card, whatever) to capture the error log. What they came up with is clever but it's very surprising that they needed it, especially given that they have very limited remote access to the units that are in the wild.
    - bongodongobob 3y ago
      No, they said some.
      - stephenr 3y ago
        Emphasis is mine: > We caught the issue almost instantly since we were watching the deploy, and were able to stop updates before any other devices picked it up. But those that already got it were down.
        MBCook 3y ago
        You’re correct. I think every box that got it started having problems (or at least most), the only reason any were still up is that updates were scattered in case of think kind of incident (and to avoid hammering out poor little server).

by MBCook 3y ago

kr0bat 3y ago

How you could have enough control over the machine to reroute the error log to (what I assume was) a Pastebin api, while also lacking access to any of the files on the machine? In my mind you'd be required to ssh into the machine to upload, and if you're ssh'd in, why not just cat the log?

MBCook 3y ago
Good question! We couldn’t SSH in, which is too bad this would all be trivial. We had no direct access to the boxes, they were often behind firewalls. In fact that was the suggested placement for security reasons. They weren’t full servers, just little embedded things. We had a little HTTP API that it was always talking to. It would call the API to send data back to us or just check in regularly, and we would return to it a little bit of status stuff like the current time to keep clocks in sync, and a list of which “commands” they need to run. Mostly the commands were things like “your calibration data is out of data, pull an update“ or “a firmware update is available“. But one of them let us run arbitrary shell commands. The system was very limited. I wasn’t a developer directly on the project but I think it was just our custom software plus busy box and a handful of other things our normal shell scripts used. I assume it had been added after some previous incident. I believe the basic idea was that during troubleshooting you could tell a box to return the output of “cat /etc/resolv.conf” or something else that we hadn’t preplanned for without having to send someone into the field. But since it was only for small things like that it couldn’t return a full file. Luckily one of the commands was either curl or wget. So we could send down “curl -whatever /log/path https://pastebin/upload” https://pastebin/upload” or whatever it was. I don’t remember if we signed up for a pastebin account so we knew where it would show up or if we had it return URL to us in the output of the curl command. This suggestion was literally a joke. We were all beating our heads against the wall trying to help and someone just said “why don’t we just stick it on pastebin“ out of frustration, and the developer on the project realized we had what we needed to do that and it would work.
lelanthran 3y ago
> In my mind you'd be required to ssh into the machine to upload, and if you're ssh'd in, why not just cat the log? Ssh on remote IoT class devices is works. The problem is rarely ssh, the problem is always some form of key management plus NATs in-between. If you've got a few thousand devices in the field, public key management can become a a real pain, especially when you want to revoke keys.
iimblack 3y ago
I’ve worked at a company where our remote access was over a super slow modem line but the machine did have access to the internet.
zimpenfish 3y ago
I was doing some proxy soak testing for a company once where we had to run the tests from the server room but there was no non-proxy connectivity from that room to where we were monitoring the tests. Simple solution: output the progress to Dropbox, watch the same file upstairs. Bit of delay, sure, but better than having no idea how things are going until the 30-60min test is done (and no, we weren't allowed to sit in the server room watching it.)

stephenr 3y ago

Your company had remote embedded devices but didn't keep one "locally" for debugging issues?

bongodongobob 3y ago
There's a zillion things that you can't necessarily test for locally. When you have a fleet of IoT devices deployed in other people's environments, there's literally no way to test everything. Your question comes down to "Why didn't you just deploy bug free code using perfect processes? That's what I always do." I mean, cmon.
- stephenr 3y ago
  I never said they should test for everything. OP's description suggests it was an error that's common to all the deployed instances that received the update, rather than some specific combination of environment and that deployment. It would have allowed them to run the same deployment locally and use physical access (serial, a display, sd card, whatever) to capture the error log. What they came up with is clever but it's very surprising that they needed it, especially given that they have very limited remote access to the units that are in the wild.
  - bongodongobob 3y ago
    No, they said some.
    - stephenr 3y ago
      Emphasis is mine: > We caught the issue almost instantly since we were watching the deploy, and were able to stop updates before any other devices picked it up. But those that already got it were down.
      - MBCook 3y ago
        You’re correct. I think every box that got it started having problems (or at least most), the only reason any were still up is that updates were scattered in case of think kind of incident (and to avoid hammering out poor little server).