4 ms·
I never said they should test for everything. OP's description suggests it was an error that's common to all the deployed instances that received the update, r
by stephenr 3y ago
I never said they should test for everything.
OP's description suggests it was an error that's common to all the deployed instances that received the update, rather than some specific combination of environment and that deployment.
It would have allowed them to run the same deployment locally and use physical access (serial, a display, sd card, whatever) to capture the error log.
What they came up with is clever but it's very surprising that they needed it, especially given that they have very limited remote access to the units that are in the wild.
- bongodongobob 3y agoNo, they said some.
- stephenr 3y agoEmphasis is mine: > We caught the issue almost instantly since we were watching the deploy, and were able to stop updates before any other devices picked it up. But those that already got it were down.
- MBCook 3y agoYou’re correct. I think every box that got it started having problems (or at least most), the only reason any were still up is that updates were scattered in case of think kind of incident (and to avoid hammering out poor little server).
- MBCook 3y agoAfter all this time I don’t remember what the bug was. We did have boxes locally that we tested on of course. But somehow this got out. It might’ve been something that only showed up under certain configurations. It might’ve been something that just should have been caught under normal circumstances and wasn’t for some reason. It may have been something that worked fine tested off-hardware until some bug in packaging things slightly changed it and broke a script. Or it could’ve been a case of “this is trivial, I know what I’m doing, it will be fine“. We were a very small operation so I’m not going to say that we had an amazing QC process. It may have been a very human mistake.