5 ms·
I built a whole remote software update mechanism for a control binary that ran on 25k+ servers across multiple data centers. Rest assured that after the first
by latchkey 3y ago
I built a whole remote software update mechanism for a control binary that ran on 25k+ servers across multiple data centers.
Rest assured that after the first time I messed it up (which required ssh into each box individually), I wrote a lot of unit and integration tests to make sure that it never failed to deploy again. One of the integration tests ensured that the app started up and could always go through the internal auto update process. This ran in CI and would fail the build if it didn't pass.
While I fully understand that this is hard to get right 100% of the time, a mess up of this level by a car manufacturer is pretty amazing to me.
- aaronbeekay 3y agoAs somebody currently working at an automaker on software systems, the amazing thing to me is that a mess up of this level doesn’t happen weekly. It’s rough out here.
- bozhark 3y agoWhat's the priority then, telemetry data? Why is it rough out there?
- reactordev 3y agoProbably due to fires, failures, and fatigue.
- firtoz 3y agoGames have AAA, autos have FFF
- jacquesm 3y agoRelatively crappy pay, complex toolchains, long build times, layer upon layer of (really bad) legacy code, badly specified (if they're specified) protocols between subsystems, subsystems that are completely opaque (no source code provided), homegrown OS's or older RTOS's, subset-of-C to keep it safe(r), tricky debugging environments and if you're really unlucky anemic hardware. I hope I didn't miss anything but I wouldn't be surprised if I did.
- ahartmetz 3y agoYeah, I think you missed something. The "software architects, heavy enterprisey tooling, and minions" approach to development where some of the architects could be good developers, but they don't develop, and the minions are often not that good and also not given any autonomy, so they are in a state of learned helplessness and just do what they're told without much thinking or initiative. It results in over-abstracted, over-complicated, slow, unreliable, and sometimes just stupid code.
- jacquesm 3y agoFair enough, yes. That's hopefully not all of them though but I don't doubt that many of the older companies work like that.
- ahartmetz 3y agoMost car companies are, in fact, quite old. Their big suppliers (who are often even worse at software, if you can believe that) are also quite old.
- foobiekr 3y agodo you guys not have confirmed boot and swizzling to fallback images?
- AlotOfReading 3y agoAutomotive varies widely between "basically modern Linux systems with proper updates" and the most janky, home-grown update systems imaginable, sometimes even within the same components and teams.
- foobiekr 3y agoYah, I know from friends at ford and vw that there's still vxworks and qnx, but even there, good grief, a-b with confirmed boot is about as basic as you can get. I confess I've seen incredible sloppiness about when a confirmation is done (too early, including in the initial init stages which is way too soon) and watchdogs (spawn off a process that has a while loop stroking the wd - just absolutely pointless).
- jacquesm 3y agoThank you. At least you're honest about it, the other day someone was trying real hard to convince me that software developers at automakers are made of magic fairy dust.
- donmcronald 3y ago> While I fully understand that this is hard to get right 100% of the time, a mess up of this level by a car manufacturer is pretty amazing to me. I feel like it's going to happen to someone that makes network devices eventually. I'm always scared to update my (several hundred) UniFi devices. Their update process isn't foolproof and they push auto-updates via the UI pretty hard. Several years ago they caused some people's devices to disconnect from the management controller when they enabled 'https' communication. Prior to that, if you were pointing devices at 'https://example.com:8080 https://example.com:8080...' they would ignore the 'https' part and do an 'http' request to port '8080'. Then they pushed their 'https' update which expected an 'https' connection and didn't fall back to the old behavior for anyone that was mistakenly using 'https' in their URL initially. Some people on their forums complained about having to manually SSH to every device to fix the issue. It was caused by an end-user mistake, but they knew it was a potential issue. AFAIK, their attitude on it hasn't changed and a lot and at the time their response was that they knew it would break some people, but that it wouldn't be that many (lol). IMO, the issue with those systems is that basic communication back to the update / config server is part of the total package which is too complex (ie: a full Debian install). I'd rather see something like Mender (mender.io) where the core communications / updates come from a hardened system with watchdog, recovery, rollback logic. Think of how crazy it is to have something like pfSense doing package based updates rather than slice based updates. At least with boot environments they could add some watchdog and rollback type logic, but it'll still be part of the total system instead of something like a hardened slice based setup where the most critical logic is isolated from everything else and treated like a princess. Do you have any insight on package vs slice based systems for updates? Did you isolate update logic from the rest of the system or am I out of touch with that opinion?
- vGPU 3y agoReminds me of my (far less critical) update process for home assistant. Every time something breaks. Currently my hvac automations are going haywire.
- code_runner 3y agoout of morbid curiosity.... how long did it take to ssh into and fix all of those servers? I imagine even automating a fix (if possible) would still take a good amount of time.
- latchkey 3y agognu parallel and sshpass is your friend. The way I built my app was that I could install it cleanly via a curl | bash. So, I just had a simple shell script that iterated through the list of IP addresses (from the DHCP leases), ran curl | bash and that cleaned up the mess pretty quickly.
- psychlops 3y agoHaving worked on 25K machines, I can assure you that it never deployed to every single machine and failed to do so in interesting ways all the time.
- postalrat 3y agoAs a frontend web developer I'm constantly deploying software to many thousands of machines. And you know what? It's pretty damn simple.
- rokkitmensch 3y ago[flagged]
- drdaeman 3y agoI used to wear your shoes in IE6/7 ages (no longer, I gave up during the "framework of the week" race and went all-backend), and it wasn't simple at all. Browser compatibility with all their rendering nuances, individual system oddities and all sort of fragile stuff. And fortunately, no one bats an eye at a slightly broken site, but everyone hates even a slightly broken vehicle.
- jrumbut 3y agoIt's simple because we tolerate certain limitations in the web platform. If you had a hard requirement that a page load could never take more than 100ms, regardless of network conditions, you'd have quite a challenge on your hands.
- foobiekr 3y agoRivian is an embedded use case, though, which is not at all like a fleet of servers. Having worked for companies that produce network devices - including devices that are unreachable for example for 6 months of the year - and on software installation and upgrade, I am baffled how this bricking is possible. For one thing, you generally use some kind of confirmed boot mechanism - you upgrade a standby partition, set an ephemeral boot value that causes device to boot the alternate image, and reboot - only when the image is declared "up" does that get persisted (and then the alternate is upgraded, in order to prevent rollback in the event of a media error). You use watchdogs that are tied to actual forward progress (and not just some demon that the kernel schedules and bangs on the watchdog even if the rest of the system is hung) and if they fail, the WD reboots you. (This is one of the reasons that event driven programming is somewhat preferred - actually processing events from a single dispatch thread makes it easier to reason about the system.) On top of that, you make sure that the core system is an immutable filesystem so that you can validate the _offline_ alternate image before rebooting (write-and-read-back-uncached) and periodically scrub the alternate image (same). Like.. this is all embedded 101, stuff people have been widely doing since the mid 1990s and I think I can find examples going back to the 70s. Sometimes you get a little more sophisticated (allow sub-packages or overlays and use a manifest to check the ensemble instead of just a single image), but it's very standard.
- ikiris 3y agoThat sounds out of scope for the MVP. We can worry about redundancies later after we ship.
- roland35 3y agoHey now, preventing SEVs doesn't lead to impact. If we all collectively let this become a raging dumpster fire we can all heroically fix it and greatly exceed expectations for the half.
- bluGill 3y agoYou too have noticed most great employee rewards go to someone who if they had done their job well wouldn't have been notived
- jdechko 3y agoAs a non-developer, the whole situation with a bad software update to the Voyager spacecraft really puts things into perspective as far as how bad remote updates can be. It’s also a testament to the way that the system was designed that they were able to get it back online.
- akira2501 3y agoWhen possible, I used a fail back mechanism. If the update failed to fully come up, then the watchdog timer would catch it, the bootloader would notice the incomplete boot, and attempt to boot from the previous known working image in that case.
- cjbprime 3y ago> This ran in CI and would fail the build if it didn't pass. I don't mean to be pedantic, but since we're talking about what should happen instead, this is insufficient. It works until the day you realize you made some kind of manual change to your CI infra, or that CI has some non-standard configuration that makes it work for you but not some significant fraction of the fleet. People should do what you described in CI, but as well as that, you need phased rollout, where e.g. the build can only be rolled out to the next percentage point of randomly selected users in a specific segment (e.g. each hardware revision and country as independent segments) after meeting a ratio of successful check-ins, in the field, from the new build by production customers in that segment. That's the actual metric for proceeding with the rollout: actual customers are successfully checking in from the new version of the software. Except, that's actually not sufficient either. What if the new build is good, but it contains an update to the updater which bricks the updater? Now you're getting successful check-ins from the new version in the field, but none of those customers will ever successfully auto-update again. So, test the new updater's ability to go forwards successfully, too.
- latchkey 3y ago> It works until the day you realize you made some kind of manual change to your CI infra, or that CI has some non-standard configuration that makes it work for you but not some significant fraction of the fleet. Nah, my CI process was solid. This was proven in the field over the course of years. > I don't mean to be pedantic... you need phased rollout You don't need to be pedantic, but better to ask the question rather than assume that was all that I did. =) You have to realize that what I built, worked flawlessly. It wasn't easy either, took a lot of trial and error. I did have a CIDR based rollout. I could specify down to the individual box that it would run a specific version. Or I could write "latest" to always keep certain boxes running on the latest build. This was another part of my testing, but ended up not being fully necessary because I had enough automated testing in CI that "latest" always worked. > but it contains an update to the updater which bricks the updater? This happened, so I wrote a lot of test code to make sure that would never happen again. My CI would catch that since I was E2E testing that it could actually run the upgrade process. Once I implemented all of this, I never had a single failure and would routinely, several times a day, deploy to the entire cluster, over the course of a couple years. It was all eventually consistent as I could also control the "check for update" frequency as well.
- ugh123 3y agoPlease tell me you scripted that ssh into across your 25k servers!
- latchkey 3y agohttps://news.ycombinator.com/item?id=38270986 https://news.ycombinator.com/item?id=38270986 One thing my little control process did on the box was to always set the password to be the same... user/1. None of these boxes needed inbound connections, so it wasn't a big deal to do that.
- sixtram 3y agoyou ssh-d into 25K servers one by one? I mean, manually?
- latchkey 3y agohttps://news.ycombinator.com/item?id=38270986 https://news.ycombinator.com/item?id=38270986