6 ms·
Required reading. Have you worked as an SRE? > Item: Rollbacks need to be possible This is the dirty secret to keeping life as an SRE unexciting. If you can't
by B-Con 7y ago
Required reading. Have you worked as an SRE?
> Item: Rollbacks need to be possible
This is the dirty secret to keeping life as an SRE unexciting. If you can't roll it back, re-engineer it with the dev team until you can. When there's no alternative, you find one anyway.
(When you really and truly cross-my-heart-and-hope-to-die can't re-engineer around it fully, isolate the the non-rollbackable pieces, break them into small pieces, and deploy them in isolation. That way if you're going to break something, you break as little as possible and you know exactly where the problem is.)
Try having a postmortem, even informal, for every rollback. If you were confident enough to push to prod, but didn't work, figure out why that happened and what you can do to avoid it next time.
> Item: New states (enums) need to be forward compatible
Our internal Protobuf style guides strongly encourage this. In face, some of the most backward-compatible-breaking features of protobuf v2 were changed for v3.
> Item: more than one person should be able to ship a given binary.
Easy to take this one for granted when it's true, but it 100% needs to be true. Also includes:
* ACLs need to be wide enough that multiple people can perform every canonical step.
* Release logic/scripts needs to be accessible. That includes "that one" script "that guy" runs during the push that "is kind of a hack, I'll fix it later". Check. It. In. Anyway.
* Release process needs to be understood by multiple people. Doesn't matter if they can perform the release if they don't know how to do it.
> Item: if one of our systems emits something, and another one of our systems can't consume it, that's an error, even if the consumer is running on someone's cell phone far far away.
Easy first step is to monitor 4xx response codes (or some RPC equivalent). I've rolled back releases because of an uptick in 4xxs. Even better is to get feedback from the clients. Having a client->server logging endpoint is one option.
And if a release broke a client, rollback and see the first point. Postmortem should include why it wasn't caught in smoke/integration testing.
- packetslave 7y agoRequired reading. Have you worked as an SRE? She was both a long-time SRE at Google and a long-time PE at Facebook.
- B-Con 7y agoI suspected. Couldn't find a resume on her site to confirm it.
- ndarwincorn 7y agoHer RSS is very much worth a subscription.
- tty7 7y agoCheck. It. In. Anyway. Amen
- silvestrov 7y ago> Release logic/scripts needs to be accessible More than accessible: all scripts must be in git just like source code because that is also source code. Building releases should be done using a fresh VM. Creating and configuring that VM should only use a script which is also in git. Everything needs to be automated using scripts. If you type "apt-get" on the command line you have lost reliability. When multiple people are involved, such manual setups will be a problem at some point: people make mistakes. Manual steps also means you have lost good testing of the build process.
- deleted 7y ago[deleted]
- pmlnr 7y ago> Rollbacks need to be possible I always feel like people who write these never faced SQL schema changes or dataset updates. I wonder what rollback plans are in place for complete MySQL replication chains, for example.
- Etheryte 7y agoBuilding datasets up via migration sets, not manual updates, and storing source data will solve both of those issues. If you also need fresh data, store snapshots which you should be doing anyway. Shortly put, with a thought out approach I don't see this as a bigger issue than anything else on the list.
- grey-area 7y agoThis is addressed in the article (in passing). You split the changes up - First make sure your code handles both old and new schemas Then introduce the schema change Then introduce the code which depends on the schema Each of these steps is performed separately and monitored for problems, with rollback possible at each stage. The most painful thing to rollback is the database, but it is possible, though rarely necessary if done in isolation and tested against the old code first. There may be a few more steps after if you want to tidy up the schema by removing old data etc. It's trading off complexity in development/deploy for reliability.
- B-Con 7y agoPretty much this. Writing code that can handle both old and new schema is admittedly annoying. But it's safe and forces a thoughtful rollout. Even data mutations can be rolled back. Dual storage writes, snapshots, etc. The goal isn't to eliminate risk, it's to reduce the risk and make it calculated and bounded. Heck, having a rollback plan that says "we'll restore a snapshot and lose X minutes worth of data mutations" is fine in some cases. It's tradeoffs. (I've seen a case where literally adding a column caused an outage, because the presence of the column triggered a new code path. Rollback was to delete the column.)
- tonyarkles 7y ago
- exlurker 7y agoSRE: Site Reliability Engineer
- cwilkes 7y agoSecond step would be to have a way for the client to report an error. I’ve seen a number of systems where this isn’t the case — just looking at the server logs everything looks okay until a week later that client calls to say they haven’t gotten a file in a week and wondering if something is wrong.
- matthewowen 7y ago"Check. It. In. Anyway." It's consistently a source of sorrow to me how many bugs exist and how much inefficiency there is because people don't want to be embarrassed by the code they quickly wrote.