17 ms·
Some items from my “reliability list”
- mahkoh 7y agoOn JSON: "if you only need 53 bits of your 64 bit numbers" JSON numbers are arbitrary precision. "blowing CPU on ridiculously inefficient marshaling and unmarshaling steps" On the other hand I am not blowing dev and qa time on learning/developing tools to replace curl/jq/browser/text editor.
- mgraczyk 7y agoThat is not true of JSON numbers either practically or technically. This specification allows implementations to set limits on the range and precision of numbers accepted. https://tools.ietf.org/html/rfc7159#section-6 https://tools.ietf.org/html/rfc7159#section-6
- SahAssar 7y agoYes it is. All that part of the spec says is that it allows implementations to set/have limits on ranges and precision, which is obvious because for example javascript only has 53 bit numbers. Other implementations can have other limits, but that does not mean that JSON itself or all implementations have the specific limits of javascript.
- megous 7y agoNot anymore. Try writing 10000000000000000000000000000000000000000000000000000000000000000000000000000000n + 1n into your browser console.
- majewsky 7y agoWell yeah, those are bigints, but JSON does not support bigints, so the point still stands.
- nemetroid 7y agoIf by "JSON" you mean "JSON parsers in web browsers", then yes.
- howdydoo 7y agoFirefox: >JSON.stringify(1n) TypeError: BigInt value can't be serialized in JSON Chrome: >JSON.stringify(1n) Uncaught TypeError: Do not know how to serialize a BigInt at JSON.stringify (<anonymous>) at <anonymous>:1:6 I don't think we're quite there yet.
- megous 7y agoIt was counterpoint to "that does not mean that JSON itself or all implementations have the specific limits of javascript." Javascript can do arbitrary precision integers now. So the last part is not true. You can generate your JSON strings with string concatenation in JS if you like. Just because JSON.stringify has some limit doesn't mean JS has.
- apaprocki 7y agoSee my comment elsewhere in the thread that shows how to properly use BigInts with JSON.stringify, if you choose to support them in your environment.
- raverbashing 7y agoI'm assuming the percentage of problems where the difference between 64 bits and 53 bits precision matter are really, really small (and you should be paying attention then, because you're bound to lose precision in other parts and not only in JSON) For everything else, it is really a non-issue.
- mgraczyk 7y agoIt is not the case that JSON numbers are "arbitrary precision". JSON numbers are in fact "implementation defined" precision. The technical and practical implication is exactly what was described in the article: Since precision is implementation defined (not uniform and with no lower-bound), JSON numbers make a poor vessel for inter-service communication.
- jcoffland 7y agoJavascript cannot handle 64-bit unsigned integers and the largest exact integral value is 2^53-1, but yeah JSON itself has no such restrictions.
- kristiandupont 7y agoI don't understand why you are being downvoted. This struck me not as something that starts to "emerge after you've dealt with enough failures in a given domain" as the author claims, but more like a pet peeve. The fact that JS has 53 bit precision will be a JS problem whether you use protobuf or anything else. On the other hand, if you are not using JS, it will almost certainly parse numbers of the precision that your language offers.
- TeMPOraL 7y agoThe original article's complaint wasn't (just) about 53 bits, but about using inefficient formats requiring constant serialization and deserialization for backend-to-backend communication.
- kristiandupont 7y agoProtobuf or Thrift would require serialization as well? I still wouldn't do that (or, in fact, I am currently not doing that) before I have an incentive, i.e. performance problems. Of course, my services are all Nodejs at the moment, so JSON feels more natural there than it does in other environments.
- mr_crankypants 7y agoOne potentially nice thing about formats like Cap'n Proto is that they don't eagerly parse the message. You can just lazily grab the values you care about, which can be a big performance win if you typically only care about a few fields out of a larger message. That's impossible with text-based formats like XML and JSON, because everything is fractally variable-length.
- bibabaloo 7y agoIt seems like its more "problems you deal with when you work at scale" like the author has at Google & FB. At smaller scales, I think the (machine) benefits of using something like protobuf don't nearly outweigh the human benefits of just using JSON.
- TeMPOraL 7y ago> On the other hand I am not blowing dev and qa time on learning/developing tools to replace curl/jq/browser/text editor. Because learning how to use correct tool for the job is so difficult and so outside of dev/qa job descriptions, that it's much better to waste compute on your servers and deal with performance and scalability problems.
- rimliu 7y agoSo the old "computer time is cheap, human time is not" is not true anymore?
- joshuamorton 7y agoStrongly typed, schema'd data interchange leads to less human time in the long run.
- EdwardDiego 7y agoYep, you just have to put the time into ensuring that convenient <Serialisation X> utilities exist that offer the (near) same functionality of curl/jq etc. It's a pain, but it's a short term pain, and necessary.
- Piskvorrr 7y agoIt is. OTOH, "these tools have been sufficient ten years ago, so I'm sticking to them forever" leads to wasting human time (by ignoring newer, potentially more efficient, tools).
- nvarsj 7y agoI think that line is mostly propaganda from defenders of poor performing language runtimes. It’s not a dichotomy - both are important. Eg AWS bills can easily eclipse engineer salaries if you use it inefficiently.
- taffer 7y ago> propaganda from defenders of poor performing language runtimes Your comment is unnecessarily inflammatory. > AWS bills can easily eclipse engineer salaries if you use it inefficiently. If this happens, it is more due to architectural and algorithms problems than to the runtime of the language. There are many large applications written in Python, Ruby and PHP that run just fine.
- patrec 7y ago> JSON numbers are arbitrary precision. > On the other hand I am not blowing dev and qa time on learning/developing tools to replace curl/jq/browser/text editor. What a pleasant surprise it will be for you when you find out that jq silently corrupts integers with more than 53 bits.
- mr_crankypants 7y agoIt's perhaps less a limitation of JSON and more a limitation of JavaScript itself[1]. But still, given that easy consumption from JavaScript is the ultimate primordial reason for choosing JSON over other formats, it seems like trying to transmit integers with more than 53 bits of precision over JSON is asking for trouble. Because it's only a matter of time until someone will want to do something like write a new service in Node, and the JavaScript parsers for other formats are at least somewhat more likely to guide people toward using BigInt for large integers. [1]: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Number/MAX_SAFE_INTEGER https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...
- apaprocki 7y agoIn TC39, we specified BigInt to not participate in JSON by default, precisely because emitting it as a native JSON number would not be able to be read back by JSON.parse() and many other environments would also not be able to parse it without extra logic if they simply use IEEE754 double. I explicitly asked for and achieved step 2. in the modified SerializeJSONProperty algorithm[1] so that users could decide and opt-in to serializing BigInts as strings if they so choose, with or without some sigil that could be interpreted by a reviver function. e.g.: > JSON.stringify(BigInt(1)) TypeError: Do not know how to serialize a BigInt ... > BigInt.prototype.toJSON = function() { return this.toString(); } > JSON.stringify(BigInt(1)) '"1"' [1]: https://tc39.es/proposal-bigint/#sec-serializejsonproperty https://tc39.es/proposal-bigint/#sec-serializejsonproperty
- awinder 7y agoOr they'd just wrap as a string and move on with life, much as people do when they send uuids which should consume 128-bit as wide strings. The cases I've encountered that actually fundamentally required 64-bit ints are completely overshadowed by the cases where some well-intentioned developer decides to pack 64-bit ints with information to turn it into a uuid as well, so I really hope the consternation over this is from people who work a lot with mathematical apis/systems and not devs doing more heinous things with 64bit ints.
- LaGrange 7y ago> On the other hand I am not blowing dev and qa time on learning/developing tools to replace curl/jq/browser/text editor. Is your turnover so high that learning a single tool is a significant portion of employee costs?
- raxxorrax 7y agoWhy not just use SOAP? I think recent experience should tell us that the easiest thing plainly wins out. JSON can be terrible, Javascript can be terrible, but limitations on large integers isn't the worst hurdle to take.
- B-Con 7y agoRequired reading. Have you worked as an SRE? > Item: Rollbacks need to be possible This is the dirty secret to keeping life as an SRE unexciting. If you can't roll it back, re-engineer it with the dev team until you can. When there's no alternative, you find one anyway. (When you really and truly cross-my-heart-and-hope-to-die can't re-engineer around it fully, isolate the the non-rollbackable pieces, break them into small pieces, and deploy them in isolation. That way if you're going to break something, you break as little as possible and you know exactly where the problem is.) Try having a postmortem, even informal, for every rollback. If you were confident enough to push to prod, but didn't work, figure out why that happened and what you can do to avoid it next time. > Item: New states (enums) need to be forward compatible Our internal Protobuf style guides strongly encourage this. In face, some of the most backward-compatible-breaking features of protobuf v2 were changed for v3. > Item: more than one person should be able to ship a given binary. Easy to take this one for granted when it's true, but it 100% needs to be true. Also includes: * ACLs need to be wide enough that multiple people can perform every canonical step. * Release logic/scripts needs to be accessible. That includes "that one" script "that guy" runs during the push that "is kind of a hack, I'll fix it later". Check. It. In. Anyway. * Release process needs to be understood by multiple people. Doesn't matter if they can perform the release if they don't know how to do it. > Item: if one of our systems emits something, and another one of our systems can't consume it, that's an error, even if the consumer is running on someone's cell phone far far away. Easy first step is to monitor 4xx response codes (or some RPC equivalent). I've rolled back releases because of an uptick in 4xxs. Even better is to get feedback from the clients. Having a client->server logging endpoint is one option. And if a release broke a client, rollback and see the first point. Postmortem should include why it wasn't caught in smoke/integration testing.
- packetslave 7y agoRequired reading. Have you worked as an SRE? She was both a long-time SRE at Google and a long-time PE at Facebook.
- B-Con 7y agoI suspected. Couldn't find a resume on her site to confirm it.
- Thorrez 7y ago>In this case, you need to make sure you can recognize the new value and not explode just from seeing it, then get that shipped everywhere. Then you have to be able to do something reasonable when it occurs, and get that shipped everywhere. Finally, you can flip whatever flag lets you start actually emitting that new value and see what happens. Can't those first 2 steps be combined together? Why do they need to be shipped separately?
- akuji1993 7y agoDepends on your updating architecture.. If you have a central backend service, but scheduled patches or the user has to manually update, that might become a problem. There could also be the case that client and backend can be updated independently and one might not be done before the other. The issue might only exist for a few minutes, or an hour, but would still be a huge impact on the software.
- Thorrez 7y agoSorry, I still don't understand. The post say you need to do 1. Have the code recognize the new value. Get that shipped everywhere. 2. Have the code do something reasonable when the new value appears. Get that shipped everywhere. 3. Start emitting the new value. My proposal is 1. Have the code recognize the new value. Have the code do something reasonable when the new value appears. Get that shipped everywhere. 2. Start emitting the new value. If the user has to manually update, and might refuse to do so, the problem exists under both the 3 step and the 2 step process. So nothing is gained by using the 3 step process. If the client and backend are updated independently, that should still be fine with the 2 step process. The 2 step process says "Get that shipped everywhere", meaning shipped to both the client and the backend.
- joshuamorton 7y agoI would rephrase this as 1. Support the new value in your schema 2. Support the new value in the client 3. Emit the new value from the server Combining 1 and 2 is probably possible, but not a great idea. Imagine if you end up rolling back 2 and 3, but there are still potentially new values in flight. If 1 is still there, you're good. If 1 and 2 are combined, you rollback all 3, but the new value is still in flight and your client crashes.
- akuji1993 7y agoI think something that's missing in this list is to have a really good and consistent QA cycle. You need to have rollbacks, I agree, but even better is, when bugs can't even make it into your production build. Having automated testing set up, (actually correctly done) code reviews and quality gateways in place can save you a lot of time rolling back your code. Catch the bug before it goes into live.
- VBprogrammer 7y agoOne of the fun challenges I have in my current job is that we provide releases to customers according to the customers schedule (which is related to needing hours of downtime because it's a creaky old system). Some customers will skip releases altogether making strategies like add a new column, back populate it online, then the next release uses the new value impossible. I guess that point is slightly moot when it'd take 2-3 releases to achieve the end goal and each release cycle is about a month.
- dillonmckay 7y agoOn the plus side for you, it seems you have each customer isolated from one another.
- jefftk 7y agoYou're not in a great situation, but there are still lessons from the post. For example, you could break upgrades into a series of changes that can be individually rolled back, and run smoke tests in between each one. But your biggest reliability improvement would come from getting this system moved to continuous deployment without downtime. Now you can make one change at a time and roll back if it doesn't work.
- aeorgnoieang 7y agoIf they're in anything like the last situation I was in that seems similar, continuous deployment may not be possible – not technically, but with respect to other considerations, e.g. management or even the business's finances. There's still a lot of small software companies maintaining, and selling, on-premises installations of systems that use, e.g. Microsoft Access as a front-end client. Even then, continuous deployment is possible, and all-else-equal, a huge improvement for the developers and support staff, but also something that lots of management or owners may be (reasonably) averse to committing to implementing.
- crummy 7y agoI don't know anything about databases. How do you roll back a significant schema change?
- abraae 7y agoFor a complex system, with many tables, rolling back a large change is very difficult.
- minxomat 7y agoIn your migration file(s) you have both forward and backward steps. If operations are reducers or otherwise destructive, the forward step backs that data up to separate "recovery" tables. The n+1 migration may delete recovery tables. That's how I do it most of the time. Sometimes with full snapshots. Ymmv
- twic 7y agoI believe the strategy described avoids needing to do that. You start by releasing a version of the software which can run with the old or new schema, then you apply the new schema, then you release a version of the software which actually uses the new schema. If you discover a problem at that point, you roll back to the previous version of the software, but leave the schema as it is. You then have time to figure out what to do, which may involve changing the schema again, but that will be as a forward change, rather than as a rollback. Some migration tools do support rollback scripts for schema changes, but unless you're actually testing these before release (deploy the new version in staging, accumulate representative data in the new schema, roll back the schema, deploy an old version of the app, test that it is doing the right thing), then they aren't really something you can rely on in production.
- tybit 7y agoIt really depends on the task but the general pattern is to split it up into separate steps, each of which is either low risk and/or easily reversible. Lets say you want to add a new non nullable column with foreign keys, to replace an old non nullable column of foreign keys to a different table that’s obsolete and needs to be deleted. 1) update the code to be ok with a new nullable column. Rollback: deploy previous version of code. 2) create the new column in DB with it’s desired constraint, but make it nullable. Roll back: delete the column. 3) have the code start populating the new column as well as the old. Rollback: deploy previous version of code. 4) start backfilling historical entries with the new column. Rollback: you can’t roll this back! 5) make new column non nullable. Rollback: make it nullable. 6) update code to read from new column, continuing to write to both. Rollback: deploy previous version of code. 7) make old column nullable. Rollback: you can’t roll this back! 8) stop writing to old column. Rollback: deploy previous version of code. 9) once you’re satisfied the old column is no longer used and the version of code from the previous step will never be deployed again, drop the old column. Rollback: you can’t roll this back! 10) rename the obsoleted table and see if anything breaks. Rollback: rename it back to its original name. 11) delete the obsoleted renamed table. Rollback: you can’t roll this back!
- ricardobeat 7y ago> using something with a solid schema that handles field names and types and gives you some way to find out when it's not even there [...] ex: protobuf In proto3 all fields are optional, and have default values, so it becomes impossible to detect the absence of data unless you explicitly encode an empty/null state in your values.
- asark 7y agoEw, what was the reason for that?
- deleted 7y ago[deleted]
- jrockway 7y agoBecause in proto2 everyone used "optional" anyway.
- akavi 7y agoGo structs behave the same way. I wonder if one influenced the other.
- tantalor 7y ago> impossible to detect the absence of data There are other ways: - Check if the value different than the default, e.g., empty string - If your data is repeated, then check number of data elements != 0
- couchand 7y ago> Check if the value different than the default, e.g., empty string which is a worse version of the GP's > explicitly encode an empty/null state in your values
- aplusbi 7y agoThis is true for primitives but not [completely] true for messages. Message still have the "hasMessage" semantics. So if you truly need to differentiate between unset and default for primitives, you can box them in messages.
- freakynit 7y agoI liked your writing style. Crisp and to the point. No BS marketing jargon.
- malicioususer11 7y ago"But what if we just break it on purpose and let the users go fuck themselves??? investors say they hate the users. I smell a blue chip." :3
- wyc 7y agoI love item #2, as it talks about writing code that can safely handle "future" enums and values as a result of rolled back code. Maybe we should call it the Sarah Connor Pattern. I haven't heard enough people discuss the deployment management of growing enums or state machine evolution. This is a problem more particular to software than hardware, as once hardware is shipped it's usually set in silicon, but growing of the state garden is an expectation in many software architectures.
- torbjorn 7y agoI love this and it makes me feel both excited and scared since I am suddenly realizing the ways my org is not in compliance with this good advice. Are there any good books that are full of more rules of thumb like these?
- dgacmu 7y agoThe Google SRE book (free online or available as dead tree): https://landing.google.com/sre/books/ https://landing.google.com/sre/books/
- barbarbar 7y agoShe has written many very interesting posts. Also one about "The One".
- jefftk 7y agoThis is a great list! One thing I would add, once you have everything on this list, is a way to experiment on your changes. Instead of flipping a flag, seeing your error rate jump, and flipping it back, you run an experiment where to flip the flag on 0.1% of requests. Now you can compare this traffic to a control group, and aren't stuck wondering "did errors go up by 5% during our rollout because we broke things, or by chance?". If things look as expected at 0.1% you can ramp to 1%, then 10% before releasing.
- C4stor 7y agoSeems like a typical article from I assume a gafa employee, good advice mixed with "how to be google even if you don't need too" advice.
- pjungwir 7y ago> Also, if you are literally having HTTP 400s internally, why aren't you using some kind of actual RPC mechanism? Do you like pain? I just had a discussion about this yesterday where we have an internal JSON API that auths a credit card, and if the card is declined it returns a status and a message. Another developer wanted it to return a 4xx error, but that made me uneasy. I think you could make a good argument either way, but to me that isn't a failure you'd present at the HTTP layer. 4xx is better than 5xx, but I was still worried how intermediate devices would interfere. (E.g. an AWS ELB will take your node out of service if it gives too many 5xxs, and IIS can do some crazy things if your app returns a 401.) Also I don't want declined cards to show up in system-level monitoring. But what do other folks think? I believe smart people can make a case either way. EDIT: Btw based on these Stack Overflow answers I'm in the minority: https://stackoverflow.com/questions/9381520/what-is-the-appropriate-http-status-code-response-for-a-general-unsuccessful-req https://stackoverflow.com/questions/9381520/what-is-the-appr...
- erikpukinskis 7y agoI always thought, if there’s a code that officially signals what we’re trying to signal, we should use it, and rely on it as “the signal”. But at the end of the day, your app must target a subset of available APIs. If you use every available API the surface will be too big and you will have snowballing complexity. So if you are already using another API which can signal what you’re trying to signal then feel free to institute a “everything we can expect returns 200” policy. That kind of decision is critical for putting an upper bound on your system complexity. I want to institute a “no default exports” policy in the JavaScript app I’m working on. It’s not that there aren’t places a default export makes sense—there are. It’s that prohibiting them will decrease the number of things our coders have to think about.
- s_kilk 7y ago> I always thought, if there’s a code that officially signals what we’re trying to signal, we should use it, and rely on it as “the signal”. This is a line I hear a lot in regards to this general situation, and the interesting part is that the 400 code really does not signal the thing the developer is trying to signal, not at all. But a lot of devs have a weird hang-up about this for some reason, and will go to any lengths to not send a 200 response. I blame the REST cultists.
- nieve 7y agoThe one I've seen missed most often in startups is directly implied by a lot of the other points and obvious to anyone with long experience: Take the time and put a lot of thought into how to break up your big transitions into smaller stages, each of which are functional. It's usually possible to at least narrow down the risky parts to a few finer grained steps and when something fails only rolling back one part to get to a good state is almost always faster and safer. It's very easy to get absorbed into the awareness of the high level change you're making and miss the details of the process. Even just sitting down together and outlining what you think is actually going to go on (and then breaking those down into what they each are comprised of) can make it really clear that you don't have to run as many giant risks. I'm occasionally amazed how brilliant people (including some with big names in devops) can forget it's an option. It's like taking small steps from stable to stable when you're going across a steep scree slope and only jumping when you have to - sometimes it feels riskier to take lots of small steps, but if you start to slide it can be a lot easier to recover from. Your chance of dying taking a big leap isn't the sum of the equivalent small steps. Perhaps complex computer systems have the equivalent of an Angle of Repose?
- maximente 7y agowhat is the load balancing story for these RPC services thusly recommended? it was completely glossed over as if it was not even relevant; i know gRPC uses HTTP/2 and persists connections so it's not as simple as throwing a proxy in front. that seems like a non-trivial point of friction when it comes to "just using solid storage/RPC formats" or whatever.
- elteto 7y agoFor gRPC there is Envoy, but I have not used it so I can’t give you more info on it. However, I think the point in the article is that you need to have well defined schemas for inter-service messaging. Something like protobuf or thrift or flatbuffers. Whether you layer gRPC on top of that is a separate concern. For example I have used Protobufs extensively at work but never gRPC, since we mostly have point-to-point connections. We checked in the message schemas into their own repo and all users across the company pull from it. We have Python and C++ codebases sharing the same schemas, it’s quite wonderful.
- punnerud 7y agoI managed releases for one of Norway’s largest hospitals, and when all in the ‘reliability list’ is checked and you have frequent releases the real headache is ‘cross system rollback’ between several systems/companies. Add that this is done with the whole hospital in emergency procedure..
- z3t4 7y agoHot patching is scary but it makes you optimize for the right things like easy to understand, easy to update, detectable early errors, easy to recover/rollback. And it makes you think and understand before writing the actual code.