4 ms·
Isn’t us-east-1 exactly that? All jokes aside, I actually asked my google cloud rep about stuff like this; they came back with some solutions but often the pro
by dijit 6y ago
Isn’t us-east-1 exactly that?
All jokes aside, I actually asked my google cloud rep about stuff like this; they came back with some solutions but often the problem with that is, what kind of failure condition are you hoping for?
Zonal outage (networking)? Hypervisor outage? Storage outage?
Unless it’s something like s3 giving high error rates then most things can actually be done manually. (And this was the advice I got back because faulting the entire set of apis and tools in unique and interesting ways is quite impossible)
- londons_explore 6y ago> Unless it’s something like s3 giving high error rates Just firewall off the real s3, and point clients at a proxy which forwards most requests to the real s3 and returns errors or delays to the rest.
- eru 6y agoI found that in practice for APIs, delays are often much worse than errors. Mostly because programmers seem to have an easier time thinking about errors, and their programming language might even encourage them to handle these kinds of errors, but arbitrary delays often slip by unanticipated.
- londons_explore 6y agoGoogles approach to RPC deadline propagation solves this. I'm sad that the vast majority of libraries don't support it. The simple concept is all services/API's should take a deadline parameter. The call should either complete before that time, or return an error. If you are writing code for a service, and you get an incoming request, you send the same deadline for any requests to other services.
- nitrogen 6y agoAs long as everything is idempotent. You don't want to have this happen: Host A calls host B Host B runs SQL+COMMIT DB commit finishes Host B deadline expires DB success reaches host B Also would need fanatical time synchronization. Maybe even PTP rather than NTP?
- londons_explore 6y agoGoogle has awesome clock sync in its datacenters (sub millisecond), and in general all RPC calls should be idempotent (since the network or either host can fail at any point). I think one might be able to make this approach work with poor clock sync by having the request contain the number of milliseconds left, rather than an absolute time. It isn't great if network delays dominate, but typically it's queueing delays that dominate most webservice responses.
- tgsovlerkhgsel 6y agoThis (or equivalent) will happen. You can either design your software to handle it correctly, or you can have buggy software. Clock sync doesn't really help, especially if a request can trigger multiple independent action that can individually succeed or fail. The simplest example where this goes wrong is posting a post on Reddit. You write your text into a text box and submit. You get an Error 500. You cannot know if your post went through (even if you go check whether your post is visible, sometimes the replica that you got your response from hasn't gotten your post yet). And they don't do any deduping, so if you just retry, you may end up with multiple copies of your reply.
- caymanjim 6y agoYeah, us-east-1 is pretty good at failing already. We lost us-east-1c for most of the day about a week ago due to a fiber line being cut. I'd estimate that AWS manages fewer than "three 9s" in us-east-1 on average. Not across the board, but at any given time something has a decent chance of not working, be it an entire AZ, or regional S3, etc. They're still pretty reliable, and I like the idea of a zone with built-in failure for testing things, but your joke about us-east-1 is based in solid fact.
- FridgeSeal 6y agoap-southeast-2-b in my experience hahaha