6 ms·
> https://landing.google.com/sre/sre-book/chapters/reliable-product-launches/ https://landing.google.com/sre/sre-book/chapters/reliable-pr... > Almost all upda
by ignoramous 8y ago
> https://landing.google.com/sre/sre-book/chapters/reliable-product-launches/ https://landing.google.com/sre/sre-book/chapters/reliable-pr...
> Almost all updates to Google’s services proceed gradually, according to a defined process, with appropriate verification steps interspersed. [..] The first stages of a rollout are usually called "canaries” — an allusion to canaries carried by miners into a coal mine to detect dangerous gases. Our canary servers detect dangerous effects from the behavior of the new software under real user traffic.
> Canary testing is a concept embedded into many of Google’s internal tools used to make automated changes, as well as for systems that change configuration files. [..] If the change doesn’t pass the validation period, it’s automatically rolled back.
Any Googlers around to explain why Canaries didn't catch configuration causing this side effect and stopping the deployment before the issue cascaded globally?
The outage here reminds me of issues CloudFront faced wrt configurations in their earlier years. Here's master presenter Harvo Jones going into a bit more detail: https://youtube.com/watch?v=n8qQGLJeUYA https://youtube.com/watch?v=n8qQGLJeUYA
Kind of vindicates AWS' decision (after major outages to DynamoDB and S3) to not just isolate regions or zones but resources within those regions/zones further into what they call "cells" https://youtube.com/watch?v=swQbA4zub20 https://youtube.com/watch?v=swQbA4zub20
Tumblr spoke abt "cells" too in the context of achieving isolation at scale, but in a quite different way: http://highscalability.com/blog/2012/2/13/tumblr-architecture-15-billion-page-views-a-month-and-harder.html http://highscalability.com/blog/2012/2/13/tumblr-architectur...
As these known-knowns (and the Google sre-book is teeming with these) get designed around, it'd be interesting to see why/how future outages would happen.
- quicklime 8y ago> Any Googlers around to explain why Canaries didn't catch configuration causing this side effect and stopping the deployment before the issue cascaded globally? I'm not (any longer) a Googler, but according to the postmortem this was a load-related incident. Canary changes only have a small amount of traffic directed at them (by definition) and the problem probably didn't become apparent until after the change was promoted from canary.
- j_coder 8y agoSo Google don't test for load? They should ;)
- londons_explore 8y agoMost peoples load tests involve taking a system to breaking point, then turning the load off and going to lunch. People need to gradually reduce the load after the breaking point to check the system recovers. Load Tests are also pretty hard to do in distributed systems. If you test the application alone, you probably won't find most of the issues. You'll need to test the application complete with all it's dependencies (databases, load balancers, failover mechanisms, external servers, etc.). You'll also probably want to test it with representative user requests, and all databases filled with representative data. That in turn typically means your loadtest system will need to be as big and expensive as your production system. Have fun explaining to the boss why you need to double the infrastructure costs. If you do it in the cloud, you can just do a loadtest for 10 mins per day and save a bunch of $$$, but you still need tooling to be able to deploy a complete replica of your system, fill it with realistic data and send it realistic requests, all automatically. Using real user data and logs of user interactions and real user requests is best for loadtesting, but comes with it's own risks. You need to make sure the loadtest systems doesn't send out emails to users, or accidentally communicate with the real production systems in any way. It also means you have to secure your loadtest infrastructure as well as your production infrastructure. GDPR data deletion requests need to apply there too, etc.
- londons_explore 8y agoGoing further... Googles 'Canary' system is sub-optimal and risky. It picks a subset of the system (say 5%), and deploys the change there. It then observes this 5% of the system, and if it's okay, it rolls out to the rest of the system. Compare that to a continual-canarying system: * Start deploying the new code gradually (say 1 container/pod/task every second). * While the above is in progress, continually collect stats from upgraded and non-upgraded systems. * If the stats differ to any significant level, halt the rollout. The continual-canarying system is much better because the sample size to see a 'significant' change depends on the size of the change and the amount of natural variation in the sample. The size of the change obviously can't be known beforehand. The variation varies widely depending on the metric of interest (cpu, memory use, request latency, etc.). It is therefore impossible in googles canarying system to choose a suitable subset size to detect (with a sufficiently high statistical power) small but important changes in a metric while still stopping the rollout early enough for catastrophic changes in a metric. Continual-canarying effectively solves this issue by gradually increasing the sample size as the rollout goes on. In reality, neither continual-canarying or google-canarying work well for detecting anomalous metrics because the rollout process itself causes applications to restart, which in itself changes their performance characteristics (cold caches, empty queues, first-use delay, jit optimization, etc.). The solution to that is to do two simultaneous continuous-canary rollouts - one of the existing version (effectively just a restart) and one of the new version, and use those two groups as input to the logic to decide if any metrics of interest have statistically significant changes. Please go implement this stuff Google! It won't be hard to do, and will really increase the safety of your rollouts. (Even though I know it wouldn't have helped this time)
- syn0byte 8y agoIn short; QA forget to ask the bartender where the bathroom was.
- cromwellian 8y agoI dunno what AWS cells are and if the terminology matches, but Google's Borg already has a concept of cells internally, a cluster can hold cells, and each cell can represents a kind of job.
- ignoramous 8y agoFrom what you describe the 'AWS cells' are very different than Borg's. My view is that at a high level aws cells are isolated, fuzzy-partitioned, migratable, heterogeneous instances of a complete service stack that are managed such that they always fail independently of each other; contact points among these cell instances, if required, are established via highly available intermediate services (like ReplicatedStateMachinesOrStorage); the clients are routed to assigned cells by a highly available routing layer (like NetworkTrafficShapers/DNS), which can also help facilitate migration of clients from one/more cell to another. I encourage you to view the re:invent presentation on it [0] if resiliency in distributed systems is something that interests you. [0] https://youtube.com/watch?v=swQbA4zub20 https://youtube.com/watch?v=swQbA4zub20
- skj 8y agoThat description matches how I think of borg cells as well. (Googler)
- ignoramous 8y agoThanks. I read the paper now and I kind of get why you guys think they're similar. I find them similar too in what they're trying to achieve [blast-radius minimization], but I think AWS' concept of cells [going by re:invent presentation] is more of an architectural pattern with strong inclination towards certain non-negotiable reqs ["thinnest possible routing layer", "workload migration among cells is a first-class citizen", "avoiding critical cross-cell tasks/dependencies other than migration", "creating/removing cells are zero-downtime events"] for different flavours of services [stateful vs stateless (dynamodb vs lambda?), zonal versus regional versus global (ebs vs kinesis vs route53), for example] than a concrete implementation like Borg. I might be wrong on all accounts, though.
- shereadsthenews 8y agoIt sounds like this public report may just be using the word "configuration" to mean anything other than the software itself. I imagine that what really happened is they got alerted to high disk usage and the operator issued a major compaction which took out the bigtable. That's caused numerous Google outages and is consistent with the report.
- puzzle 8y agoCompactions do NOT reduce disk usage. On the contrary, they rewrite the data, so they generate new and presumably smaller SSTables. Which means more disk usage. It's the garbage collector on the master that removes files. You can combine the two, but that requires some tooling that compacts a section of the table, then triggers (or waits for, I can't recall) the garbage collector.
- shereadsthenews 8y agoThis will not be of general interest to HN, but I disagree. If compaction is behind then commanding a higher compaction rate can free space, and it can also bring down your service. You can also call for higher rates of splits and merges with the same effect. I have no idea of those actions contributed to this outage.
- puzzle 8y agoCompactions happen on the tablet server. They only write data. By definition, they increase disk usage, right away. Only the garbage collector, which needs global state and thus runs on the master, can actually delete files in Colossus and bring down disk usage. It doesn't always run as frequently as one would think or hope. It definitely doesn't run immediately after one tablet has been compacted. I remember one incident where a team got an alert that they were at 90% of quota or so. They compacted data too aggressively and they actually hit 100% before the GC could do its job. That's why I mentioned that you should compact a fraction of the table, then let GC run. Google being Google, I'm sure such a tool has been written, in several variants, by N different teams.
- masto 8y agoObviously not authorized to release more details than have already been made public, but when the book hits the real world sometimes you find new failure modes, software has bugs, or humans find creative mistakes. It's also very hard to build global scale systems with zero possibly of global failure. But every time a crack is found, you learn something and do what you can to eliminate the whole class of related failure modes. Disclaimer. I don't work on the thing that broke here but I am in SRE. Speaking in broad generalities.
- joatmon-snoo 8y agoGoogler here, but haven't looked at any of the details of this incident. One of the tricky things about canaried releases and staggered rollouts is that to implement them, you need another layer of configuration to manage that. And then that's another potential source of outages - and scary ones too, because your configuration is so closely tied to prod itself, . It's kind of the whole "my test needs a test, and the test of my test needs a test, and ..." issue. Plus, you can only spend so much time testing out disaster scenarios before other priorities take over - if your customers are asking for features A, B, C, well, at some point it becomes a judgment call whether you spend time on the new features or testing the resiliency of your service. Even when you do test the hell out it, it's hard to guarantee you've tested literally every edge case. The other thing is that, as you point out, as you add more and more mitigations for failure scenarios that you have experience with, the fraction of the failures that you've never seen before starts to increase. It's somewhat obvious, because by definition a regression test safeguards against failures that you've experienced, but it's an important thing to keep in mind when thinking about exactly what the tests you do write will keep you safe from.
- dsl 8y ago> into what they call "cells" Google literally invented this concept, and it has been spread to other companies by Xooglers. https://pdos.csail.mit.edu/6.824/papers/borg.pdf https://pdos.csail.mit.edu/6.824/papers/borg.pdf
- ignoramous 8y ago> Google literally invented this concept Did they? Here's a summary of James Hamilton's paper (then a Microsoft employee) on building large-scale internet applications published in 2007, 8 years (?) before Borg was made public knowledge: https://blog.acolyer.org/2016/09/12/on-designing-and-deploying-internet-scale-services/ https://blog.acolyer.org/2016/09/12/on-designing-and-deployi... Partition the system in such a way that partitions are infinitely adjustable and fine-grained, and not bounded by any real world entity (person, collection, customer, etc.). > it has been spread to other companies by Xooglers. https://twitter.com/bcantrill/status/1092849514229059585 https://twitter.com/bcantrill/status/1092849514229059585 I guess the notion that Google is an all-superior tech giant that faces unique problems and solves them in the best way possible is wide spread misplaced notion among Googlers?
- joshuamorton 8y agoI don't know about this case, it wouldn't surprise me either way. Borg is certainly old, probably pre 2008. The same story as Mr/Hadoop wouldn't surprise me.