3 ms·
That was a pretty humble and good read. I don't think I'd have seen the autovacuuming issue coming. Actually, this quote is a perfect example of how subtle and
by nickpsecurity 10y ago
That was a pretty humble and good read. I don't think I'd have seen the autovacuuming issue coming. Actually, this quote is a perfect example of how subtle and ridiculous these issues can be:
"During the event, one of the shard databases had all queries on our primary table blocked by a three-way interaction between the data path queries that wanted shared locks, a "transaction wraparound" autovacuum that held a shared lock and ran for several hours, and an errant query that wanted an exclusive table lock."
That's with well-documented, well-debugged components doing the kinds of things they're expected to do. Still downed by a series of just three interactions creating a corner case. Three out of a probably ridiculous number over a large amount of time. Any system redoing and debugging components plus dealing with these interaction issues will fare far worse. Hence, both of our recommendations to avoid that risk.
Note: Amazon's TLA+ reports said they model-checkers for finding bugs that didn't show up until 30+ steps in the protocols. An unlikely set of steps that actually was likely in production per logs. Reading such things, I have no hope that code review or unit tests will save my ass or my stack if I try to clean-slate Google or Amazon infrastructure. Not even gonna try haha.
- agentultra 10y ago> Note: Amazon's TLA+ reports said they model-checkers for finding bugs that didn't show up until 30+ steps in the protocols. An unlikely set of steps that actually was likely in production per logs. Reading such things, I have no hope that code review or unit tests will save my ass or my stack if I try to clean-slate Google or Amazon infrastructure. Not even gonna try haha. For those unfamiliar with the reference, there was an eye-opening report from Amazon engineers who'd used formal methods to find bugs in the design of S3 and other systems several years ago [0]. I highly recommend reading it and then watching as many of Leslie Lamports talks on TLA+ and system specifications as possible. [0] http://research.microsoft.com/en-us/um/people/lamport/tla/formal-methods-amazon.pdf http://research.microsoft.com/en-us/um/people/lamport/tla/fo...
- nl 10y agoAnyone who's worked with Postgres at scale would guess autovacuum. Postgres doesn't have many weaknesses, but most of them relate to autovacuum.
- nickpsecurity 10y ago"Anyone who's worked with Postgres at scale would guess autovacuum." Well, there's knowing it's autovacuum-related then there's the specific way it's causing a failure. First part was obvious. The rest took work. "Postgres doesn't have many weaknesses, but most of them relate to autovacuum." Sounds like that statement should be on a bug submission or something. They probably need to replace that with something better.
- anarazel 10y agoIt's known to the postgres developers, and we are working on it. This specific issue (anti-wraparound vacuums being a lot more expensive) should be fixed in the upcoming 9.6.
- nickpsecurity 10y agoAwesome! I already push Postgres and praise its team for the quality focus. Just extra evidence in your favor. :)