4 ms·
Braintree (IIRC) had a really clever migration strategy, although I can't seem to find the blog post now. They paused all traffic at the load balancer, cut ove
by luhn 5y ago
Braintree (IIRC) had a really clever migration strategy, although I can't seem to find the blog post now. They paused all traffic at the load balancer, cut over to the new DB, and then resumed traffic. No requests failed, just a slight bump in latency while the LBs were paused.
This app apparently had robust enough retry mechanisms that they were able just eat the errors and not have customer issues—Color me impressed! I'm not sure how many teams can make that claim; that's a hard thing to nail down.
- scrollaway 5y agoI remember reading something like that about Adyen. That might be why you're unable to find it.
- nezirus 5y agopgbouncer has PAUSE comnand, which can be used for seamless restarts/failovers or similar
- rattray 5y agoI wonder if that was it (Adyen and Braintree are competitors). Here's the post: https://www.adyen.com/blog/updating-a-50-terabyte-postgresql-database https://www.adyen.com/blog/updating-a-50-terabyte-postgresql... Discussed here: https://news.ycombinator.com/item?id=26535357 https://news.ycombinator.com/item?id=26535357 They architected their application to be able to tolerate 15-30min of postgres downtime.
- xyst 5y agoprobably had a good circuit breaker design. would love to read the article though.
- simonw 5y agoI think that's covered in this talk (I've not watched the video though): https://www.braintreepayments.com/blog/ruby-conf-australia-high-availability-at-braintree/ https://www.braintreepayments.com/blog/ruby-conf-australia-h...