6 ms·
How many more can we expect before they develop appreciation for testing _before_ they push to prod?
by 0xbear 9y ago
How many more can we expect before they develop appreciation for testing _before_ they push to prod?
- toyg 9y agoPushing changes on Monday morning... way to ruin everyone's week!
- cjbprime 9y agoNewly introduced errors in site code are only one of many sources of failure for a site like GitHub, and probably a fairly rare one.
- 0xbear 9y agoIt's a nearly weekly occurrence. Avoidable one, I might add. When was the last time a major Google service had a major outage?
- cjbprime 9y agoYou're still missing the point, by assuming that the outages are code related when you call them avoidable. Distributed systems at scale are terrifyingly hard to control. When I said "probably a fairly rare one", I didn't mean that the outages are rare. I meant that new code is probably a rare cause of the outages that happen. They have other causes unrelated to new code. (I'm also skeptical that GitHub major outages happen "nearly weekly", but I don't have data.)
- 0xbear 9y agoI'm not "missing" anything. I worked at Google for 7 years much of which was spent working on, you guessed it, distributed systems infrastructure. You guard against this by carefully canarying things and putting robust testing, monitoring, and deployment procedures in place. A release might take a few days, but you can be reasonably certain your users won't be your guinea pigs, and if shit does hit the fan, rollback is easy, and you can reroute traffic elsewhere while you roll back. Most of the time no rollback is needed: you just flip a flag and do a rolling restart on the job in Borg. For some types of outages (most of which users never even see) Google has bots that calculate monetary loss. And the figures can be quite staggering and motivating, so people do postmortems and try their best to make sure the outages don't happen again.
- cjbprime 9y agoWhy are you continuing to assume that this outage was caused by a release of some kind?
- peterwwillis 9y agoEvery change is a release if you squint right.
- cjbprime 9y agoI'm not even assuming this outage was caused by a "change". There are DoS attacks, infrastructure/network outages, storage pool problems.. immediately assuming that someone pushed some code and it broke things seems like an extremely short-sighted view on how production systems fail.
- user5994461 9y agoYou're too advanced for the typical reader. It's a startup site with half the people not having a test environment.
- concede_pluto 9y agoEveryone has a test environment; some people happen to send prod traffic to it.
- paublyrne 9y agoSo no Google service has ever experienced an outage? I distinctly remember Gmail being down on several occasions.
- 0xbear 9y agoGmail is several orders of magnitude larger than Github will ever be, and in recent memory I can only recall it being down once, and for a very small subset of users.
- kyberias 9y agoThey don't do any testing before push to prod?
- 0xbear 9y agoIt's called continuous delivery, man. Merge to master and it goes out the door automatically, even if it's busted to hell. All the cool kids are doing it.
- notyourday 9y agoHow was it merged to master if it did not pass the tests?
- 0xbear 9y agoShitty tests? Lack of integration tests? Lack of test coverage for a particular scenario? Test environment that does not represent the config that's deployed in production? There are literally hundreds of reasons why things like that could happen. Depending on the system in question you could have an isolated environment on which you replay prod traffic before cutting a new release and then investigate failures. All new features are engineered so that they could be easily turned off using flags. Once that's done you could canary your release to, say, 5% of your user base (at Google it'd be in the least loaded Borg cell), and if something goes wrong, you quickly disable busted features in response to monitoring and alerts. You let that stew for a while, then deploy your release worldwide, and start working on the next one.
- notyourday 9y agoLet me get it straight... So the advantage of CI/CD is not automatic testing + roll outs, rather it is removal of a test requirement? Why waste resources on CI/CD in that case? Just remove the test requirements and deploy. In fact, remove canary as well - the more traffic hits the broken release the faster it will become obvious that release is broken. The above was sarcasm. If your org has CD and it does not have CI validation step that provides 100% test coverage of your code, then you do not have a CD - you have MSP - Merge->Ship->Pray system in place.