12 ms·
An update on recent service disruptions
- HoyaSaxa 5y agoOne thing that has caused me much frustration and is probably putting Microsoft in tremendous legal risk is that GitHub Actions will often be incorrectly billed during these incidents. Others have said the only recourse is to contact support. I finally pulled the trigger today.
- speedgoose 5y agoIt’s interesting to read that so many systems and activities are dependent on a single point of failure : the main primary MySQL node at GitHub.
- eatonphil 5y agoI can imagine what they're in a rush to refactor.
- cyberpunk 5y agoI mean, they have $MEGABUCKS, they could probably get 1/2 the team who maintains mariadb to come in and work for them if they wanted, and they still have a giant single db node doing writes and struggle to fail it over. We're doomed >_< You would think it wouldn't be THAT hard to shard something like GitHub effectively. I mean, all user accounts/repos starting with the letter 'a' go to the 'a' cluster and so on seems not exactly science-fiction levels of technology.
- sillysaurusx 5y ago> they could probably get 1/2 the team who maintains mariadb to come in and work for them if they wanted The Mythical Man Month has a few things to say about that. (It's tempting to feel that the information is outdated, but in my experience it still seems true.)
- cyberpunk 5y agoMy point was, that they've not done that, is because it wouldn't help. This is an architectural problem, which even if they had the massive expensive brains behind something like mysql on their team they couldn't fix it. (at least, I'm guessing, I think this kinda architecture doesn't scale even if they could kick the can down the road a few times..)
- sillysaurusx 5y agoOh, my apologies! Yes, that makes sense.
- deleted 5y ago[deleted]
- throwusawayus 5y agoit's profoundly strange that github has not properly sharded yet. essentially all large social networks are or have used sharded mysql successfully, this is not rocket science livejournal, facebook, twitter, linkedin, tumblr, pinterest all use (or formerly used) sharded mysql and most of these are at larger db size than github i will also repeat my comment from another recent thread: i just cannot understand how 20+ former github db and infra people recently left to join a db sharding company. this makes no sense whatsoever in light of github's lack of successful sharding. wtf is going on in the tech world these days
- drewbug01 5y ago> i just cannot understand how 20+ former github db and infra people recently left to join a db sharding company. this makes no sense whatsoever in light of github's lack of successful sharding. I believe you have the chain of causality backwards here. In fact, I think it suggests that talent that went to planet scale is perhaps not the issue.
- throwusawayus 5y agoi'm not suggesting that these departures are the cause of github's issue. rather, i'm saying i don't understand why such a large group from github was hired by planetscale if they did not have experience successfully sharding or successfully leveraging vitess this is like if you were building a high-rise condo, would you hire the architects or management company from the building that collapsed in surfside florida? sure, they know what NOT to do next time, but that doesn't mean they do know what TO do
- drewbug01 5y agoI understand what you’re getting at… but why is it you presume that the employees now at planetscale are the reason GH couldn’t shard out their databases? Like, there’s another angle here: management, yeah? Another way of reframing it is “maybe the folks hiring at planetscale know the inside baseball about GH infrastructure”. For example: https://www.linkedin.com/in/isamlambert https://www.linkedin.com/in/isamlambert
- pcj-github 5y agoLots of people commenting here about how easy it must be to shard mysql at GitHub scale...
- otterley 5y agoJust because you have money doesn’t mean you can hire anyone you want to. Talent is scarce, especially in highly-specialized roles, and people have free will. I think the complexity of GitHub’s data management requirements would surprise you. Better to withhold judgment until you possess all the facts.
- prepend 5y agoMy fear is that this seems like a cover excuse for moving off MySQL. The bug will be too hard and they’ll move off. They will choose SQLServer and take a lot of time to convert and then have even more outages.
- speedgoose 5y agoI bet on Azure Cosmos DB.
- deleted 5y ago[deleted]
- nimbius 5y ago100% agreed. a lift-and-shift migration to Galera and modern MariaDB wouldnt be hard, but knowing MS there are mid managers waiting in the wings to swoop in and drive this into the ground with azure/sqlserver, the former of which posted 8 outages in the past 90 days alone. this is classic Microsoft. spend a ton of money for something very valuable -- in this case virtually all developer marketshare -- and then casually pedal it into the ground while you lie about the KPI's to C levels (IIS marketshare on netcraft as a function of parked websites at GoDaddy to dominate over Apache) and keep it on life support with other revenue streams (XBox) for the next 16 quarters until it becomes a repulsive enough carbuncle to shareholders that it gets the axe (Microsoft phone.) then in a year, limp into the barn with another product nobody else but you could afford to buy (minecraft) and slowly turn it into a KPI farm for Microsoft account metrics to drive some other failing product (Azure) and keep the C level happy while you alienate virtually every player with mechanics or requirements they hate.
- throwusawayus 5y agogalera is not a solution for scaling out writes, full stop galera has lower max writes/sec than a traditional async single master because it's a cluster. the other members of the cluster need to ack the writes, and all members are doing all the writes, so adding machines does not increase your max writes
- wincent 5y agoLinked in the article is this other one, "Partitioning GitHub’s relational databases to handle scale" (https://github.blog/2021-09-27-partitioning-githubs-relational-databases-scale/ https://github.blog/2021-09-27-partitioning-githubs-relation...). That describes how there isn't just one "main primary" node; there are multiple clusters, of which `mysql1` is just one (the original one — since then, many others have been partitioned off).
- throwusawayus 5y agofrom that article it sounds like they are mostly doing "functional partitioning" (moving tables off to other db primary/replica clusters) rather than true sharding (splitting up tables by ranges of data) functional partitioning is a band-aid. you do it when your main cluster is exploding but you need to buy time. it ultimately is a very bad thing, because generally your whole site is depenedent on every single functional partition being up. it moves you from 1 single point of failure to N single points of failure!
- wincent 5y agoTowards the end: > In addition to vertical partitioning to move database tables, we also use horizontal partitioning (aka sharding). This allows us to split database tables across multiple clusters, enabling more sustainable growth.
- throwusawayus 5y agoyes i know, the fact that this is a small blurb at the bottom of the article (which is largely about functional partitioning) exactly proves my point
- dikei 5y ago>> functional partitioning is a band-aid. I disagree, functional partitioning is not a band-aid, but an architectural changes that in the end can reap much more benefit than simple data sharding. >> your whole site is dependent on every single functional partition being up. it moves you from 1 single point of failure to N single points of failure! Not necessarily, it can also be that only some parts of your site are dead while others work perfectly fine.
- zamalek 5y agoThis is what happens when you handwave CAP theorem. I am shocked at the number of people who are shocked that Github couldn't easily resolve this. Tech debt has to be paid in full, whether it's person-hours, or downtime, or both.
- KMag 5y agoSomehow, I just assumed that because git is based on content-addressed storage DAG that GitHub internally heavily leveraged a distributed hash table and content-addressed storage DAG. With content-addressed storage, all of the stored data is immutable, so caching is significantly simplified, and a DHT can scale very well horizontally. Care still needs to be taken around the transactions that change DAG roots (heads of branches), but the rest is just making sure immutable blobs are sufficiently replicated in the DHT. A DHT isn't a great fit for really rapidly changing data like metrics and such, but I figured that everything required to keep the basic git clone, commit, and web serving functionality is a pretty natural fit for a DHT. I figured even code search had a relatively small amount of storage for indexing recent commits, with compaction of that data resulting in per-token skiplists stored into the DHT. That way, failure outside of the DHT serving path still allows code search for all commits older than the latest (incremental) compaction. Distributed refcounting or Bloom filters could be used for garbage collecting the immutable blobs in the DHT. The probability of a reference cycle in SHA-256 hashes of even quadrillions of immutable blobs is vanishingly small.
- crdrost 5y agoPermissioning/ACLs is something that you really want to get synchronous whenever possible, you don't want caching and pipeline delays if you can help it. Some maintainer is acting in bad faith, someone else quickly locks them out of all the repositories that they have not yet thought to corrupt, then it looks really really bad if they find out that they still have permission to run the CI/CD scripts, maybe with malicious substitutions, even though you blocked them from being able to push commits up. The other part of that is, it's not too hard to get right. See what GitHub was doing. People don't change ACLs that often, nowhere near as often as you read them. If they did, you could rate-limit them. And the rest of the problem is, you have one writer and many read copies in one place and you can enforce boundaries on how stale the data gets on the read copies. Two hard problems, first is cache invalidation. Of course, they have found out that they are big enough to shard, one would have hoped that they would have found that out in a gentler way, much sooner. But let's not pretend that their architecture makes no sense, it's a fine architecture, they just happened to outgrow it in a bad way.
- paxys 5y agoOutage checklist - Was it DNS? - Was it a bad config update? - Was it an overloaded single point of failure? There's rarely a #4
- doublerabbit 5y ago#5 Is it plugged in? Build&Configure server, A few hour drive it down to the DC, rack the server. Get back home, try to access. No luck. Turned out I had forgotten I had forgotten to connect power and turn it on.
- HL33tibCe7 5y agoA proposed fourth: was it BGP? Although most of those fall under “bad config update” (although likewise that applies to DNS).
- jenny91 5y agoWhen has BGP caused a serious outage at a website?
- karlding 5y agoThe big Facebook outage from last fall [0]? [0] https://blog.cloudflare.com/october-2021-facebook-outage/ https://blog.cloudflare.com/october-2021-facebook-outage/
- jenny91 5y agoThe cause wasn't BGP though, it was just caught in the middle of a big mess and made it very hard to undo a botched config change.
- cranekam 5y agoBGP itself didn’t cause this problem. An operational error/bug/whatever that manipulated network configurations was the cause of the outage. BGP is just the messenger: if you misconfigure it your network will break, but it’s not BGP’s fault. A more accurate checklist item would be “is it the network?”
- longcommonname 5y agoOur internal monitoring has seen more outages than they listed here. Theres been 4 full days where github actions have been mixed between completely broken and degraded status. It's nice to finally get some comms, but this is incredibly late and incomplete.
- sk5t 5y agoGH Actions felt so fundamentally janky and under-documented when I last looked at it (October?) that this doesn't come as a surprise. I really can't foresee using it with requirements any stricter than "nice when it works."
- jhugo 5y agoYup, same here. We had periods of more than 8 hours (during Asia daytime / US nighttime) where we couldn't push or pull, which were not mentioned on their status page at all.
- swiftcoder 5y agoI would not be surprised to learn that they have similar procedures to one of my past employers, where exec signoff was needed to even publish an update to the service status dashboard...
- calcifer 5y agoTLDR: They still don't know why this is happenning.
- rvz 5y agoOh dear. So other than those double outages, I should expect at the very least that GitHub should have at least one outage every month, not 3 or 5 in one month? I don't think we are going to see GitHub be up for a full month without an incident anytime soon and I guess my entire comment chain [0] on the whole situation has aged for two straight years in a row, especially yesterday's one from [1]: >> Until the next time GitHub goes down again (hopefully that won't be in another month's time). *Goes down the very next day* That says it all really. Lets reset the counter and try this again. [0] https://news.ycombinator.com/item?id=30779275 https://news.ycombinator.com/item?id=30779275 [1] https://news.ycombinator.com/item?id=30767821 https://news.ycombinator.com/item?id=30767821
- throwdbaaway 5y agoThey mentioned nothing about CPU nor disk I/O, with the immediate cause being "reaching maximum number of connections", i.e. the following error from vitess vttablet: ResourceExhausted desc = transaction pool connection limit exceeded My unsolicited advice: this looks like a classic case of https://bugs.mysql.com/bug.php?id=53825 https://bugs.mysql.com/bug.php?id=53825, which remains unfixed after 12 years. Here's the solution space: * Oracle added NOWAIT and SKIP LOCKED to MySQL 8.0, and called it a day. This is a bad solution because it just changes from one extreme (unbounded queue) to another (no queue). * AliSQL added Statement Queue via hint (their older implementation extented the UPDATE syntax instead), which should work great, but do require modifying the write queries: https://partners-intl.aliyun.com/help/en/doc-detail/144127.htm https://partners-intl.aliyun.com/help/en/doc-detail/144127.h... * At $prevjob we added block-level queue within InnoDB, based on an older experimental patch from AliSQL which they didn't actually bring into production. No app change required. * That bug was reported by Facebook, but there is nothing relevant in their MySQL fork. They probably outsource the lock queuing logic to their Memcached layer. The problem with this bug is that, without a correct analysis, the knee-jerk reactions are typically: * adding more CPUs * throwing in even faster disks * bolting on a sharding layer like vitess * blaming your DBaaS providers (well, they are to be blamed if they intend to stick with "vanilla Oracle MySQL") Question to Github: I see you running pprof on vttablet, but have you run perf on mysqld?
- renewiltord 5y agoHuh, one guy on HN did say it was the DB that was the problem earlier on. Neat!
- EscargotCult 5y agoFor anyone looking, here's the comment: https://news.ycombinator.com/item?id=30779291 https://news.ycombinator.com/item?id=30779291
- egberts1 5y agoThe hardest part is drafting a series of questions for the end-user to understand and answer before we get that “MAGIC-PRESTO-CHANGO” get those configuration files that just works. I blame the program providers. Some Debian maintainers are trying to do this simple querying of complex configurations (dpkg-reconfigure <`package-name>`). And I applaud their limited inroad efforts there because no else one has seem to bother. I have made a bash script to configure for each Chronyd, named, sshd, dhcpd, dhclient, NetworkManager, systemd-networkd, /etc/resolv.conf, amongst many. They try and ask simple questions and glue appropriate settings then run their own syntax checkers (most are provided by the original stream). Postfix, Shorewall, and Exim4 remain a nightmare to my evolving design. CISecurity and other government hardening docs were applied as well and then some I took even further like Chrony had its file permissions/ownership even further and MitM block feature as well. These are dangerous scripts where it can write files as root but as a user, you will instead get configuration files written out in appropriate directories under `build` subdirectory. If these designs work across Redhat/Fedora/CentOS, Debian/Devuan, and ArchLinux well, I may forge even further. https://github.com/egberts/easy-admin https://github.com/egberts/easy-admin
- rattray 5y agoI find it surprising that: - The team was not able to sufficiently diagnose the bad query program, and/or devise emergency mitigations, by 1400 UTC March 17 (24h after the first incident). - They were not able to reduce the load on the db more quickly (eg; cranking up rate limiting, shutting down async or noncritical services and features, etc). At $prevjob we had the ability to do things like this at the push of a button, and generally would have within 90min of incident onset to help systems heal. Here they did eventually figure out they could throttle webhooks but it took several days of repeated incidents. - They did not see connections creeping up towards their limits and have emergency mitigations prepared in case of spikes like this. - The simple failover described on March 22nd (for example) took almost 3h to complete. - They decided to enable profiling during the approximate time window of their load spikes, without sufficient resources. - They lept to failover their db when the proxy had issues, when it seems the failover can take multiple hours to successfully complete(??). - They did not have sufficient sampled trace/log data to form a hypothesis without this profiling. - New services like Packages and Codespaces rely on the `mysql1` primary db. Certainly I have much empathy for the responding teams – I have scars from responding to some very painful degradations myself – but as a Github customer this does leave me concerned.
- thrdbndndn 5y agoWhat is "$prevjob"? I thought it's meant to be a stock symbol (by Twitters' convention at least) but I can't find any company with this ticker.
- jakon89 5y agoI am just wondering if this is Vitess related. I am moving tens of PBs into Vitess and it would be really bad if these incidents are vitess related...
- foxknox 5y agoI know a lot of GitHubbers and they say the irony is that this is one of the only clusters that is not on Vitess.
- 0xbadcafebee 5y agoI mean, you can rest assured that you will have problems with Vitess at some point. All software is buggy, and there is no guarantee one of those bugs won't be catastrophic. You should plan for a method to completely replace your entire production cluster in the event of failure. This can be quite difficult with horizontally-scaled databases. It's also not uncommon to hit a scaling limit for which there's no simple solution, or not be able to build out a replacement production cluster due to unplanned-for limitations.
- dabeeeenster 5y agoI can't understand why they are not: a: building new systems on an internet scale non-relational datastore b: migrating relevant features onto an internet scale non-relational datastore Are there a lot of folk who are operating at Github's scale still deploying core services to platforms like MySQL?
- vorpalhex 5y agoNon-relational stores are not a replacement for relational ones. They have different consistency and CAP behavior.
- dabeeeenster 5y agoI understand that but you can still write your services around datastores that can scale out better than MySQL?
- MrStonedOne 5y agoNot if you need the data to have a different cap and consistency behavior.
- jpgvm 5y agoMySQL scales out just fine. Anyone who thinks otherwise is simply not well versed enough in how to properly shard data and manage replication topologies. I say this as someone that despises MySQL. Same obviously applies for PostgreSQL but is actually slightly harder for a number of reasons with less out of the box solutions like Vitess available.
- 0xbadcafebee 5y agoThis reminds me of something I often forget about. The value of GitHub to Microsoft may be significant brand-wise, but it might not be a huge revenue source comparatively. Even if it were a huge revenue source, most of the revenue might still be shuffled off to the corporate behemoth to cover its balance sheet rather than reinvested in improving the product. Meanwhile the actual value of GitHub to organizations around the world is probably a couple order of magnitude higher than its actual revenue. At some point it's actually cheaper for a coalition of international organizations to fund inventing a backwards-compatible GitHub replacement that is very resilient to failure, rather than wait for GitHub to get a measely enough budget increase from Daddy Micro$oft to shore up their legacy MySQL database.
- krageon 5y ago> fund inventing a backwards-compatible GitHub replacement There are so, so many and there have been for years. Folks use Github because it's what they know, not because an alternative is hard to find.
- eek2121 5y agoGitHub is becoming a huge revenue source for them FYI. They are stealing a TON of business from competitors. Shoot, if Microsoft had a real competitor for Jira, they would probably steal even more business.
- freeone3000 5y agoWhat happened to Azure DevOps?
- 8n4vidtmkvmk 5y agofwiw i moved to gh from bitbucket after the mercurial fiasco (they decided to just straight up delete everyones repos)
- ksec 5y ago>(they decided to just straight up delete everyones repos) I had to look it up. Wow... just wow. https://bitbucket.org/blog/sunsetting-mercurial-support-in-bitbucket https://bitbucket.org/blog/sunsetting-mercurial-support-in-b...
- polote 5y agoThe issue with transparency is not that when you are good everyone see it but that when you don't know what you are doing everyone see it too. It is honestly a bit humiliating for a company like github to both says : We have had issues with our database for years and have still not found a solution. I mean what? We have been down several days in a row and we have no idea yet how to solve the issue apart from throttle limiting webhooks
- taywrobel 5y agoThat’s what happens when all your best DB engineers jump ship to go to PlanetScale I guess.
- samlambert 5y agoI know this could be taken as a compliment but I don’t feel it’s a fair assessment. Maintaining large scale SaaS services is a constant challenge, and the GitHub team work really hard to build a platform that serves a gigantic community of developers. Downtime happens to everyone.
- taywrobel 5y agoI worked at GitHub up until last October. I’m keenly aware of the challenges of maintaining it, and how much hard work the teams puts in. However I think it’s hard to ignore in light of recent issues that there has been a large amount of attrition in the last year or so, specifically to PlanetScale by some of the most senior/database focused engineers. Edit: just noticed that you’re the CEO of PlanetScale. I don’t mean this as a slight against you or GitHub or the folks that moved. I genuinely do just think that the number of people who shifted over left GitHub with a fairly large experience and knowledge gap. I don’t think anyone is to blame, it’s just an outcome of organic evolution and changing of a technical organization.
- HALtheWise 5y agoIt seems to me like one thing that would have made these failures way more manageable would be a solid adaptive load throttling framework. Presumably, the total load only exceeded their maximum capacity by a couple percent, so a good identity-aware throttling framework should be able to maintain full functionality for ~97% of users by shedding requests early in the flow, or probably better if some automated traffic is prioritized lower and can be dropped first. If I remember correctly, Tencent had a nice paper about how they do this, but I can't find a link now.
- HALtheWise 5y agoEdit: https://www.cs.columbia.edu/~ruigu/papers/socc18-final100.pdf https://www.cs.columbia.edu/~ruigu/papers/socc18-final100.pd...
- getcrunk 5y agoTldr?
- HALtheWise 5y agoWhen a request hits the first server in the backend, it gets assigned a priority based on the service priority (i.e. wechat pay is prioritized over games) and a hash of the user's ID and the current date. When any service gets overloaded, it propagates a message to the other services that depends on it telling them to drop traffic below some threshold priority. That means the traffic gets dropped as early as possible, and the username hashing means people don't just keep refreshing until they can get through.
- getcrunk 5y agoOh wow. To both ur tldr and the method. Hmm gives me ... Many ideas
- smaudet 5y agoYikes. "All write operations were unable to function during this outage, including git operations". I know git does not need to consult any database but its own when committing. Auth is solved with tokens and url matching. I get the need for the fancy database for the all the non-git stuff, but it is very concerning this stuff is in front of the meat and potatoes. Sounds like Microsoft is making another Skype/Teams disaster...
- jhugo 5y agoAs I write this, yet again, I cannot push to or pull from GitHub repositories since about an hour now, and yet again the GitHub status page is all green. It's starting to seem like it never gets updated outside of US business hours.
- yla92 5y ago> Unfortunately, this caused a new load pattern that introduced connectivity issues on the new failed-over primary, and applications were once again unable to connect to mysql1 while we worked to reset these connections. > We were able to identify the load pattern during this incident and subsequently implemented an index to fix the main performance problem. What actually is "load pattern" in these cases?
- EricE 5y agoProbably cascading failures - that's certainly an unhealthy pattern of loads. One part of the house of cards collapses putting previously unseen pressure on another part of the house of cards - that then promptly collapses - so forth and so on. Technical debt - here's a real world example of what those words really mean!