5 ms·
What is going on over there? Third day in a row is... kind of impressive.
by dbingham 3y ago
What is going on over there? Third day in a row is... kind of impressive.
- frde 3y agoMy money is on some significant backend architecture migration gone wrong without a viable way to roll back the time machine :)
- lallysingh 3y agoAzure?
- capableweb 3y agoFeels like an organization as big as Microsoft must surely have some sort contingency plan in place before doing such a large migration. Right?
- isbvhodnvemrwvn 3y agoSales handing out Office365 discounts and trying to convince people that AWS and GCP is going to steal their data, judging by companies I worked for that used Azure.
- hardware2win 3y agoWasnt it true? Thas Amazon abused their AWS position and stole their competitors data, so thats why Germany's retail businesses are building their own Clouds
- jacooper 3y agoAny links? Interested on reading more about this
- belmont_sup 3y agoHere’s one https://www.wsj.com/articles/amazon-scooped-up-data-from-its-own-sellers-to-launch-competing-products-11587650015 https://www.wsj.com/articles/amazon-scooped-up-data-from-its... The gist is that yes it’s true. They’d come out with their own Amazon Basics branded stuff and push it to the top.
- jacooper 3y agoBut that's amazon, not AWS.
- xeromal 3y agoI worked on the volume licensing part of Microsoft years ago and deployments were stressful. They'd start at friday late like 8pm or so and go until 8am in the morning. Everyone was on a long call the entire time. I hated it.
- whynotmaybe 3y agoAt that point, I wondering if that's not me just because I updated some libraries on my local build agent.
- aranw 3y agoLots of copilot generated code failing
- practice9 3y agoNot only Copilot, seems like some Microsoft services like Bing AI and Bing Image Creator have some issues today as well with 4xx / 5xx, and incorrect region authorization (had to switch the account region from a European country to US to make it work again on mobile)
- tesin 3y agoI think you may have missed the joke - they were implying Github was using Copilot internally, causing the outages, due to poor output. Not that Copilot itself was unavailable (although that may be true, also)
- candiddevmike 3y agoToday seems worse than yesterday. I'm getting wildly inconsistent results when viewing repositories after a push. Hard to tell if my push actually went through, and it's not triggering actions.
- robofanatic 3y agoexact same issue
- SideburnsOfDoom 3y agoIt comes in around 09:30 on US east coast. I suspect that it's related to high load.
- iepathos 3y agoThey blamed the march and april outages on some database query that was changed due to an infrastructure change they rolled out. I'm guessing their infrastructure change caused some other race condition issue that they are only seeing after major production failure due to not load testing enough in their staging environment https://github.blog/2023-05-03-github-availability-report-april-2023/ https://github.blog/2023-05-03-github-availability-report-ap...
- edgyquant 3y agoAs much as I’ve been frustrated by these outrages, we’ve all been there
- qmacro 3y agoNow that is a good typo
- robofanatic 3y agocould be on purpose
- pera 3y agoMaybe they fired too many people this time? https://news.ycombinator.com/item?id=35334705 https://news.ycombinator.com/item?id=35334705
- tonyhb 3y agoFrom an SRE, one of their DB clusters failed. They use Vitess which is great, but it can be prone to hotspots and doesn't auto-shard. Heavy usage (esp. from large customers, rogue jobs) can take down the cluster. When it goes down, it's a PITA to resolve.
- samlambert 3y agoThis literally isn't true and looks awfully like the talking points of one of our competitors.
- tonyhb 3y agoAh, unbalanced shards via wrong sharding keys was an issue at one point, IIRC. I remember talking with an SRE there when something bad happened at GitHub last year, and I know that this time the current DB cluster failed. To be clear, I _was_ mapping previous incidents with this year's incident — no competitor or hard feelings involved. I really like Vitess, fwiw. And the only thing I really love is FoundationDB :)
- samlambert 3y agoThat wasn't clear. Side note: "Autosharding" is largely a myth that unproven databases are touting. Sharding is complex and requires planning and control. Databases that start shuffling data round without oversight produce nasty surprises. Trying to be too magic is normally always a mistake with databases.
- tonyhb 3y agoYeah, fair, totally get it. Wasn't aiming to spread FUD, and I know that FDB is a little hard to compare against... it is pretty magic with how it routes and shards :D (https://forums.foundationdb.org/t/keyspace-partitions-performance/168/2 https://forums.foundationdb.org/t/keyspace-partitions-perfor...)
- 3y ago
- zamalek 3y agoJust another day doing DevOps for a Ruby on Rails product.