18 ms·
An Update on Our Outage
- reilly3000 5y agoDid they run out of internal ipv4 address space somehow? I’ll be curious for the post-mortem, this is definitely the longest outage of a global web service I can remember in… forever? This incident puts them under 2 9’s.
- kevan 5y agoMy guess is something related to service mesh/discovery. If that goes down, deployment infra also depends on it, and you don't have a solid black start protocol then you're in for a very bad week.
- dannyw 5y agoNot a lot of details here but I'd suspect it's yet another example of autoscaling policies gone wrong.
- reilly3000 5y agoWhat does that tend to look like? Do you have any examples?
- atomicnumber3 5y agoIn my experience, it tends to be that you find out one of your components has some tipping point as you scale out horizontally where everything goes to hell. Random example: you scale a service horizontally and suddenly postgres disk space usage goes from "this line looks horizontal" to "we just used 2 years' worth of disk use growth in 40 seconds". And the postmortem is basically like "yep can't really blame anyone for not knowing this weird ass thing or unfound bug". Software, man.
- asn1parse 5y agoyep.
- deanCommie 5y agoThis totally happens, and is a thing, but tracing a "unexpected change in metrics on our DB server to "increase in metric showing call volume to DB server" to "increase in metric showing increase in fleet size of app servers that call the DB server" takes minutes to identify, not 24 hours. And the remediation procedure is quite simple too - scale down. If the fleet can't handle it, increase your throttling. If even with increased throttling you can't handle the load and everybody's timing out (but you still can't scale out), you start spinning up new load balancers, and applying weights in DNS where at least some percentage of your callers are getting through while others see full failure. The key point is in anything related to such an outage, even if it takes a long time to recover, your customers shouldn't see a full outage. So it was either an entirely different class of problems or Roblex has a huge ops skills gap. I'm betting the former. edit: never mind, maybe it's the latter after all: https://twitter.com/NIDeveloper/status/1454773313792880640 https://twitter.com/NIDeveloper/status/1454773313792880640
- suresk 5y agoI think there are two big things that can be really hard with scalability - 1) There are a lot of dimensions I think it is fairly common to focus on a throughput-related dimension like requests per second and test that very thoroughly, while not paying enough attention to other aspects. We have services where, if we were told requests per second were going to go up 100x tomorrow, I wouldn't be too concerned, but if we were told request size were going up 2x, I'd be freaking out. Even when you think you've identified all of the dimensions, there are usually ones you missed and/or they interact in weird and unexpected ways. 2) Performance can appear to be linear when it really isn't So many outages I've seen have resulted from everything being fine until some threshold is hit, at which point it doesn't start to slowly degrade, but instead immediately explodes. Often times due to feedback loops (GC activity in garbage-collected languages often can behave this way) or because some cache overflowed (data that is queried at a high rate no longer fitting in memory on a DBMS comes to mind).
- mr-karan 5y agoHuh, strangely the link doesn't open for me. Seems like an infinite redirect loop. $ curl -i -L https://blog.roblox.com/2021/10/update-on-our-outage/ HTTP/1.1 301 Moved Permanently Server: nginx Content-Type: text/html; charset=UTF-8 Content-Length: 0 X-Redirect-By: WordPress Location: https://blog.roblox.com/2021/10/update-on-our-outage/ X-Powered-By: WP Engine X-Cacheable: bot X-Cache-Group: bot Strict-Transport-Security: max-age=63072000; includeSubDomains; preload Cache-Control: must-revalidate, max-age=7108 Expires: Mon, 01 Nov 2021 05:39:59 GMT Date: Mon, 01 Nov 2021 03:41:31 GMT Connection: keep-alive
- nitinreddy88 5y agoPretty much same here
- dreadlordbone 5y agoWorks for me, possibly a cached 301?
- ramchip 5y agoSame, I can open https://blog.roblox.com/2021/10/ https://blog.roblox.com/2021/10/ and other linked blog posts, but not this specific one.
- andreareina 5y agoSame with Firefox: > The page isn’t redirecting properly > An error occurred during a connection to blog.roblox.com.
- vmception 5y agoVPN from somewhere else and use a new browser Most likely your edge node and your client side have cached something not ready yet
- shalmanese 5y agoLooking forward to and incident analysis of why their incident analysis post wasn't posted correctly.
- 5y ago
- deleted 5y ago[deleted]
- _nickwhite 5y agoMy 3 youngest kids, aged 9, 9, and 8 told me Roblix were having an extended outage, world wide. Good job to the team getting everything back up. Been there, done that. #hugops
- kingofpandora 5y ago> My 3 youngest kids, aged 9, 9, and 8 ... That must have been quite a year in the White household!
- zomglings 5y agoQuite a decade more like.
- richardwhiuk 5y agoTwins, and then probably ~18 months later a younger child.
- _nickwhite 5y agoTwins + adoption actually. The current picture is 8, 9, 9, 16, 16. Figure that one out. :)
- mmmBacon 5y agoMy 8 y.o. daughter ran through the house with an excitement typically I only see on Christmas morning exclaiming “Roblox is back!”
- Railworks2 5y agoFeels like that for me, suddenly I can actually reopen Roblox Studio and do some work.
- aahortwwy 5y ago> We will publish a post-mortem with more details once we’ve completed our analysis, along with the actions we’ll be taking to avoid such issues in the future. Excellent. That should be a very interesting read. > In addition, we will implement a policy to make our creator community economically whole as a result of this outage. Good to hear.
- joe-collins 5y agoThat latter statement wins a lot of respect in my book. I'm not sure I could name any other platform or game company that I would expect to see go so far.
- dastbe 5y agoi’m very interested to hear more! it’s hard to speculate without more details, but based off the bits here i’m suspecting that their load balancing/service discovery system hit an issue that resulted in a death spiral. often times a lot of these systems are built with subtle nonlinear scaling bottlenecks that also don’t allow the system to fail into a stable state.
- leohonexus 5y agoThere's a redirect loop between https://blog.roblox.com/2021/10/update-recent-service-outage/ https://blog.roblox.com/2021/10/update-recent-service-outage... and https://blog.roblox.com/2021/10/update-on-our-outage/ https://blog.roblox.com/2021/10/update-on-our-outage/ I'm unable to open the link and seems Wayback Machine cannot archive it either.
- danso 5y agoIt’s great to hear that it wasn’t the result of something malicious (e.g. a hack). But this has to be one of the longest outages by a company this big ($50B market cap), at least in the past decade it seems?
- vmception 5y agoWhat I love most then is that its the largest and it doesn't matter! In this market they will become an even larger company because competency has no correlation to how much money subscribers/advertisers will pay, and none of that has any correlation to what investors will pay! Anybody optimizing for competence is exchanging time for food and shelter and just gets to be reminded of the much harder game they are playing that produces much less optimal results of money.
- Thorrez 5y agoIf it's down for a couple days every month kids will start playing other games and get hooked on them. Roblox would lose a fair amount of users. Losing users means losing investor money.
- cottager2 5y agoMaybe your definition of competence is wrong? A lot of people can build systems with some number of 9s of availability, but how many people can build a 50B company?
- djbusby 5y agoI can do the 9s. So, that's common I'd wager. $50B is hella hard tho. It's hard work to just get to $10M (w/o outside money) and even with outside money 100M is still very hard.
- akersten 5y agoOne is luck and the other is engineering. From the looks of it, this is a high luck, low competency endeavor.
- rackjack 5y agoThis was probably some junior dev's worst first day
- Gigachad 5y agoThe junior devs were probably chucked in a corner with some colored wooden blocks while this was being resolved.
- jpgvm 5y agoGiven the scope and the description it's highly unlikely to be caused (or investigated by) juniors. It definitely sounds like the service-discovery/configuration store (which is Consul from what I understand) or their orchestration layer (Nomad) shat itself. So most likely their most senior SREs/infrastructure folk were the ones sweating bullets.
- celim307 5y agoBad timing too, I bet holiday events like Halloween are a big deal in the roblox culture
- Railworks2 5y agoRoblox Dev here (Make the experiences, not employee), Halloween the second most important weekend for us developers in terms of income. The Friday and Saturday are critical days in getting revenue from any Halloween specific events we may do. Such as a Halloween update or discount. I'm happy to see Roblox is refunding adverts, especially after seeing how much support this has. https://devforum.roblox.com/t/reimbursement-on-sponsors-and-advertisements-when-the-website-goes-down/117605/30 https://devforum.roblox.com/t/reimbursement-on-sponsors-and-...
- perryizgr8 5y agoCan anyone cache this page and link that? Or paste the text here in a comment? This link shows a redirect loop for me.
- registeredcorn 5y agohttps://news.ycombinator.com/item?id=29062912 https://news.ycombinator.com/item?id=29062912
- mythz 5y agoLooks like everyone got the cause of this outage wrong on previous speculative posts.
- htrp 5y agoAs most of the Roblox community is aware, we recently experienced an extended outage across our platform. We are sorry for the length of time it took us to restore service. A key value at Roblox is “Respect the Community,” and in this case, we apologize for the inconvenience to our community. On Thursday afternoon, October 28th, users began having trouble connecting with our platform. This immediately became our highest priority. Teams began working around the clock to identify the source of the problem and get things back to normal. This was an especially difficult outage in that it involved a combination of several factors. A core system in our infrastructure became overwhelmed, prompted by a subtle bug in our backend service communications while under heavy load. This was not due to any peak in external traffic or any particular experience. Rather the failure was caused by the growth in the number of servers in our datacenters. The result was that most services at Roblox were unable to effectively communicate and deploy. Due to the difficulty in diagnosing the actual bug, recovery took longer than any of us would have liked. Upon successfully identifying this root cause, we were able to resolve the issue through performance tuning, re-configuration, and scaling back of some load. We were able to fully restore service as of this afternoon. We will publish a post-mortem with more details once we’ve completed our analysis, along with the actions we’ll be taking to avoid such issues in the future. In addition, we will implement a policy to make our creator community economically whole as a result of this outage. There are more details on this to come. As part of our “Respect the Community” value, we will continue to be transparent in our post-mortem. To the best of our knowledge, there has been no loss of player persistence data, and your Roblox experience should now be fully back to normal. You can always contact our support team if you experience any hiccups using Roblox now or in the future. We are grateful for the patience and support of our players, developers, and partners during this time. Verbatim Text post for anyone having redirect issues. TLDR; We have no clue what happened.
- hn_throwaway_99 5y agoThank you for posting the text of the blog post. However, I don't understand your TLDR. They clearly were able to identify the root cause, and that is what allowed them to restore service.
- 5y ago
- lwansbrough 5y agoSomeone on here posted the other day that someone they knew at Roblox said it was their secret store that became overloaded. Presumably a Hashicorp Vault type service (or something similar.) This update appears to support that claim. That's definitely a problem you only get at scale.
- judge2020 5y agoThere's a hashicorp case study on Roblox, so likely Vault specifically: https://www.hashicorp.com/case-studies/roblox https://www.hashicorp.com/case-studies/roblox
- inside_edition 5y agoNot far off. Having chatted with friends from the company, consul failed due to a issue with streaming which was introduced in 1.9 (https://www.hashicorp.com/blog/announcing-hashicorp-consul-1-9 https://www.hashicorp.com/blog/announcing-hashicorp-consul-1...) which caused it to have massively reduced tps in certain circumstances. No functional consul meant no vault, no consul/vault meant no nomad, no nomad meant no application servers. Furthermore, the services were down for long enough that the asset caches went fully cold, so spinning back up to 100% capacity would put far more load than usual on the servers meaning recovery needed to be a very slow incremental rollout to warm them. Messy situation all around, but if you want the real root cause pay attention to the consul change log over the next few weeks I’d say.
- randomperson_24 5y agoThe link seems to be in an infinite redirect loop. https://webcache.googleusercontent.com/search?q=cache:d4DE5BioZlEJ:https://blog.roblox.com/2021/10/update-recent-service-outage/+&cd=1&hl=en&ct=clnk&gl=in https://webcache.googleusercontent.com/search?q=cache:d4DE5B...
- slg 5y agoMy heart goes out to the people who likely had to work crazy hours to fix this, but it really is wild that it was down for so long. What was the last service of this size that was down for 4 days? That is a failure in architecture that goes way beyond whatever the specific cause was here. That post-mortem is going to be a doozy.
- raxxorrax 5y agoWe have hospitals being knocked out for weeks after ransomware attacks. If we complain about roblox I think we should reflect on our priorities.
- CrazyStat 5y agoWhy not complain about both?
- raxxorrax 5y agoYou can, wasn't the point I tried to make. I tried to point out the arbitrary expectations here. Depending on the fault such downtime can last a few hours to a few weeks. I don't think you can make conclusions on architecture in this case. Perhaps it would be possible to roll back everything in just a few minutes but some users might lose a lot of work. If such special cases are to be resolved, it may take a lot of time. You just cannot really infer quality of architecture by downtime alone.
- kaesar14 5y agoThe last outage of this kind I can even remember is the summer PSN was down for like, a month?
- slg 5y agoAlthough that PSN downtime probably fails into a different category as it was a response to being hacked. Sony intentionally took the service down and kept it down to investigate and fix their security holes. Basically the equivalent of temporarily closing a brick and mortar store after a robbery compared to being involuntarily closed because no one could find the right set of keys to unlock the store's front door.
- rubyist5eva 5y agoThis "game" is pure cancer for children, it should never come back.
- knownjorbist 5y agoGet a grip.
- rubyist5eva 5y agoDamn, you convinced me.
- geuis 5y agoPutting aside the content of the page, can we talk about the design of the page? It has this HUGE hero image that pushes the content down below the fold on my 4k monitor.
- sam0x17 5y ago> A key value at Roblox is “Respect the Community,” Yet they still take a 30% cut on all transactions that go to content creators / developers
- busymom0 5y agoThat high percentage doesn’t mean they do or do not respect the community. Saying this as someone who doesn’t know much about Roblox but does give 15-30% to Apple and Google. Google definitely doesn’t respect their developers. Apple kind of does.
- sam0x17 5y agoApple and Google also disrespect their developer communities, probably more than any companies out there. Anything over 5% is ridiculous and was seen as such originally before Google and Apple made this the norm.
- busymom0 5y agoI don't disagree with you. I was just stating that Google is absolutely horrible whereas Apple at least let's us developers talk to a human being.
- sam0x17 5y agoApple is the worst of them all. They are much more restrictive and arbitrary with their app store policies (see previous HN articles over the last 5 years), they have done much more to lock down their devices to prevent circumvention of things like banning emulators, and they don't license the apple device model to other vendors, so they are the true definition of a walled garden. Google has plenty of problems too, but at least random device companies can make android devices, and third party app stores are (sort of) allowed to exist.
- busymom0 5y agoYou aren't wrong. Though I was mostly talking about customer service part and development relations in terms of that. Google regularly bans developers for absurd reasons or no reasons at all, doesn't let them talk to any humans until the developer causes enough twitter storm and gets some tech sites to write about it to get some attention from Google. Apple at least lets the dev talk to human beings.
- weird-eye-issue 5y agoFirst a redirect loop Now, a header image that takes up my entire 15" laptop screen before I can even see the title of the blog post. Impressive https://imgur.com/256LkYb https://imgur.com/256LkYb
- zz865 5y ago> performance tuning, re-configuration, and scaling back of some load. That seems to be a great thing if you can sit at the edge and scale the number of connections coming in. I just wish our system could do that.
- jpgvm 5y agoMy guess is Consul shit itself. It works until it doesn't. For now I prefer Zookeeper because at-least I have experienced most of it's failure modes. i.e they are probably all prone to blowing up but I have lost my eyebrows enough times in ZK explosions that atleast I know what I'm up against. Consul isn't widely used enough for me to have the same confidence for now, same goes for Vault and Nomad tbh. I really like the design of Vault and it's dynamic secrets system but I am probably just going to implement something similar on k8s secrets so that I don't have to carry around something that might spontaneously combust on me.
- faizshah 5y agoJust curious, what are the common ZK failure modes?
- jpgvm 5y agoIn high scale situations, somewhat common if you aren't ontop of metrics etc. In most other cases it's just people not turning on auto-purge and having disks fill up with snapshots. So it varies depending on your teams experience with ZK and/or if you are using some sort of community managed deployment primitive that has sensible defaults baked in.
- strictfp 5y agoMy stance is: why even have any of those? Are you running in a local datacenter?
- jpgvm 5y agoGenerally you need them even when running in a "Cloud" of some description. Namely because they provide distributed locking primitives at a speed that can't be matched by any other mechanism. If you only need service discovery you can probably get away with whatever you platform provides (EC2 API, k8s API, hell DNS works, etc), similarly if you only need slow master election there are alternatives there too (DynamoDB w/conditional writes, k8s configmap). However for what these systems do best, i.e very high performance distributed locking and consistent metadata with fast reads w/watches and relative fast writes there is no replacement. You either need one of ZK/etcd/consul/something else based on raft/paxos/zab. EDIT: I realize now you might be talking about Nomad/Vault also. So in this case I think they are running their own DCs on bare metal and they opted for Nomad over k8s. Vault is somewhat special because it has capabilities no other secrets system does out of the box, specifically it has integrations to create short-lived credentials on the fly for clients - so called dynamic secrets have many advantages and is why services like AWS STS are so popular (the magic sauce behind AssumeRole).
- tilolebo 5y agoThis reminds me of the Target TAP outage some years ago: https://danveloper.medium.com/on-infrastructure-at-scale-a-cascading-failure-of-distributed-systems-7cff2a3cd2df https://danveloper.medium.com/on-infrastructure-at-scale-a-c...
- softwaredoug 5y ago> we will implement a policy to make our creator community economically whole as a result of this outage. Mad props for this. Who else would do such a thing for an outage? I wonder how they will do this of course. The devil will be in the details :)
- xbar 5y agoI am disappointed by the lack of technical transparency in this update. If Roblox knew enough about the problem to address it, they could have included enough technical words to describe it. Which piece was overwhelmed and in which piece of backend did the subtle bug exist? A deeper post-mortem will be nice. An actual description of the problem seems like SRE table-stakes for outages.
- tofuahdude 5y agoThey do say in the post that they're going to explain more, later.