15 ms·
Roblox has been down for days and it’s not because of Chipotle
- bastardoperator 5y agoWho wants to bet there will be a Kubernetes versus Nomad/Consul debate coming to a Roblox meeting soon? I'd like to hear what Hashicorp has to say here.
- devmunchies 5y agoback online https://twitter.com/Roblox/status/1454954676726628356 https://twitter.com/Roblox/status/1454954676726628356
- christkv 5y agoI feel for the people in the trenches on this one. It’s got to suck bad.
- tgsovlerkhgsel 5y agoIn the immediate short term, yes, obviously - stress, long hours, etc. In the immediate aftermath, probably too: Burdensome mandates from management that isn't always aware of reality to "make sure this never happens again". But if the organization is functional, in the medium term, this may also mean staffing understaffed teams, hiring SREs, etc. - which can mean less stress, no more 24/7 pager duty, better pay etc.
- christkv 5y agoThey are still a public company so I guess we will see the damage on Monday. Might be an opportunity to buy low.
- i386 5y ago> Burdensome mandates from management that isn't always aware of reality to "make sure this never happens again". When something like this threatens to end the entire party (no one pays or get paid) you god damn want to figure out why and not make it happen again. That’s not burdensome, that’s business.
- waych 5y ago> management that isn't always aware of reality This is the burdensome part.
- hourislate 5y agoA couple of possibilities... I know they said they weren't hacked but they were hacked. or They are completely inept and have no disaster recovery plan in place, etc.
- reilly3000 5y agoIt feels like some kind of catastrophic data loss. I can’t imagine that app servers or network infrastructure could have been the root cause, especially because they are running on AWS and there hasn’t been any reports of outages or other customers impacted. Restoring an old backup and rebuilding data from logs seems like the only thing that could take so long. That or an entirely dysfunctional IT org that can’t get out of its own way in a crisis. Best to them.
- bink 5y agoThis article from 2019 suggests they use a mix of cloud and a dedicated data center. https://portworx.com/blog/architects-corner-roblox-runs-platform-70-million-gamers-hashicorp-nomad/ https://portworx.com/blog/architects-corner-roblox-runs-plat...
- jitl 5y agoI wonder if they have a major Postgres database that hit transaction ID wraparound? Postgres uses int32 for transaction IDs, and IDs are only reclaimed by a vacuum maintenance process which can fall behind if the DB is under heavy write load. Other companies have been bitten by this before, eg Sentry in 2015 (https://blog.sentry.io/2015/07/23/transaction-id-wraparound-in-postgres https://blog.sentry.io/2015/07/23/transaction-id-wraparound-...). Depending on the size of the database, you could be down several days waiting for Postgres to clean things up. Even though it’s a well documented issue with Postgres and you have an experienced team keeping an eye on it, a new write pattern could accelerate things into the danger zone quite quickly. At Notion we had a scary close call with this about a year ago that lead to us splitting a production DB over the weekend to avoid hard downtime. Whatever the issue is, I’m wishing the engineers working on it all the best.
- vbg 5y agoThe latest version of Postgres addresses this issue in various ways, although it’s not entirely solved, it should be significantly mitigated.
- xyst 5y agoI thought it was a certificate issue? I am looking at robox.com and the issuing CA is GoDaddy...
- jitl 5y agoI’m just speculating. I didn’t do any in depth research - none of the articles or tweets by Roblox I saw offered anything more than “an internal issue”.
- isuckatcoding 5y agoYeah but wouldn’t the impact be more widespread then?
- mateo411 5y agoCertificate issues don't take that long to resolve.
- xyst 5y agoI wonder how the market is going to react when it opens tomorrow. I am thinking a quick scalp with weekly $RBLX puts, then when it recovers double up on cheap long call options.
- pkulak 5y agoI like slots myself.
- cinntaile 5y agoIt went up when it was known that the site was down, so it can be a bit hard to predict.
- downrightmike 5y agoprobably a bunch of bots that bought on any news is good news.
- xyst 5y agoDown 3% since opening. Roblox still has not recovered. Might be a free fall if they have not recovered by mid week.
- ctvo 5y agoPretty remarkable to think the market prices things like this, or that public information everyone is privy to gives you an edge.
- Infinitesimus 5y agoThere are trading bots that react to news articles so its not unreasonable to see significant swings especially in options activity this week. They also have earnings coming up.
- amelius 5y agoIs there a possibility this is caused by ransomware?
- PeterisP 5y agoFrom TFA - “We believe we have identified an underlying internal cause of the outage with no evidence of an external intrusion,” says a Roblox spokesperson.
- amelius 5y agoThanks, but can we trust any company in this situation to be transparent, given that such an attack would also mean that user data could have been leaked?
- Groxx 5y agoIt's possible, but that'd be a theory entirely founded in paranoia, as it both needs no evidence (we have no sign it's an attack) and accepts no counter-evidence (they have said it is not an attack). So while it's always a possibility, it's also kinda pointless to wonder about until there's supporting evidence.
- deleted 5y ago[deleted]
- PeterisP 5y agoReports of IT incidents from public companies tend to be obscure, vague, missing or technically true but misleading but they almost universally stop short of outright lying (mainly because for public company officers lying to shareholders i.e. the general public has harsher personal risks&consequences than the company losing lots of money or even shutting down; being incompetent is completely legal but lying to your shareholders about material aspects of company finances is a crime and also makes them personally financially liable), so if they say that they have seen no evidence of an external intrusion, then I would presume that it's definitely not ransomware, where signs of external intrusion - namely, the ransom demand - would be obvious and undeniable. Perhaps (though IMHO not that likely) it may be some other kind of attack e.g. one intended to secretly steal customer data, which does not give signs of external intrusion if the company doesn't look for them much (and it might have motivation to not look very hard), but if it was ransomware, I'm quite sure they would not say what they said.
- tschellenbach 5y agoThat's a very long outage, I wonder how this happened. - Perhaps internal systems they've developed, and the people who created them left. So it's not just fix the thing, but first understand what the thing is doing and then fix it. - Data recovery can take forever if you run into edge cases with your databases Anyone found any articles about their architecture?
- ekovarski 5y agoHashicorp has a case study with some details of what they use, https://www.hashicorp.com/case-studies/roblox https://www.hashicorp.com/case-studies/roblox
- ytjohn 5y ago> We have people who are first-time system administrators deploying applications. building containers, maintaining Nomad. There is a guy on our team who worked in IT help desk for eight years - just today he upgraded an entire cluster himself.”
- sidlls 5y agoI know everyone has to start somewhere, but this just sounds like a sure way to have a platform blow up impressively.
- breadzeppelin__ 5y agooof (from the link). Debugging nightmare to have a fairly inexperienced team trying to diagnose stuff in these incredibly complex systems. > 4 SREs managing Nomad, Consul, and Vault for 11,000+ nodes across 22 clusters, serving 420+ internal developers > "We have people who are first-time system administrators deploying applications, building containers, maintaining Nomad. There is a guy on our team who worked in the IT help desk for eight years — just today he upgraded an entire cluster himself."
- kawsper 5y agoI bet it is some sort of Vault/Consul shenanigans that's going on.
- roamerz 5y agoInternal cause does not necessarily mean a technical mishap. Read rogue sysadmin or other employee initiated event.
- TekMol 5y agoHas any billion dollar company ever been down for 2 days? Or is this the highscore?
- jlokier 5y agoTesco online grocery shopping in the UK was down for about 2 days recently, and Tesco as a whole has a market cap of $29.1B.
- TekMol 5y agoDid they publish what happened?
- omnicognate 5y agoThey claimed it was an attack (an "attempt to interfere with our systems") but haven't given any details.
- mysterydip 5y agoI assume you could still go in the store and get groceries though, right? Not familiar if they're online-only right now.
- arichard123 5y agoYes, Tesco was basically open the whole time.
- cinntaile 5y agoThe supermarket chain Coop in Sweden was down longer than 2 days. They closed their shops during the ransomware attack. https://www.bbc.com/news/technology-57707530 https://www.bbc.com/news/technology-57707530
- deleted 5y ago[deleted]
- sithlord 5y ago
- xakahnx 5y agoAdding to the speculation here, I'm willing to bet some component of their issue is not entirely technical. Regardless of the underlying cause (PKI was mentioned), for downtime to last this long it almost definitely means some persistent data was lost or corrupted. Of course they can recover from a backup (I'm confident they have clean backups) but what does that mean for the business? "We irrecoverably lost 12 hours of data" could have severe implications, for example legal or compliance risks.
- pm90 5y agoWhy are you confident they have clean backups? It’s been my experience that backup infrastructure is usually not given much thought, engineers infrequently test that recovery from backups work as expected. Not saying that’s what it is but not sure if it can be ruled out.
- wly_cdgr 5y ago"Roblox is very popular, especially with kids — more than 50 percent of Roblox players are under the age of 13. More than 40 million people play it daily" ....from misleading logical non sequitur to parroting Roblox marketing numbers in under 30 words, nice. Verge is such trash
- breakingcups 5y agoAs longs as we're speculating.. One of the few things I can think of that can't reasonably be sped up is data integrity recovery. Say some data got in an inconsistent state and now they have to manually restore a whole bunch of financial transactions or something before opening the game up again, because otherwise customers would get very mad at missing stuff they've paid for, traded, etc. If they were to resume the game before restoring these issues, they would only exacerbate with state moving even further from where it was originally.
- g123g 5y agoWouldn't it be cheaper to directly compensate such customers rather than keeping the whole website down for 3 days?
- bryanrasmussen 5y agomaybe, but wouldn't you have to recuperate the data to be able to figure out what customers you owed and how much?
- charcircuit 5y agoJust ask people to send in support tickets
- fieldcny 5y agoWell you can get that from your transaction processor, doubt they do that themselves
- swatkat 5y agohttps://twitter.com/Bloxy_News/status/1454861081021587456 https://twitter.com/Bloxy_News/status/1454861081021587456 "STATUS UPDATE: Roblox is incrementally opening the website to groups of users and will continue to open up to more over the course of the day..."
- abricot 5y agoThey write that, but according to various game Discords it's absolutely not true. No one is allowed to log in.
- huhtenberg 5y agoMy friend's kid got in half an hour ago.
- fragmede 5y agoabsolutely? how many people are in these discords, and where do their players fall in the database sharding key? because thundering herd is definitely a problem when letting people back on to a system, and oh also we have no idea what the underlying issue actually is because Roblox has been basically silent this entire time. (Unsubstantiated Internet rumor about it being the secrets store also doesn't count.)
- yibg 5y agoYou have confirmation 0 Roblox users are able to log in?
- meheleventyone 5y agoConfirmation on the official account: https://twitter.com/roblox/status/1454900890180063238 https://twitter.com/roblox/status/1454900890180063238
- jeffal 5y agoLike when Camelcamelcamel was down for a week in 2019 https://news.ycombinator.com/item?id=19038198 https://news.ycombinator.com/item?id=19038198
- romanhn 5y agoThe lack of communication for an outage this big is absolutely shameful. I put this on the leadership, not any of the engineers working round the clock. Having been in the middle of a critical service outage that lasted over 24 hours I totally get the craziness of the situation, but Roblox seriously needs to revisit their incident management and customer update process. Even though kids are the main consumers of the app, the near total silence speaks volumes about their business's lack of preparedness for disaster scenarios. If nothing else, I hope they'll see this as an opportunity to learn and do better next time.
- swiley 5y agoMeh. This is part of playing a game where all the multiplayer goes through one service.
- golemiprague 5y agoWhat exactly need to be communicated? it is just gossip. They will fix it when they fix it, they probably don't know when exactly and the kids who play it don't care about the details.
- CheezeIt 5y agoGood grief. It's a game for kids. Nobody needs updates. They could even take the weekend off.
- charcircuit 5y ago>It's a game for kids There are companies that rely on Roblox for hosting their games so they can make money. This is the equivalent of cloud hosting going down.
- WhisperingShiba 5y agoI think you have a Kernel of truth in your statement, but perhaps you are overlooking how silly it is for someone's well-being to rely on a couple weeks of video game micro transaction. My heart goes out to the Devs in this category, but hopefully this is further impetus not to stake too much on tech services.
- EastOfTruth 5y agoApparently, according to roblox.com, some player are able to play: "We are incrementally opening to groups of players and will continue rolling out." https://i.imgur.com/KgDxNsg.png https://i.imgur.com/KgDxNsg.png
- abricot 5y agoThey write that, but according to various game Discords it's absolutely not true. No one is allowed to log in.
- Matheus28 5y agoFun fact: I operate many web mmo games (or “io games” how people like to call them), and traffic is up around 20-100% since the Roblox outage started.
- schnebbau 5y agoThat's cool, I love io games. Which ones?
- Matheus28 5y agoMost well known one is probably agar.io. I made a couple of other ones like diep.io, and most recently digdig.io
- truetraveller 5y agoThis guy's too humble. He's Matheus Valadares, and he's the creator of Agar.io [1]. This is the game that pretty much started the ".io game" genre [2]. [1]: https://en.wikipedia.org/wiki/Agar.io https://en.wikipedia.org/wiki/Agar.io [2]: "Around 2015 a multiplayer game, Agar.io, spawned many other games with a similar playstyle and .io domain". See https://en.wikipedia.org/wiki/.io https://en.wikipedia.org/wiki/.io
- bombcar 5y agoSimilar to John Carmack coming into a thread and saying he’s worked on a game or two.
- jimmydorry 5y agoHi M28!
- mromanuk 5y agoLove agar.io and HN for this
- tetron 5y agoBeen checking the #roblox hashtag on Twitter and the two main themes are addicts going through withdrawal and devs saying how they wouldn't have their llama appreciation fan site be down this long let alone your core business.
- intunderflow 5y agoRoblox made their developer tooling require you to be always online and signed in, even though it doesn't actually need this to function. This means that all development workflows have been bricked during this outage too. https://twitter.com/RBXStatus/status/1454815143607607300/photo/1 https://twitter.com/RBXStatus/status/1454815143607607300/pho...
- lotophage 5y agoDevelopment has been bloxxed
- winrid 5y agoI see one of their DBs is Mongo. I wonder if they ran into some sharding related nightmare.
- aderdale 5y agoSeems to work at least partly now. I just jumped into one of the more popular games and a bunch of people playing.
- chipotle_coyote 5y agoWell, I'm glad I'm finally off the hook.
- mukundmr 5y agoRobolox is back online. https://twitter.com/roblox/status/1454994222583455744?s=21 https://twitter.com/roblox/status/1454994222583455744?s=21
- QuercusMax 5y agoI've been getting recruiting emails from Roblox recently.... Maybe they really do need my help.
- T3RMINATED 5y agoI heard the CEO is resigning and took the servers down with him.