13 ms·

Roblox has been down for over 24 hours

238 points by intunderflow 5y ago

sjg007 5y ago
Worse the the FB outage for parents… help!
johnmarcus 5y ago
I interviewed with their ops team there and turned it down. Just had a bad smell. I think they hire 'SRE' that can write good leet code. The problem with those folks for ops is that they over engineer complex solutions, and then completely forget the simple but very important stuff. I wish them luck in getting it back together.
- ripper1138 5y ago
  Leet code style questions for an SRE is so dumb. SRE is all about solving unexpected, unique problems. Opposite of leet code!
intunderflow 5y ago
There's probably a lesson in here about managing incidents, very few updates or acknowledgments there's even a major problem. Will be interesting to find out what caused an incident big enough to kill their entire platform for so long (if they release details or they come out through the grapevine)
- stingraycharles 5y ago
  Seems to me that a routing / networking issue would have been resolved by now, and application bug would have been rolled back. If I were to speculate, I would say that it must have something to do with databases / storage. Something must have gone wrong and broke some database, and it’s difficult to restore.
  - intunderflow 5y ago
    They had an incident in the past where they returned nil for all get operations to their database, that caused some chaos at the time so it's not unprecedented: https://devforum.roblox.com/t/update-datastores-incident-data-consistency-checker-api/832679 https://devforum.roblox.com/t/update-datastores-incident-dat...
  - polack 5y ago
    I agree that DB is the prime suspect here. They said the problem was identified over 24h ago, cant imagine it being something other than a data issue that takes that long to resolve.
- deathanatos 5y ago
  The status page acknowledges there is an incident, that it's pretty bad — everything is marked as down, and it's in red! What more could you want? (/s) Compare to Azure: most incidents never see the status page. Hell, getting support to acknowledge that an incident exists even once your internal investigation has reached certainty on "oh, yeah, it was them" is hard. There was an AAD outage earlier this year (?; IDK — I've lost track of the passage of time in the pandemic…) and the status page was down, and even once you managed to get the status page (IIRC you could hit the IP if you knew it magically which the Twitterverse did) … most services were still green, even if completely offline as far as one could tell by issuing queries to the service… And I'm comparing a kid's game with a "major" cloud PaaS… I'm definitely suffering from Stockholm syndrome.
  - aliswe 5y ago
    a kids game yes, but worth 50B
  - KronisLV 5y ago
    > Compare to Azure: most incidents never see the status page. That sounds like a conscious decision on their part. Everyone always talks about disclosure being the best policy, but at the same time there's plenty who believe that not informing anyone about an outage or even a breach is the correct thing to do, since then they'll probably get into less trouble themselves, or at least will create the illusion of not having as many outages as "those other guys". After all, informing everyone about an outage that will noticeably affect only some probably has a larger impact on the company's reputation than having it be dragged through mud in smaller communities for its dishonesty. Then again, with many of the larger services, it's not like you have much of a choice of using or not using it - you just get a corporate policy passed down upon you and that's that. Thus, sweeping problems under the rug and pretending that they don't exist is a dishonest, yet valid way of handling outages, breaches and so on. Personally, i'd avoid any service that does that, though it's not like that's being done just because of incompetence.
    - deathanatos 5y ago
      > That sounds like a conscious decision on their part. It is; I've been told by their support that they don't want to cause alarm. I think it's a bad way to run a PaaS, though. If I'm looking at your status page, it is because I suspect an outage and am trying to confirm it. Very willing to give some leeway to fix problems (an SLA — and Azure could do better here too — exists to establish what that allowable leeway is) I just need to know "is it me, or not?" and it's nicer to just get the answer when it's not me. As it is, I have to jump through a support hoop to get at "I think you are having an outage" and even then, it's typically multiple cycles before support seems to query engineering (and — that's another problem: support doesn't just know that there is an outstanding issue…) and gets to the bottom of it. It needs to be easy for a customer, experiencing an issue with a service, to drive resolution of that problem. I can forgive small service outages, but it's this lack of any ability to get resolution or closure or some "yeah, we had a failure, here's what we're doing to prevent it going forward" that is the real problem. Sadly, there is only so much choice I have in the matter of which cloud provider we're using…
numbsafari 5y ago
Interestingly... the stock is up 3% over the course of this outage so far.
- jasoncartwright 5y ago
  Sounds like when Fastly went down - the stock went up. Everyone suddenly realised how much of the internet they powered?
  - gruez 5y ago
    roblox powers much of the internet? I find that unlikely. If you don't have kids, this is probably the first you heard of it. I guess there's the "my kid is complaining to me that it's down and now I know about it" angle, but it's hard to draw any conclusions from your kid's purchasing habits alone.
    - pope_meat 5y ago
      roblox has the most under 18 users of anyone out there.
      - gruez 5y ago
        Right, but do you really need an outage for everyone to find that out? It's hard to notice that fastly powers a given site, but it's trivial to notice the game your kid's playing.
        kyawzazaw 5y ago
        yes, my parents wouldn't understand playstation or xbox
        numbsafari 5y ago
        My parents didn’t notice any of the games we were playing.
    - deleted 5y ago
rvz 5y ago
Well everything can be taken 'down', from blockchains to even real life by governments. For the metaverse to succeed, it should never be 'down'. Good luck with that if it is on a single platform or a single point of failure. Perhaps an interoperable metaverse / hyperverse then?
- throwaway1777 5y ago
  I think you’re right. Even Fb/Meta said interoperability is a key feature of the “metaverse”.
  - kwertyoowiyop 5y ago
    With everyone plugging into the Meta hub, of course.
  - ceejayoz 5y ago
    Sure. Interoperability between Facebook, Instagram, Messenger, and WhatsApp.
- kwertyoowiyop 5y ago
  Perhaps some sort of MetaChain…of MetaBlocks…is needed?
- jpindar 5y ago
  The various Open Simulator grids (known collectively as the hypergrid) are interoperable like that. As long as the one you created your avatar on (which can easily be a self-hosted one) is up, there are many worlds to teleport to.
EastOfTruth 5y ago
Wow... seems like a good time to sell Roblox stocks anyways... because most of Roblox' popular games appear to haven been cloned into Minecraft... earnings are coming up real soon though... I guess we'll find out in less then a week. edit: I edited this comment a bit to make it clearer because HN would not let me reply to the child comment.
- ALittleLight 5y ago
  I would think the opposite. This is a short term thing that presumably won't affect the business in any meaningful way. They will (probably) recover and be fine and nobody will be too worried that a kid's game has some, limited, downtime from time to time. If this outage causes a drop in share price, it's probably a good time to pick up a bit at a discount.
  - dylan604 5y ago
    >nobody will be too worried that a kid's game has some, limited, downtime from time to time except for the parents of kids addicted to the game
    - MiguelX413 5y ago
      We need to keep kids more offline.
    - EastOfTruth 5y ago
      lets make June, the month of no Internet for kids.
      - deathanatos 5y ago
        A whole month? Kid me would stage a coup on the spot.
        dylan604 5y ago
        that's okay, kids don't have rights. coup all you want. just do it in your room so the grown-ups can talk
dang 5y ago
Recent and related: Roblox Service Disruption - https://news.ycombinator.com/item?id=29034909 https://news.ycombinator.com/item?id=29034909 - Oct 2021 (40 comments)
fredley 5y ago
An insider told me that their internal secret store became overloaded, which caused every other service to suddenly fail to have access to any credentials.
- lexandstuff 5y ago
  Seems surprising that it would cause 24+ hours of downtime. My guess is they're recovering data from backup.
  - sidlls 5y ago
    It points to it being a serious infrastructure issue really. You don't have a hugely popular platform be dead for two days unless there is some fundamental issue with the infrastructure. Likely some stupid misconfiguration in some overengineered "cloud" architecture is the root cause, and they're mired in trying to debug and resolve the problem, fighting the complexity of the system the whole way.
  - justapassenger 5y ago
    Not surprising to me at all. No idea about their infra, but for complex enough, built in Silicon Valley move fast style (and they are likely in that bucket) restarting after such a failure can be a huge problem. You can have thundering herds, that you may not have levers to control, infrastructure needed to build and deploy fixes maybe down, you can have cyclical dependencies sneaking in the past, that makes it impossible to startup some systems, etc, etc
    - api 5y ago
      I really hate overengineered cloud infrastructure. It's the 2020s equivalent of 1990s to early 2000s enterprise Java except it's "use every conceivable cloud service in as baroque a deployment as possible" instead of "use every single design pattern in every part of the program." I refer to software built this way as "write once run once" as in it's so complex it could barely be redeployed and might not even be possible to restart. "Write once run once" is a play on the old "write once run anywhere" tagline for Java. But hey it makes Amazon rich. Amazon figured out how to monetize the tendency of sophomore engineers to add as much complexity as possible. I bet the authors of the original 1994 design patterns book are kicking themselves for not finding a way to charge for every implementation of the factory pattern.
dwarfstarlinux2 5y ago
lol
Wowfunhappy 5y ago
Completely unfounded speculation—what if they were hit by a ransomware attack? I wonder only because, well, after a full 24 hours... a purposefully malicious attack starts to seem more plausible to me than a simple freak accident.
cale- 5y ago
Amusing, John Carmack mentioned the metaverse will be something like Roblox https://arstechnica.com/gaming/2021/10/john-carmack-sounds-a-skeptical-note-over-metas-metaverse-plans/ https://arstechnica.com/gaming/2021/10/john-carmack-sounds-a... . FB (meta) taking drastic action to shutdown the competition
- petermcneeley 5y ago
  I think this is a pretty strong statement to make without any citation behind it.
  - derwiki 5y ago
    Not GP but I would interpret that as sarcasm
  - xeromal 5y ago
    I think the poster was joking
bemmu 5y ago
I'm curious where the traffic is now going that isn't going to Roblox? Steam player count doesn't seem to be jumping much. Maybe just mobile games?
- sgc 5y ago
  Youtube kids videos.
- empressplay 5y ago
  Probably. Roblox's original value proposition of 'kids making games for kids' has devolved into 'adults making dopamine slot-machines for kids' which is pretty sad :(
  - Bombthecat 5y ago
    For free!
- midwestemo 5y ago
  Steam is usually targeted 13+, since most users are under 13 I doubt most are on steam. Probably mobile games (as you said), Minecraft, YouTube, etc.
  - bonergarage 5y ago
    Can confirm. They are still picture-in-picture on YouTube, but playing Minecraft instead of Roblox.
- mensetmanusman 5y ago
  The kids are getting sleep they need? https://www.healthline.com/health-news/children-lack-of-sleep-health-problems https://www.healthline.com/health-news/children-lack-of-slee... Parent advice: Get your kids to sleep, it gives them a huge unfair advantage in school :) https://news.mit.edu/2019/better-sleep-better-grades-1001 https://news.mit.edu/2019/better-sleep-better-grades-1001
  - 5y ago
farmerstan 5y ago
Can someone please do the same for Minecraft? Maybe my kids will finally read a book
- shmde 5y ago
  Maybe your kids are already reading books from The Uncensored Library server in Minecraft :) Edit: son to kids.
anticensor 5y ago
Guilded integration?
lexe092666 5y ago
That’s weird

238 points by intunderflow 5y ago

Worse the the FB outage for parents… help!

I interviewed with their ops team there and turned it down. Just had a bad smell. I think they hire 'SRE' that can write good leet code. The problem with those folks for ops is that they over engineer complex solutions, and then completely forget the simple but very important stuff. I wish them luck in getting it back together.

ripper1138 5y ago
Leet code style questions for an SRE is so dumb. SRE is all about solving unexpected, unique problems. Opposite of leet code!

intunderflow 5y ago

There's probably a lesson in here about managing incidents, very few updates or acknowledgments there's even a major problem. Will be interesting to find out what caused an incident big enough to kill their entire platform for so long (if they release details or they come out through the grapevine)

stingraycharles 5y ago
Seems to me that a routing / networking issue would have been resolved by now, and application bug would have been rolled back. If I were to speculate, I would say that it must have something to do with databases / storage. Something must have gone wrong and broke some database, and it’s difficult to restore.
- intunderflow 5y ago
  They had an incident in the past where they returned nil for all get operations to their database, that caused some chaos at the time so it's not unprecedented: https://devforum.roblox.com/t/update-datastores-incident-data-consistency-checker-api/832679 https://devforum.roblox.com/t/update-datastores-incident-dat...
- polack 5y ago
  I agree that DB is the prime suspect here. They said the problem was identified over 24h ago, cant imagine it being something other than a data issue that takes that long to resolve.
deathanatos 5y ago
The status page acknowledges there is an incident, that it's pretty bad — everything is marked as down, and it's in red! What more could you want? (/s) Compare to Azure: most incidents never see the status page. Hell, getting support to acknowledge that an incident exists even once your internal investigation has reached certainty on "oh, yeah, it was them" is hard. There was an AAD outage earlier this year (?; IDK — I've lost track of the passage of time in the pandemic…) and the status page was down, and even once you managed to get the status page (IIRC you could hit the IP if you knew it magically which the Twitterverse did) … most services were still green, even if completely offline as far as one could tell by issuing queries to the service… And I'm comparing a kid's game with a "major" cloud PaaS… I'm definitely suffering from Stockholm syndrome.
- aliswe 5y ago
  a kids game yes, but worth 50B
- KronisLV 5y ago
  > Compare to Azure: most incidents never see the status page. That sounds like a conscious decision on their part. Everyone always talks about disclosure being the best policy, but at the same time there's plenty who believe that not informing anyone about an outage or even a breach is the correct thing to do, since then they'll probably get into less trouble themselves, or at least will create the illusion of not having as many outages as "those other guys". After all, informing everyone about an outage that will noticeably affect only some probably has a larger impact on the company's reputation than having it be dragged through mud in smaller communities for its dishonesty. Then again, with many of the larger services, it's not like you have much of a choice of using or not using it - you just get a corporate policy passed down upon you and that's that. Thus, sweeping problems under the rug and pretending that they don't exist is a dishonest, yet valid way of handling outages, breaches and so on. Personally, i'd avoid any service that does that, though it's not like that's being done just because of incompetence.
  - deathanatos 5y ago
    > That sounds like a conscious decision on their part. It is; I've been told by their support that they don't want to cause alarm. I think it's a bad way to run a PaaS, though. If I'm looking at your status page, it is because I suspect an outage and am trying to confirm it. Very willing to give some leeway to fix problems (an SLA — and Azure could do better here too — exists to establish what that allowable leeway is) I just need to know "is it me, or not?" and it's nicer to just get the answer when it's not me. As it is, I have to jump through a support hoop to get at "I think you are having an outage" and even then, it's typically multiple cycles before support seems to query engineering (and — that's another problem: support doesn't just know that there is an outstanding issue…) and gets to the bottom of it. It needs to be easy for a customer, experiencing an issue with a service, to drive resolution of that problem. I can forgive small service outages, but it's this lack of any ability to get resolution or closure or some "yeah, we had a failure, here's what we're doing to prevent it going forward" that is the real problem. Sadly, there is only so much choice I have in the matter of which cloud provider we're using…

numbsafari 5y ago

Interestingly... the stock is up 3% over the course of this outage so far.

jasoncartwright 5y ago
Sounds like when Fastly went down - the stock went up. Everyone suddenly realised how much of the internet they powered?
- gruez 5y ago
  roblox powers much of the internet? I find that unlikely. If you don't have kids, this is probably the first you heard of it. I guess there's the "my kid is complaining to me that it's down and now I know about it" angle, but it's hard to draw any conclusions from your kid's purchasing habits alone.
  - pope_meat 5y ago
    roblox has the most under 18 users of anyone out there.
    - gruez 5y ago
      Right, but do you really need an outage for everyone to find that out? It's hard to notice that fastly powers a given site, but it's trivial to notice the game your kid's playing.
      - kyawzazaw 5y ago
        yes, my parents wouldn't understand playstation or xbox
      - numbsafari 5y ago
        My parents didn’t notice any of the games we were playing.
  - deleted 5y ago

rvz 5y ago

Well everything can be taken 'down', from blockchains to even real life by governments. For the metaverse to succeed, it should never be 'down'. Good luck with that if it is on a single platform or a single point of failure. Perhaps an interoperable metaverse / hyperverse then?

throwaway1777 5y ago
I think you’re right. Even Fb/Meta said interoperability is a key feature of the “metaverse”.
- kwertyoowiyop 5y ago
  With everyone plugging into the Meta hub, of course.
- ceejayoz 5y ago
  Sure. Interoperability between Facebook, Instagram, Messenger, and WhatsApp.
kwertyoowiyop 5y ago
Perhaps some sort of MetaChain…of MetaBlocks…is needed?
jpindar 5y ago
The various Open Simulator grids (known collectively as the hypergrid) are interoperable like that. As long as the one you created your avatar on (which can easily be a self-hosted one) is up, there are many worlds to teleport to.

EastOfTruth 5y ago

Wow... seems like a good time to sell Roblox stocks anyways... because most of Roblox' popular games appear to haven been cloned into Minecraft... earnings are coming up real soon though... I guess we'll find out in less then a week. edit: I edited this comment a bit to make it clearer because HN would not let me reply to the child comment.

ALittleLight 5y ago
I would think the opposite. This is a short term thing that presumably won't affect the business in any meaningful way. They will (probably) recover and be fine and nobody will be too worried that a kid's game has some, limited, downtime from time to time. If this outage causes a drop in share price, it's probably a good time to pick up a bit at a discount.
- dylan604 5y ago
  >nobody will be too worried that a kid's game has some, limited, downtime from time to time except for the parents of kids addicted to the game
  - MiguelX413 5y ago
    We need to keep kids more offline.
  - EastOfTruth 5y ago
    lets make June, the month of no Internet for kids.
    - deathanatos 5y ago
      A whole month? Kid me would stage a coup on the spot.
      - dylan604 5y ago
        that's okay, kids don't have rights. coup all you want. just do it in your room so the grown-ups can talk

dang 5y ago

Recent and related: Roblox Service Disruption - https://news.ycombinator.com/item?id=29034909 https://news.ycombinator.com/item?id=29034909 - Oct 2021 (40 comments)

fredley 5y ago

An insider told me that their internal secret store became overloaded, which caused every other service to suddenly fail to have access to any credentials.

lexandstuff 5y ago
Seems surprising that it would cause 24+ hours of downtime. My guess is they're recovering data from backup.
- sidlls 5y ago
  It points to it being a serious infrastructure issue really. You don't have a hugely popular platform be dead for two days unless there is some fundamental issue with the infrastructure. Likely some stupid misconfiguration in some overengineered "cloud" architecture is the root cause, and they're mired in trying to debug and resolve the problem, fighting the complexity of the system the whole way.
- justapassenger 5y ago
  Not surprising to me at all. No idea about their infra, but for complex enough, built in Silicon Valley move fast style (and they are likely in that bucket) restarting after such a failure can be a huge problem. You can have thundering herds, that you may not have levers to control, infrastructure needed to build and deploy fixes maybe down, you can have cyclical dependencies sneaking in the past, that makes it impossible to startup some systems, etc, etc
  - api 5y ago
    I really hate overengineered cloud infrastructure. It's the 2020s equivalent of 1990s to early 2000s enterprise Java except it's "use every conceivable cloud service in as baroque a deployment as possible" instead of "use every single design pattern in every part of the program." I refer to software built this way as "write once run once" as in it's so complex it could barely be redeployed and might not even be possible to restart. "Write once run once" is a play on the old "write once run anywhere" tagline for Java. But hey it makes Amazon rich. Amazon figured out how to monetize the tendency of sophomore engineers to add as much complexity as possible. I bet the authors of the original 1994 design patterns book are kicking themselves for not finding a way to charge for every implementation of the factory pattern.

dwarfstarlinux2 5y ago

lol

Wowfunhappy 5y ago

Completely unfounded speculation—what if they were hit by a ransomware attack? I wonder only because, well, after a full 24 hours... a purposefully malicious attack starts to seem more plausible to me than a simple freak accident.

cale- 5y ago

Amusing, John Carmack mentioned the metaverse will be something like Roblox https://arstechnica.com/gaming/2021/10/john-carmack-sounds-a-skeptical-note-over-metas-metaverse-plans/ https://arstechnica.com/gaming/2021/10/john-carmack-sounds-a... . FB (meta) taking drastic action to shutdown the competition

petermcneeley 5y ago
I think this is a pretty strong statement to make without any citation behind it.
- derwiki 5y ago
  Not GP but I would interpret that as sarcasm
- xeromal 5y ago
  I think the poster was joking

bemmu 5y ago

I'm curious where the traffic is now going that isn't going to Roblox? Steam player count doesn't seem to be jumping much. Maybe just mobile games?

sgc 5y ago
Youtube kids videos.
empressplay 5y ago
Probably. Roblox's original value proposition of 'kids making games for kids' has devolved into 'adults making dopamine slot-machines for kids' which is pretty sad :(
- Bombthecat 5y ago
  For free!
midwestemo 5y ago
Steam is usually targeted 13+, since most users are under 13 I doubt most are on steam. Probably mobile games (as you said), Minecraft, YouTube, etc.
- bonergarage 5y ago
  Can confirm. They are still picture-in-picture on YouTube, but playing Minecraft instead of Roblox.
mensetmanusman 5y ago
The kids are getting sleep they need? https://www.healthline.com/health-news/children-lack-of-sleep-health-problems https://www.healthline.com/health-news/children-lack-of-slee... Parent advice: Get your kids to sleep, it gives them a huge unfair advantage in school :) https://news.mit.edu/2019/better-sleep-better-grades-1001 https://news.mit.edu/2019/better-sleep-better-grades-1001
- 5y ago

farmerstan 5y ago

Can someone please do the same for Minecraft? Maybe my kids will finally read a book

shmde 5y ago
Maybe your kids are already reading books from The Uncensored Library server in Minecraft :) Edit: son to kids.

anticensor 5y ago

Guilded integration?

lexe092666 5y ago

That’s weird