3 ms·

I mean this is honestly a PR/lawyer blog post. > A core system in our infrastructure became overwhelmed, prompted by a subtle bug in our backend service commu

by htrp 5y ago

I mean this is honestly a PR/lawyer blog post. > A core system in our infrastructure became overwhelmed, prompted by a subtle bug in our backend service communications while under heavy load. Translation Something important went down, we couldn't trace the bug. >This was not due to any peak in external traffic or any particular experience. Translation We're not blaming Chipotle for this > most services at Roblox were unable to effectively communicate and deploy. Translation We couldn't throw more compute at the problem to manage the traffic or to put up a failover copy of anything. >To the best of our knowledge, there has been no loss of player persistence data Translation We literally restored our entire infrastructure from scratch and probably our last set of backups. You might have lost some stuff but we don't know what you lost since we couldn't get the last copy of our databases.

judge2020 5y ago
> We literally restored our entire infrastructure from scratch and probably our last set of backups. You might have lost some stuff but we don't know what you lost since we couldn't get the last copy of our databases. What a wild extrapolation. The line in the blog post is not them writing to hint at what might have happened, it's so that the customers reading don't think this means they lost any of their data/items/etc. The Roblox marketplace is vast and high-volume; if anyone lost <any> data from them restoring a backup, we'd already know about it.
deleted 5y ago
[deleted]
toast0 5y ago
You missed the part where they described the root issue: > Rather the failure was caused by the growth in the number of servers in our datacenters. Their config server ran out of sockets and screwed up everything. Or something like that. Too many servers = nothing works has many ways to unfold and isn't obvious the first time.
- rawoke083600 5y ago
  >too many servers... My best advise about scaling, I once read (might have been right here on HN) is many devs know their primary bottleneck. Be it the "master-db", 3rd-party-api, login-service etc. But few devs knows the "2nd slowest/bottleneck" in their service stack. Not saying this was the case here, but always good to think (at least has a mental exercise) about what comes after your primary-bottleneck. If it's a long standing bottleneck, one day it gets fixed (sw update, better arch) and suddenly everything goes down since now you hitting a new bottleneck you never even thought about scaling.

by htrp 5y ago

judge2020 5y ago

> We literally restored our entire infrastructure from scratch and probably our last set of backups. You might have lost some stuff but we don't know what you lost since we couldn't get the last copy of our databases. What a wild extrapolation. The line in the blog post is not them writing to hint at what might have happened, it's so that the customers reading don't think this means they lost any of their data/items/etc. The Roblox marketplace is vast and high-volume; if anyone lost <any> data from them restoring a backup, we'd already know about it.

deleted 5y ago

[deleted]

toast0 5y ago

You missed the part where they described the root issue: > Rather the failure was caused by the growth in the number of servers in our datacenters. Their config server ran out of sockets and screwed up everything. Or something like that. Too many servers = nothing works has many ways to unfold and isn't obvious the first time.

rawoke083600 5y ago
>too many servers... My best advise about scaling, I once read (might have been right here on HN) is many devs know their primary bottleneck. Be it the "master-db", 3rd-party-api, login-service etc. But few devs knows the "2nd slowest/bottleneck" in their service stack. Not saying this was the case here, but always good to think (at least has a mental exercise) about what comes after your primary-bottleneck. If it's a long standing bottleneck, one day it gets fixed (sw update, better arch) and suddenly everything goes down since now you hitting a new bottleneck you never even thought about scaling.