4 ms·

That's a very long outage, I wonder how this happened. - Perhaps internal systems they've developed, and the people who created them left. So it's not just fix

That's a very long outage, I wonder how this happened. - Perhaps internal systems they've developed, and the people who created them left. So it's not just fix the thing, but first understand what the thing is doing and then fix it. - Data recovery can take forever if you run into edge cases with your databases Anyone found any articles about their architecture?

ekovarski 5y ago
Hashicorp has a case study with some details of what they use, https://www.hashicorp.com/case-studies/roblox https://www.hashicorp.com/case-studies/roblox
- ytjohn 5y ago
  > We have people who are first-time system administrators deploying applications. building containers, maintaining Nomad. There is a guy on our team who worked in IT help desk for eight years - just today he upgraded an entire cluster himself.”
  - sidlls 5y ago
    I know everyone has to start somewhere, but this just sounds like a sure way to have a platform blow up impressively.
- breadzeppelin__ 5y ago
  oof (from the link). Debugging nightmare to have a fairly inexperienced team trying to diagnose stuff in these incredibly complex systems. > 4 SREs managing Nomad, Consul, and Vault for 11,000+ nodes across 22 clusters, serving 420+ internal developers > "We have people who are first-time system administrators deploying applications, building containers, maintaining Nomad. There is a guy on our team who worked in the IT help desk for eight years — just today he upgraded an entire cluster himself."
kawsper 5y ago
I bet it is some sort of Vault/Consul shenanigans that's going on.
- wernerb 5y ago
  When I saw "secret store" I guessed it had to be Vault. Vault's is amazing but it lets you configure things that can blow up on you in X time. For example, issuing 50 secrets per second but have every secret expire after a week (or never). It would mean (multiple) goroutine per secret checking status on the lease. This kind of thing unfortunately, is easy to miss and occur in Vault.
  - tentacleuno 5y ago
    Secret expiration seems like a better thing to do from the application: check the token every time it's used, and if it's past a week mark it as expired. Combine this with caching, etc. Is there an advantage to having such a system in the database?
    - wernerb 5y ago
      What you suggest does scale with performance but not with organization its why Vault is used in the first place.
- flutas 5y ago
  The person mentioning secret store, and the blog post they put up does seem to somewhat point to Vault, but no name drop. > A core system in our infrastructure became overwhelmed, prompted by a subtle bug in our backend service communications while under heavy load. This was not due to any peak in external traffic or any particular experience. Rather the failure was caused by the growth in the number of servers in our datacenters. The result was that most services at Roblox were unable to effectively communicate and deploy. https://blog.roblox.com/2021/10/update-on-our-outage/ https://blog.roblox.com/2021/10/update-on-our-outage/

ekovarski 5y ago

Hashicorp has a case study with some details of what they use, https://www.hashicorp.com/case-studies/roblox https://www.hashicorp.com/case-studies/roblox

ytjohn 5y ago
> We have people who are first-time system administrators deploying applications. building containers, maintaining Nomad. There is a guy on our team who worked in IT help desk for eight years - just today he upgraded an entire cluster himself.”
- sidlls 5y ago
  I know everyone has to start somewhere, but this just sounds like a sure way to have a platform blow up impressively.
breadzeppelin__ 5y ago
oof (from the link). Debugging nightmare to have a fairly inexperienced team trying to diagnose stuff in these incredibly complex systems. > 4 SREs managing Nomad, Consul, and Vault for 11,000+ nodes across 22 clusters, serving 420+ internal developers > "We have people who are first-time system administrators deploying applications, building containers, maintaining Nomad. There is a guy on our team who worked in the IT help desk for eight years — just today he upgraded an entire cluster himself."

kawsper 5y ago

I bet it is some sort of Vault/Consul shenanigans that's going on.

wernerb 5y ago
When I saw "secret store" I guessed it had to be Vault. Vault's is amazing but it lets you configure things that can blow up on you in X time. For example, issuing 50 secrets per second but have every secret expire after a week (or never). It would mean (multiple) goroutine per secret checking status on the lease. This kind of thing unfortunately, is easy to miss and occur in Vault.
- tentacleuno 5y ago
  Secret expiration seems like a better thing to do from the application: check the token every time it's used, and if it's past a week mark it as expired. Combine this with caching, etc. Is there an advantage to having such a system in the database?
  - wernerb 5y ago
    What you suggest does scale with performance but not with organization its why Vault is used in the first place.
flutas 5y ago
The person mentioning secret store, and the blog post they put up does seem to somewhat point to Vault, but no name drop. > A core system in our infrastructure became overwhelmed, prompted by a subtle bug in our backend service communications while under heavy load. This was not due to any peak in external traffic or any particular experience. Rather the failure was caused by the growth in the number of servers in our datacenters. The result was that most services at Roblox were unable to effectively communicate and deploy. https://blog.roblox.com/2021/10/update-on-our-outage/ https://blog.roblox.com/2021/10/update-on-our-outage/