4 ms·

Did they run out of internal ipv4 address space somehow? I’ll be curious for the post-mortem, this is definitely the longest outage of a global web service I ca

Did they run out of internal ipv4 address space somehow? I’ll be curious for the post-mortem, this is definitely the longest outage of a global web service I can remember in… forever? This incident puts them under 2 9’s.

kevan 5y ago
My guess is something related to service mesh/discovery. If that goes down, deployment infra also depends on it, and you don't have a solid black start protocol then you're in for a very bad week.
- bigiain 5y ago
  That or DNS. ;-)
  - sodality2 5y ago
    Or maybe BGP >:)
    - bigiain 5y ago
      Or a compiler bug... http://r6.ca/blog/20200929T023701Z.html http://r6.ca/blog/20200929T023701Z.html
  - tjr225 5y ago
    Service mesh is basically abstracted DNS, so…
dannyw 5y ago
Not a lot of details here but I'd suspect it's yet another example of autoscaling policies gone wrong.
- reilly3000 5y ago
  What does that tend to look like? Do you have any examples?
  - atomicnumber3 5y ago
    In my experience, it tends to be that you find out one of your components has some tipping point as you scale out horizontally where everything goes to hell. Random example: you scale a service horizontally and suddenly postgres disk space usage goes from "this line looks horizontal" to "we just used 2 years' worth of disk use growth in 40 seconds". And the postmortem is basically like "yep can't really blame anyone for not knowing this weird ass thing or unfound bug". Software, man.
    - asn1parse 5y ago
      yep.
    - deanCommie 5y ago
      This totally happens, and is a thing, but tracing a "unexpected change in metrics on our DB server to "increase in metric showing call volume to DB server" to "increase in metric showing increase in fleet size of app servers that call the DB server" takes minutes to identify, not 24 hours. And the remediation procedure is quite simple too - scale down. If the fleet can't handle it, increase your throttling. If even with increased throttling you can't handle the load and everybody's timing out (but you still can't scale out), you start spinning up new load balancers, and applying weights in DNS where at least some percentage of your callers are getting through while others see full failure. The key point is in anything related to such an outage, even if it takes a long time to recover, your customers shouldn't see a full outage. So it was either an entirely different class of problems or Roblex has a huge ops skills gap. I'm betting the former. edit: never mind, maybe it's the latter after all: https://twitter.com/NIDeveloper/status/1454773313792880640 https://twitter.com/NIDeveloper/status/1454773313792880640
    - suresk 5y ago
      I think there are two big things that can be really hard with scalability - 1) There are a lot of dimensions I think it is fairly common to focus on a throughput-related dimension like requests per second and test that very thoroughly, while not paying enough attention to other aspects. We have services where, if we were told requests per second were going to go up 100x tomorrow, I wouldn't be too concerned, but if we were told request size were going up 2x, I'd be freaking out. Even when you think you've identified all of the dimensions, there are usually ones you missed and/or they interact in weird and unexpected ways. 2) Performance can appear to be linear when it really isn't So many outages I've seen have resulted from everything being fine until some threshold is hit, at which point it doesn't start to slowly degrade, but instead immediately explodes. Often times due to feedback loops (GC activity in garbage-collected languages often can behave this way) or because some cache overflowed (data that is queried at a high rate no longer fitting in memory on a DBMS comes to mind).

kevan 5y ago

My guess is something related to service mesh/discovery. If that goes down, deployment infra also depends on it, and you don't have a solid black start protocol then you're in for a very bad week.

bigiain 5y ago
That or DNS. ;-)
- sodality2 5y ago
  Or maybe BGP >:)
  - bigiain 5y ago
    Or a compiler bug... http://r6.ca/blog/20200929T023701Z.html http://r6.ca/blog/20200929T023701Z.html
- tjr225 5y ago
  Service mesh is basically abstracted DNS, so…

dannyw 5y ago

Not a lot of details here but I'd suspect it's yet another example of autoscaling policies gone wrong.

reilly3000 5y ago
What does that tend to look like? Do you have any examples?
- atomicnumber3 5y ago
  In my experience, it tends to be that you find out one of your components has some tipping point as you scale out horizontally where everything goes to hell. Random example: you scale a service horizontally and suddenly postgres disk space usage goes from "this line looks horizontal" to "we just used 2 years' worth of disk use growth in 40 seconds". And the postmortem is basically like "yep can't really blame anyone for not knowing this weird ass thing or unfound bug". Software, man.
  - asn1parse 5y ago
    yep.
  - deanCommie 5y ago
    This totally happens, and is a thing, but tracing a "unexpected change in metrics on our DB server to "increase in metric showing call volume to DB server" to "increase in metric showing increase in fleet size of app servers that call the DB server" takes minutes to identify, not 24 hours. And the remediation procedure is quite simple too - scale down. If the fleet can't handle it, increase your throttling. If even with increased throttling you can't handle the load and everybody's timing out (but you still can't scale out), you start spinning up new load balancers, and applying weights in DNS where at least some percentage of your callers are getting through while others see full failure. The key point is in anything related to such an outage, even if it takes a long time to recover, your customers shouldn't see a full outage. So it was either an entirely different class of problems or Roblex has a huge ops skills gap. I'm betting the former. edit: never mind, maybe it's the latter after all: https://twitter.com/NIDeveloper/status/1454773313792880640 https://twitter.com/NIDeveloper/status/1454773313792880640
  - suresk 5y ago
    I think there are two big things that can be really hard with scalability - 1) There are a lot of dimensions I think it is fairly common to focus on a throughput-related dimension like requests per second and test that very thoroughly, while not paying enough attention to other aspects. We have services where, if we were told requests per second were going to go up 100x tomorrow, I wouldn't be too concerned, but if we were told request size were going up 2x, I'd be freaking out. Even when you think you've identified all of the dimensions, there are usually ones you missed and/or they interact in weird and unexpected ways. 2) Performance can appear to be linear when it really isn't So many outages I've seen have resulted from everything being fine until some threshold is hit, at which point it doesn't start to slowly degrade, but instead immediately explodes. Often times due to feedback loops (GC activity in garbage-collected languages often can behave this way) or because some cache overflowed (data that is queried at a high rate no longer fitting in memory on a DBMS comes to mind).