8 ms·

My heart goes out to the people who likely had to work crazy hours to fix this, but it really is wild that it was down for so long. What was the last service o

by slg 5y ago

My heart goes out to the people who likely had to work crazy hours to fix this, but it really is wild that it was down for so long. What was the last service of this size that was down for 4 days? That is a failure in architecture that goes way beyond whatever the specific cause was here. That post-mortem is going to be a doozy.

raxxorrax 5y ago
We have hospitals being knocked out for weeks after ransomware attacks. If we complain about roblox I think we should reflect on our priorities.
- CrazyStat 5y ago
  Why not complain about both?
  - raxxorrax 5y ago
    You can, wasn't the point I tried to make. I tried to point out the arbitrary expectations here. Depending on the fault such downtime can last a few hours to a few weeks. I don't think you can make conclusions on architecture in this case. Perhaps it would be possible to roll back everything in just a few minutes but some users might lose a lot of work. If such special cases are to be resolved, it may take a lot of time. You just cannot really infer quality of architecture by downtime alone.
kaesar14 5y ago
The last outage of this kind I can even remember is the summer PSN was down for like, a month?
- slg 5y ago
  Although that PSN downtime probably fails into a different category as it was a response to being hacked. Sony intentionally took the service down and kept it down to investigate and fix their security holes. Basically the equivalent of temporarily closing a brick and mortar store after a robbery compared to being involuntarily closed because no one could find the right set of keys to unlock the store's front door.
- rStar 5y ago
  looks familiar to me also. was roblox also hacked?
  - Thorrez 5y ago
    From the blog post it sounds like no. They say a service got overloaded due to an increase in the number of datacenters and triggered a bug.
dilyevsky 5y ago
Not necessarily directed at roblox but honestly I’m surprised this doesn’t happen more often given how many teams I see run software they don't understand or don’t even have access to its source code. Edit: but yeah must’ve been tough 72+hrs i hope their version of reliability team can use this to bash the support they need out of the management and not get scapegoated instead
- sverhagen 5y ago
  I think it happens all the time. There just aren't that many services, relatively speaking, that you'd hear about when this happens. The more-popular services (that you'd hear about) probably already have a bias toward having their stuff better in order, since it's often also really just a matter of making the right investments.
  - donkeyd 5y ago
    > really just a matter of making the right investments Right. But the person deciding on those investments is a product owner who's looking to release new features because that's what management wants. At least that's my experience more often than not. My guess is that the bias might be the opposite. Maybe these companies become more popular because they are reliable and have the mindset to focus on that.
    - TheCoelacanth 5y ago
      By the time a company gets large enough to be a household name, they have usually been burned a few times and have someone technical who can push back against short-termism from POs.
    - lallysingh 5y ago
      Either you listen to and trust your people or your don't.
- that_guy_iain 5y ago
  At my company there was a service outage that lasted 2-3 weeks for a specific feature we have. This was caused by everyone quitting and no one having any experience with this service. The rest of the application remained working so it wasn't so noticable to the outside world. But internally and for customers it was massive since it was the billing system that went down. There was another incident that took down everything that was caused by a Spring cloud config being changed that no one in the company had access to.
keyle 5y ago
In the game industry, working yourself to death for 4 days, that's called a hot fix ;)
- bsenftner 5y ago
  And that is after you've been working 6 days a week for the last 9 months. As an EA spouse survivor - fuck that industry.
  - chasd00 5y ago
    “Spouse survivor”, I like that! I worked on the tech/distribution/data reporting side of the Covid vaccine and consider my wife, and all the sacrifices she made and extra things she handled up on while I was on the keyboard and phone, a critical component of my team’s success.
    - bsenftner 5y ago
      "EA Spouse" is the name is a specific period during the early 90's when Electronic Arts burned out an entire generation of game developers.
      - munificent 5y ago
        Here is the source of the term: https://ea-spouse.livejournal.com/274.html https://ea-spouse.livejournal.com/274.html
- Thaxll 5y ago
  People working on the cloud infra are not the people working on games.
hota_mazi 5y ago
It won't be a doozy: it will be a few paragraphs of general statements without saying anything specific, just like this communication (the first of consequence in days, another big failure on their part). Roblox has to start taking infrastructure a lot more seriously than they have so far.
withinboredom 5y ago
This is a benefit of working remote: I work at Automattic and have worked on teams where the members are literally around the world. When shit hits the fan, people are able to hand-off this kind of work to people who are fresh after a good night’s sleep and no one needs (or should need to) work crazy hours.
- op00to 5y ago
  This is a benefit of a global company, not working remote. You could do all that stuff in multiple offices around the world.
  - CrazyStat 5y ago
    In a global but not-remote company the team responsible for the particular service that failed is probably still concentrated in one office. Bringing in people from other offices who aren't familiar with the problem service probably isn't that helpful.
    - Steltek 5y ago
      You can have knowledge silos in either kind of company. We didn't magically get better at spreading work around or writing documentation just because we went remote.
    - iamstupidsimple 5y ago
      This has nothing to do with remote. Remote teams are usually concentrated in the same timezone too. In-office global companies make a point of having at least a few teams owning a service for this very reason.
- blowski 5y ago
  Automattic always sounds awesome to work for - I know a couple of people who work there as well. In your experience, when something like this happens, what can leaders do to help in the moment? I'm an engineering manager, and watching people getting over-stressed is much easier to do when you're physically in the office. Working remotely, it's easy to slip in the trap of either assuming no news is good news, or the opposite approach of asking "are you OK?" every 2 minutes. The only solution I've found is to all sit on a call, and pay attention to people's tone of voice, but I don't know whether that actually contributes to people's stress levels.
frabcus 5y ago
My favourite was the Skype outage in 2007 where a peer to peer algorithm didn’t scale - they had to work this out, and change the algorithm to fix it. It took ages! https://www.wired.com/2007/08/update-skype-ou/ https://www.wired.com/2007/08/update-skype-ou/
nodejs_rulez_1 5y ago
I am sure they got a lot of stock.
- Cthulhu_ 5y ago
  I mean I'd gladly pull an all-nighter if I were a millionaire thanks to the thing I'm fixing. I think. Then I'd quit and do something leisurely.
  - yawgmoth 5y ago
    Over what time frame would you tolerate poor working conditions to make a million bucks? Four year equity vesting? A million dollars in 2025 too, not a million dollars in 2010.
    - sidlls 5y ago
      Just one million? About 6 months, as I currently earn over $2MM per year from my options alone at the company I work for thanks to their IPO. Once those dry up it's going to be a hard sell for me to stay. Maybe I'll coast for a year on cash + RSU salary (which is much less than salary plus pre-IPO options) to top off my $5MM investment portfolio and ensure enough of it is invested to produce an income, but after that? No thanks.
swader999 5y ago
Not of this size or scale but ransomeware attacks typically take days to resolve. The pipeline attack and the large multinational vet company come to mind.
ksec 5y ago
I wonder if Engineers from Roblox are now worth a little more just because they have this experience. I also think it is time that people should take a look again at Chaos Engineering [1] from Netflix. It is sometimes ironic that the best technology often comes from companies that aren't a technology company at all. [1] https://principlesofchaos.org https://principlesofchaos.org
- bsenftner 5y ago
  And I wonder if their tech management is worth less for this clusterf** to happen during their oversight?
  - JohnJamesRambo 5y ago
    This is how I felt reading his comment. Were Volkswagen emission engineers and the captain of the Exxon Valdez worth more after they screwed up?
- darkerside 5y ago
  Are you implying Netflix is not a tech company?
  - ksec 5y ago
    I definitely see Netflix as a Media Company, much like Disney. Disney and Pixar have great tech too, but they are not tech company in any shape or form. How Netflix got lumped into "FAANG" aka Big Tech is still a mystery to me.
    - kerng 5y ago
      By that logic Facebook certainly wouldn't be a tech company either.
      - geffchang 5y ago
        Why not? Didn't Facebook create tools like GraphQL and React?
nonameiguess 5y ago
Not a software outage, but the Texas ice storm of February resulted in me not receiving mail for two weeks, not being able to use the streets for at least a week (they never plowed and it just eventually melted), rolling blackout where I had power for about four hours a day, and no Internet service at all for the first two days. No water for most of the first day. I'd argue keeping basic city services and roads operational is more important than a gaming platform, but city and state governments in the US often seem to not agree and are more than happy to save money skimping on maintenance and resilience.
- AnIdiotOnTheNet 5y ago
  That behavior is a reflection of their citizenry who have been convinced by propaganda that government can never be effective at anything and therefore should not be funded properly.
  - BuyMyBitcoins 5y ago
    >”citizenry who have been convinced by propaganda that government can never be effective at anything and therefore should not be funded properly.” People are treating the Texas ice storm like it was some comeuppance event but the truth of the matter is that Texas rarely experiences this kind of winter weather and that’s the reason why it didn’t handle the big freeze the same way as a midwestern or northeastern state would. The reason why cities like DFW do not maintain a fleet of snowplows isn't because of anti-government propaganda, but because it is completely uneconomical to do so. Even during the big freeze there wasn’t a foot of snow on the ground. The major roadways were salted, though, and you could drive on them. Snow and icing events tend to only happen for a day or two each year in my part of Texas. And it is often the case that when temperatures do dip below zero there isn’t preceding rain or snow. But when that does happen, most people just ride it out for a day or two and wait for things to melt.
    - harrylove 5y ago
      NERC created a report after the Texas 2011 ice storm power outage with recommendations for ERCOT generators to improve their weatherization standards [0]. In that report they referenced not only the 2011 incident but previous incidents where the grid generators failed to adequately prepare for potential problems due to weather related issues. In the 2021 storm, hundreds of people died. Comeuppance is not the right word. I don't know the right one, but I believe it's closer to willful blindness or predatory delay. [0] https://www.ferc.gov/sites/default/files/2020-04/08-16-11-report.pdf https://www.ferc.gov/sites/default/files/2020-04/08-16-11-re...
sidlls 5y ago
The immediate technical root cause and the true root cause (often down to poor decision making among the senior technical staff and management) are two different things, and I suspect the public and internal post-mortems both will focus on the former rather than the latter, sadly.
madeofpalk 5y ago
> What was the last service of this size that was down for 4 days Was "only" two days, but Fortnite just shutdown for 2 days as a part of an end-of-season event back in 2019 https://www.polygon.com/fortnite/2019/10/13/20911691/fortnite-season-10-event-black-hole-map-disappears https://www.polygon.com/fortnite/2019/10/13/20911691/fortnit... https://www.polygon.com/fortnite/2019/10/13/20911691/fortnite-season-10-event-black-hole-map-disappears https://www.polygon.com/fortnite/2019/10/13/20911691/fortnit...

by slg 5y ago

raxxorrax 5y ago

We have hospitals being knocked out for weeks after ransomware attacks. If we complain about roblox I think we should reflect on our priorities.

CrazyStat 5y ago
Why not complain about both?
- raxxorrax 5y ago
  You can, wasn't the point I tried to make. I tried to point out the arbitrary expectations here. Depending on the fault such downtime can last a few hours to a few weeks. I don't think you can make conclusions on architecture in this case. Perhaps it would be possible to roll back everything in just a few minutes but some users might lose a lot of work. If such special cases are to be resolved, it may take a lot of time. You just cannot really infer quality of architecture by downtime alone.

kaesar14 5y ago

The last outage of this kind I can even remember is the summer PSN was down for like, a month?

slg 5y ago
Although that PSN downtime probably fails into a different category as it was a response to being hacked. Sony intentionally took the service down and kept it down to investigate and fix their security holes. Basically the equivalent of temporarily closing a brick and mortar store after a robbery compared to being involuntarily closed because no one could find the right set of keys to unlock the store's front door.
rStar 5y ago
looks familiar to me also. was roblox also hacked?
- Thorrez 5y ago
  From the blog post it sounds like no. They say a service got overloaded due to an increase in the number of datacenters and triggered a bug.

dilyevsky 5y ago

Not necessarily directed at roblox but honestly I’m surprised this doesn’t happen more often given how many teams I see run software they don't understand or don’t even have access to its source code. Edit: but yeah must’ve been tough 72+hrs i hope their version of reliability team can use this to bash the support they need out of the management and not get scapegoated instead

sverhagen 5y ago
I think it happens all the time. There just aren't that many services, relatively speaking, that you'd hear about when this happens. The more-popular services (that you'd hear about) probably already have a bias toward having their stuff better in order, since it's often also really just a matter of making the right investments.
- donkeyd 5y ago
  > really just a matter of making the right investments Right. But the person deciding on those investments is a product owner who's looking to release new features because that's what management wants. At least that's my experience more often than not. My guess is that the bias might be the opposite. Maybe these companies become more popular because they are reliable and have the mindset to focus on that.
  - TheCoelacanth 5y ago
    By the time a company gets large enough to be a household name, they have usually been burned a few times and have someone technical who can push back against short-termism from POs.
  - lallysingh 5y ago
    Either you listen to and trust your people or your don't.
that_guy_iain 5y ago
At my company there was a service outage that lasted 2-3 weeks for a specific feature we have. This was caused by everyone quitting and no one having any experience with this service. The rest of the application remained working so it wasn't so noticable to the outside world. But internally and for customers it was massive since it was the billing system that went down. There was another incident that took down everything that was caused by a Spring cloud config being changed that no one in the company had access to.

keyle 5y ago

In the game industry, working yourself to death for 4 days, that's called a hot fix ;)

bsenftner 5y ago
And that is after you've been working 6 days a week for the last 9 months. As an EA spouse survivor - fuck that industry.
- chasd00 5y ago
  “Spouse survivor”, I like that! I worked on the tech/distribution/data reporting side of the Covid vaccine and consider my wife, and all the sacrifices she made and extra things she handled up on while I was on the keyboard and phone, a critical component of my team’s success.
  - bsenftner 5y ago
    "EA Spouse" is the name is a specific period during the early 90's when Electronic Arts burned out an entire generation of game developers.
    - munificent 5y ago
      Here is the source of the term: https://ea-spouse.livejournal.com/274.html https://ea-spouse.livejournal.com/274.html
Thaxll 5y ago
People working on the cloud infra are not the people working on games.

hota_mazi 5y ago

It won't be a doozy: it will be a few paragraphs of general statements without saying anything specific, just like this communication (the first of consequence in days, another big failure on their part). Roblox has to start taking infrastructure a lot more seriously than they have so far.

withinboredom 5y ago

This is a benefit of working remote: I work at Automattic and have worked on teams where the members are literally around the world. When shit hits the fan, people are able to hand-off this kind of work to people who are fresh after a good night’s sleep and no one needs (or should need to) work crazy hours.

op00to 5y ago
This is a benefit of a global company, not working remote. You could do all that stuff in multiple offices around the world.
- CrazyStat 5y ago
  In a global but not-remote company the team responsible for the particular service that failed is probably still concentrated in one office. Bringing in people from other offices who aren't familiar with the problem service probably isn't that helpful.
  - Steltek 5y ago
    You can have knowledge silos in either kind of company. We didn't magically get better at spreading work around or writing documentation just because we went remote.
  - iamstupidsimple 5y ago
    This has nothing to do with remote. Remote teams are usually concentrated in the same timezone too. In-office global companies make a point of having at least a few teams owning a service for this very reason.
blowski 5y ago
Automattic always sounds awesome to work for - I know a couple of people who work there as well. In your experience, when something like this happens, what can leaders do to help in the moment? I'm an engineering manager, and watching people getting over-stressed is much easier to do when you're physically in the office. Working remotely, it's easy to slip in the trap of either assuming no news is good news, or the opposite approach of asking "are you OK?" every 2 minutes. The only solution I've found is to all sit on a call, and pay attention to people's tone of voice, but I don't know whether that actually contributes to people's stress levels.

frabcus 5y ago

My favourite was the Skype outage in 2007 where a peer to peer algorithm didn’t scale - they had to work this out, and change the algorithm to fix it. It took ages! https://www.wired.com/2007/08/update-skype-ou/ https://www.wired.com/2007/08/update-skype-ou/

nodejs_rulez_1 5y ago

I am sure they got a lot of stock.

Cthulhu_ 5y ago
I mean I'd gladly pull an all-nighter if I were a millionaire thanks to the thing I'm fixing. I think. Then I'd quit and do something leisurely.
- yawgmoth 5y ago
  Over what time frame would you tolerate poor working conditions to make a million bucks? Four year equity vesting? A million dollars in 2025 too, not a million dollars in 2010.
  - sidlls 5y ago
    Just one million? About 6 months, as I currently earn over $2MM per year from my options alone at the company I work for thanks to their IPO. Once those dry up it's going to be a hard sell for me to stay. Maybe I'll coast for a year on cash + RSU salary (which is much less than salary plus pre-IPO options) to top off my $5MM investment portfolio and ensure enough of it is invested to produce an income, but after that? No thanks.

swader999 5y ago

Not of this size or scale but ransomeware attacks typically take days to resolve. The pipeline attack and the large multinational vet company come to mind.

ksec 5y ago

I wonder if Engineers from Roblox are now worth a little more just because they have this experience. I also think it is time that people should take a look again at Chaos Engineering [1] from Netflix. It is sometimes ironic that the best technology often comes from companies that aren't a technology company at all. [1] https://principlesofchaos.org https://principlesofchaos.org

bsenftner 5y ago
And I wonder if their tech management is worth less for this clusterf** to happen during their oversight?
- JohnJamesRambo 5y ago
  This is how I felt reading his comment. Were Volkswagen emission engineers and the captain of the Exxon Valdez worth more after they screwed up?
darkerside 5y ago
Are you implying Netflix is not a tech company?
- ksec 5y ago
  I definitely see Netflix as a Media Company, much like Disney. Disney and Pixar have great tech too, but they are not tech company in any shape or form. How Netflix got lumped into "FAANG" aka Big Tech is still a mystery to me.
  - kerng 5y ago
    By that logic Facebook certainly wouldn't be a tech company either.
    - geffchang 5y ago
      Why not? Didn't Facebook create tools like GraphQL and React?

nonameiguess 5y ago

Not a software outage, but the Texas ice storm of February resulted in me not receiving mail for two weeks, not being able to use the streets for at least a week (they never plowed and it just eventually melted), rolling blackout where I had power for about four hours a day, and no Internet service at all for the first two days. No water for most of the first day. I'd argue keeping basic city services and roads operational is more important than a gaming platform, but city and state governments in the US often seem to not agree and are more than happy to save money skimping on maintenance and resilience.

AnIdiotOnTheNet 5y ago
That behavior is a reflection of their citizenry who have been convinced by propaganda that government can never be effective at anything and therefore should not be funded properly.
- BuyMyBitcoins 5y ago
  >”citizenry who have been convinced by propaganda that government can never be effective at anything and therefore should not be funded properly.” People are treating the Texas ice storm like it was some comeuppance event but the truth of the matter is that Texas rarely experiences this kind of winter weather and that’s the reason why it didn’t handle the big freeze the same way as a midwestern or northeastern state would. The reason why cities like DFW do not maintain a fleet of snowplows isn't because of anti-government propaganda, but because it is completely uneconomical to do so. Even during the big freeze there wasn’t a foot of snow on the ground. The major roadways were salted, though, and you could drive on them. Snow and icing events tend to only happen for a day or two each year in my part of Texas. And it is often the case that when temperatures do dip below zero there isn’t preceding rain or snow. But when that does happen, most people just ride it out for a day or two and wait for things to melt.
  - harrylove 5y ago
    NERC created a report after the Texas 2011 ice storm power outage with recommendations for ERCOT generators to improve their weatherization standards [0]. In that report they referenced not only the 2011 incident but previous incidents where the grid generators failed to adequately prepare for potential problems due to weather related issues. In the 2021 storm, hundreds of people died. Comeuppance is not the right word. I don't know the right one, but I believe it's closer to willful blindness or predatory delay. [0] https://www.ferc.gov/sites/default/files/2020-04/08-16-11-report.pdf https://www.ferc.gov/sites/default/files/2020-04/08-16-11-re...

sidlls 5y ago

The immediate technical root cause and the true root cause (often down to poor decision making among the senior technical staff and management) are two different things, and I suspect the public and internal post-mortems both will focus on the former rather than the latter, sadly.

madeofpalk 5y ago

> What was the last service of this size that was down for 4 days Was "only" two days, but Fortnite just shutdown for 2 days as a part of an end-of-season event back in 2019 https://www.polygon.com/fortnite/2019/10/13/20911691/fortnite-season-10-event-black-hole-map-disappears https://www.polygon.com/fortnite/2019/10/13/20911691/fortnit... https://www.polygon.com/fortnite/2019/10/13/20911691/fortnite-season-10-event-black-hole-map-disappears https://www.polygon.com/fortnite/2019/10/13/20911691/fortnit...