4 ms·

I work at OpenAI and I was the Incident Commander for yesterday's outage. We had a routing error within our infra that caused issues for some of our products.

I work at OpenAI and I was the Incident Commander for yesterday's outage. We had a routing error within our infra that caused issues for some of our products. It was not related to the Astra launch. We don't comment on other providers' outages.

dboreham 22d ago
Straight from the turkey's beak.
swader999 22d ago
Did you negotiate extra hard for the 'Incident Commander' title? I'm a bit jealous to be honest.
- eli 22d ago
  It's actually a standard term for the person who plays this role during incident response https://www.pagerduty.com/resources/incident-management-response/learn/what-is-incident-commander/ https://www.pagerduty.com/resources/incident-management-resp...
- theaxeonthanksg 22d ago
  [dead]
- OfficialTurkey 22d ago
  Ha. It's a role/title for the lifetime of the incident -- it's useful to have someone to keep things moving, keep track of workstreams, and to know who the decision-maker is, especially for bigger incidents. I'm just a SWE who works on infrastructure.
  - swader999 22d ago
    Hopefully you get a decked out command center too.
- stillpointlab 22d ago
  Note: Incident Command is almost certain an allusion (or implementation) of the Incident Command System [1] It is a common system in all kinds of emergency response scenarios, including local emergency services (fire/police/ambulance) and it scales all the way to massive disasters. It is especially useful to clarify command structures when multiple response entities need to coordinate. That is true even within organizations like public companies, where the reporting structures may be distinct. 1. https://en.wikipedia.org/wiki/Incident_Command_System https://en.wikipedia.org/wiki/Incident_Command_System
- alanfranz 22d ago
  It’s a standard term for a role during an incident.
theaxeonthanksg 22d ago
[dead]
deleted 22d ago
[deleted]
wnmurphy 22d ago
Curious then as to why the incident coincided with similar issues with Anthropic and xAI.
- wnmurphy 22d ago
  Actually, nevermind... this looks more like Anthropic and xAI coincided due to shared xAI infra (6:23am and 6:30am), and OpenAI's issue was more likely then a coincidence (7:43am).
  - reilly3000 22d ago
    Thank you. This should be pinned or something.
    - wnmurphy 22d ago
      xAI mentioned a Memphis outage, which is where Anthropic was leasing capacity on Colossus 1. I do think this is the explanation.
ncb_asj 22d ago
Shouldn't Astra be the Incident Commander?
- sshine 22d ago
  Secretly, they are.
grist_4_da_chil 22d ago
[dead]
honeybadger1 22d ago
At the same time the outages happened I noticed cloudflare also had reported issues with HTTP/3 and WARP. Related?

dboreham 22d ago

Straight from the turkey's beak.

swader999 22d ago

Did you negotiate extra hard for the 'Incident Commander' title? I'm a bit jealous to be honest.

eli 22d ago
It's actually a standard term for the person who plays this role during incident response https://www.pagerduty.com/resources/incident-management-response/learn/what-is-incident-commander/ https://www.pagerduty.com/resources/incident-management-resp...
theaxeonthanksg 22d ago
[dead]
OfficialTurkey 22d ago
Ha. It's a role/title for the lifetime of the incident -- it's useful to have someone to keep things moving, keep track of workstreams, and to know who the decision-maker is, especially for bigger incidents. I'm just a SWE who works on infrastructure.
- swader999 22d ago
  Hopefully you get a decked out command center too.
stillpointlab 22d ago
Note: Incident Command is almost certain an allusion (or implementation) of the Incident Command System [1] It is a common system in all kinds of emergency response scenarios, including local emergency services (fire/police/ambulance) and it scales all the way to massive disasters. It is especially useful to clarify command structures when multiple response entities need to coordinate. That is true even within organizations like public companies, where the reporting structures may be distinct. 1. https://en.wikipedia.org/wiki/Incident_Command_System https://en.wikipedia.org/wiki/Incident_Command_System
alanfranz 22d ago
It’s a standard term for a role during an incident.

theaxeonthanksg 22d ago

[dead]

deleted 22d ago

[deleted]

wnmurphy 22d ago

Curious then as to why the incident coincided with similar issues with Anthropic and xAI.

wnmurphy 22d ago
Actually, nevermind... this looks more like Anthropic and xAI coincided due to shared xAI infra (6:23am and 6:30am), and OpenAI's issue was more likely then a coincidence (7:43am).
- reilly3000 22d ago
  Thank you. This should be pinned or something.
  - wnmurphy 22d ago
    xAI mentioned a Memphis outage, which is where Anthropic was leasing capacity on Colossus 1. I do think this is the explanation.

ncb_asj 22d ago

Shouldn't Astra be the Incident Commander?

sshine 22d ago
Secretly, they are.

grist_4_da_chil 22d ago

[dead]

honeybadger1 22d ago

At the same time the outages happened I noticed cloudflare also had reported issues with HTTP/3 and WARP. Related?