12 ms·

I work at OpenAI and I was the Incident Commander for yesterday's outage. We had a routing error within our infra that caused issues for some of our products.

I work at OpenAI and I was the Incident Commander for yesterday's outage. We had a routing error within our infra that caused issues for some of our products. It was not related to the Astra launch. We don't comment on other providers' outages.

dboreham 2mo ago
Straight from the turkey's beak.
swader999 2mo ago
Did you negotiate extra hard for the 'Incident Commander' title? I'm a bit jealous to be honest.
- eli 2mo ago
  It's actually a standard term for the person who plays this role during incident response https://www.pagerduty.com/resources/incident-management-response/learn/what-is-incident-commander/ https://www.pagerduty.com/resources/incident-management-resp...
- theaxeonthanksg 2mo ago
  [dead]
- OfficialTurkey 2mo ago
  Ha. It's a role/title for the lifetime of the incident -- it's useful to have someone to keep things moving, keep track of workstreams, and to know who the decision-maker is, especially for bigger incidents. I'm just a SWE who works on infrastructure.
  - swader999 2mo ago
    Hopefully you get a decked out command center too.
- stillpointlab 2mo ago
  Note: Incident Command is almost certain an allusion (or implementation) of the Incident Command System [1] It is a common system in all kinds of emergency response scenarios, including local emergency services (fire/police/ambulance) and it scales all the way to massive disasters. It is especially useful to clarify command structures when multiple response entities need to coordinate. That is true even within organizations like public companies, where the reporting structures may be distinct. 1. https://en.wikipedia.org/wiki/Incident_Command_System https://en.wikipedia.org/wiki/Incident_Command_System
- alanfranz 2mo ago
  It’s a standard term for a role during an incident.
theaxeonthanksg 2mo ago
[dead]
deleted 2mo ago
[deleted]
wnmurphy 2mo ago
Curious then as to why the incident coincided with similar issues with Anthropic and xAI.
- wnmurphy 2mo ago
  Actually, nevermind... this looks more like Anthropic and xAI coincided due to shared xAI infra (6:23am and 6:30am), and OpenAI's issue was more likely then a coincidence (7:43am).
  - reilly3000 2mo ago
    Thank you. This should be pinned or something.
    - wnmurphy 2mo ago
      xAI mentioned a Memphis outage, which is where Anthropic was leasing capacity on Colossus 1. I do think this is the explanation.
ncb_asj 2mo ago
Shouldn't Astra be the Incident Commander?
- sshine 2mo ago
  Secretly, they are.
grist_4_da_chil 2mo ago
[dead]
honeybadger1 2mo ago
At the same time the outages happened I noticed cloudflare also had reported issues with HTTP/3 and WARP. Related?

dboreham 2mo ago

Straight from the turkey's beak.

swader999 2mo ago

Did you negotiate extra hard for the 'Incident Commander' title? I'm a bit jealous to be honest.

eli 2mo ago
It's actually a standard term for the person who plays this role during incident response https://www.pagerduty.com/resources/incident-management-response/learn/what-is-incident-commander/ https://www.pagerduty.com/resources/incident-management-resp...
theaxeonthanksg 2mo ago
[dead]
OfficialTurkey 2mo ago
Ha. It's a role/title for the lifetime of the incident -- it's useful to have someone to keep things moving, keep track of workstreams, and to know who the decision-maker is, especially for bigger incidents. I'm just a SWE who works on infrastructure.
- swader999 2mo ago
  Hopefully you get a decked out command center too.
stillpointlab 2mo ago
Note: Incident Command is almost certain an allusion (or implementation) of the Incident Command System [1] It is a common system in all kinds of emergency response scenarios, including local emergency services (fire/police/ambulance) and it scales all the way to massive disasters. It is especially useful to clarify command structures when multiple response entities need to coordinate. That is true even within organizations like public companies, where the reporting structures may be distinct. 1. https://en.wikipedia.org/wiki/Incident_Command_System https://en.wikipedia.org/wiki/Incident_Command_System
alanfranz 2mo ago
It’s a standard term for a role during an incident.

theaxeonthanksg 2mo ago

[dead]

deleted 2mo ago

[deleted]

wnmurphy 2mo ago

Curious then as to why the incident coincided with similar issues with Anthropic and xAI.

wnmurphy 2mo ago
Actually, nevermind... this looks more like Anthropic and xAI coincided due to shared xAI infra (6:23am and 6:30am), and OpenAI's issue was more likely then a coincidence (7:43am).
- reilly3000 2mo ago
  Thank you. This should be pinned or something.
  - wnmurphy 2mo ago
    xAI mentioned a Memphis outage, which is where Anthropic was leasing capacity on Colossus 1. I do think this is the explanation.

ncb_asj 2mo ago

Shouldn't Astra be the Incident Commander?

sshine 2mo ago
Secretly, they are.

grist_4_da_chil 2mo ago

[dead]

honeybadger1 2mo ago

At the same time the outages happened I noticed cloudflare also had reported issues with HTTP/3 and WARP. Related?