3 ms·
We don't know anything about these outages and their nature. Maybe OpenAI outage was caused by a typo somewhere in code and Google had rolled out system wide mo
by lossolo 4y ago
We don't know anything about these outages and their nature. Maybe OpenAI outage was caused by a typo somewhere in code and Google had rolled out system wide monitoring + SQL change that had cascading issues and required many not obvious steps and processing hundrerds of TBs of data.
I mean I had outages that were 60 seconds long with magnitude less people than OpenAI and serving more req/s than they do. What does it tell you? Nothing at all.
- brookst 4y agoInternal people should care about why, but customers/users really don’t. We expect openai / Google to take steps to balance likelihood of outage with impact of outage, and plan accordingly. So what this tells us, based on very limited data, is that this level out outage happens to even the biggest of companies.
- lossolo 4y agoOpenAI whole infra is magnitudes less complex compared to Google cloud and their infra, OpenAI is not creating cloud for thousands/millions of customers with different use cases, they use Azure to host their software, that's why I find that comparison irrelevant because it's comparing apples to oranges. > plan accordingly You can't plan for every major issue unless you solved the halting problem and unless you as customer want to pay magnitudes more for services you use. For every major client facing issue you probably have 10s/100s/1000s (depending on the system complexity) incidents that were prevented and you know nothing about.