5 ms·

To put this in perspective: the outage was 52 min. The total number of employees OpenAI have are 375. The launch date was just some months ago. At Google, the

To put this in perspective: the outage was 52 min. The total number of employees OpenAI have are 375. The launch date was just some months ago. At Google, the Cloud SQL dashboard was unavailable for around 12 hours a couple of weeks ago if I read this correctly: https://status.cloud.google.com/incidents/xg2qrL1UuSJiPDZALJpu https://status.cloud.google.com/incidents/xg2qrL1UuSJiPDZALJ... The total number of employees of Google are 156 500. Google Cloud was launched 2008. So when people say scaling is hard... it's not a solved problem and you shouldn't be surprised when these things happen. A solid theoretical foundation and more testing is better of course.

jeffbee 4y ago
That's a completely bizarre comparison. No number of Android developers will lead to higher availability of Cloud SQL observability. The gross number of employees at an org is meaningless.
- danielscrubs 4y ago
  Sure. But many Android developers are likely to use Cloud SQL and give quick feedback. How big the benefits are is let as an exercise to the reader.
  - verdverm 4y ago
    They have automated monitoring that would alert SREs before a support ticket likely would. Do you really think someone working on an OS would be using Cloud SQL regularly to notice? Maybe a developer of an Android app is (1) using Cloud SQL (over firestore?) and (2) has monitoring that would alert them effectively and (3) determine the cause is Google and send a support ticket. As an exercise, which is likely to reach the SRE first?
    - danielscrubs 4y ago
      > They have automated monitoring that would alert SREs before a support ticket likely would. Yes, but how likely is this unlikely situation? It took 12 hours to debug and fix after all.
      - verdverm 4y ago
        It was the dashboard, applications depending on the database service were not impacted. You're comparing apples and oranges and ignoring both a much larger context and the nuances
        danielscrubs 4y ago
        And most of those customers are paying customers? All I'm saying is that some outages are to be expected from a young company and they are doing excellent work.
        verdverm 4y ago
        Are they doing excellent work? We all have raved about various startups and how great they were, until the externalities became widely understood. Is there something that excludes OpenAI from this possibility? My near term concern is that these language models are going to cause a host of issues for society. Botnets, misinformation, students using for homework, influence operations... I believe trust in information is a bigger concern than global warming. As an aside, what is OpenAI's relative contribution to CO2 emissions compared with other companies of similar size?
  - riku_iki 4y ago
    > Android developers are likely to use Cloud SQL "likely" is your hypothesis. They are two different orgs, and internal services can use different infra for analysis.
- xdavidliu 4y ago
  presumably the data scientists at OpenAI would also not be able to directly lead to higher availability of ChatGPT; it's just an order of magnitude comparison.
  - verdverm 4y ago
    Depends on the reason for the outage. If it is an application error, like OOM or other performance issue, then yes, they might be directly tied to the root cause and solution. If it's an infra issue, then maybe not so much. I once spent an entire weekend swapping shifts trying to keep the main server of an AI company up because the devs deployed on friday and were unreachable to fix their memory error. The best thing that happened is the boss came in Monday and froze feature development for a sprint, inventing the "tech debt sprint" that became a quarterly activity. Everyone loved it, uptime became better, and even application development got easier. Size of a company doesn't matter, the magnitude of product offerings aligns with head count. Team size is what matters and is likely a much closer ratio between the companies. Google does have head count and dedicated teams to solve common problems and build solutions at the platform layer. Likely OpenAPI is benefiting from Google's efforts here (from the open source and knowledge sharing which are bar none imho). From an OpenAPI job opening: > The Engineering team wraps a massive fleet of GPUs in scalable, robust, infrastructure powered by Kubernetes, Go, Python, Terraform, Kafka, Postgres, and Snowflake. Our APIs are powered by Python Flask and OpenAPI with a React frontend.
    - htrp 4y ago
      I get everything except Snowflake.... Anyone care to elaborate on why they have snowflake in their stack?
      - disgruntledphd2 4y ago
        For data transformations and analysis, because snowflake is a pretty rocking SQL database if you're cloud hosted.
verdverm 4y ago
How Google, and most likely all businesses, should think about outages is more complex than being up or down. Is it just part of the product, like Slack threads yesterday? Is it 1/2 or 1/1M requests or operations? The Google SRE book is an excellent work: https://sre.google/sre-book/introduction/ https://sre.google/sre-book/introduction/ Also some perspective, if you look at the OpenAPI outages for Jan 2023, the up time is worse than Google Cloud SQL. https://status.openai.com/uptime https://status.openai.com/uptime
- danielscrubs 4y ago
  Agreed, Google's public incident reports could be improved with those numbers. Some chapters in the book include "Being On-Call", "Effective Troubleshooting" & "Postmortem Culture: Learning from Failure". Which signals to me, a normal work culture and time will improve stability in most companies.
kerpotgh 4y ago
[dead]
lossolo 4y ago
We don't know anything about these outages and their nature. Maybe OpenAI outage was caused by a typo somewhere in code and Google had rolled out system wide monitoring + SQL change that had cascading issues and required many not obvious steps and processing hundrerds of TBs of data. I mean I had outages that were 60 seconds long with magnitude less people than OpenAI and serving more req/s than they do. What does it tell you? Nothing at all.
- brookst 4y ago
  Internal people should care about why, but customers/users really don’t. We expect openai / Google to take steps to balance likelihood of outage with impact of outage, and plan accordingly. So what this tells us, based on very limited data, is that this level out outage happens to even the biggest of companies.
  - lossolo 4y ago
    OpenAI whole infra is magnitudes less complex compared to Google cloud and their infra, OpenAI is not creating cloud for thousands/millions of customers with different use cases, they use Azure to host their software, that's why I find that comparison irrelevant because it's comparing apples to oranges. > plan accordingly You can't plan for every major issue unless you solved the halting problem and unless you as customer want to pay magnitudes more for services you use. For every major client facing issue you probably have 10s/100s/1000s (depending on the system complexity) incidents that were prevented and you know nothing about.
riku_iki 4y ago
> The total number of employees OpenAI have are 375 they outsource infra to MS?
naet 4y ago
Not sure what "perspective" you are trying to put this in by offering these comparison points between OpenAI and Google. Are you trying to say that OpenAI is more reliable than Google? For additional perspective, OpenAI has a very small number of highly similar offerings (neural net with API access), while Google has a huge host of very different offerings, from web indexing and search, to email, to file hosting, to video streaming, to cloud compute... etc. Google also has a vastly larger user pool by at least a couple orders of magnitude. Google's core services are extremely solid even at ridiculous scale and have few outages, any of which would be considered major news. Google also operates all this at a profit, while OpenAI works at a deficit. Google has had to scale larger than nearly any other service while maintaining profitability while OpenAI is more or less free to throw more compute power to solve problems, at any cost, to build valuation. Google has written entire programming languages to help them keep up at an unprecedented scale. Comparing a single minor Google offering going down to an OpenAI outage isn't a fair comparison to either company. Yes, Google has, by your numbers, about 400 times the number of OpenAI employees. I'd be willing to bet that a single large Google service like YouTube handles more than 400 times the amount of compute, data, and traffic that OpenAI does. I wouldn't draw a comparison between the size and efficacy of the employees, but again, Google operates at a very different scale, and has operated at that scale very well. Also anecdotally, half the time I've tried to use ChatGPT it's "at capacity" or throws an internal error, and I've also seen Dall-E unavailable even though I've barely tried to use it, so I wouldn't say that OpenAI service has been ironclad this whole time.
IMTDb 4y ago
To put in perspective, my hello world service has less than 1 employee working on it, has been launched 10 years ago and has roughly 5 minutes of downtime per year when I restart the server. I am available for hire to teach a thing or two to Google (and OpenAI). /s
ironick09 4y ago
The is the worst comparison in the history of comparisons.
nindalf 4y ago
> The total number of employees of Google are 156 500 This is a non-sequitur. This statement has nothing to do with the outage. It’s just thrown in there to pad the number of words in the comment.

jeffbee 4y ago

That's a completely bizarre comparison. No number of Android developers will lead to higher availability of Cloud SQL observability. The gross number of employees at an org is meaningless.

danielscrubs 4y ago
Sure. But many Android developers are likely to use Cloud SQL and give quick feedback. How big the benefits are is let as an exercise to the reader.
- verdverm 4y ago
  They have automated monitoring that would alert SREs before a support ticket likely would. Do you really think someone working on an OS would be using Cloud SQL regularly to notice? Maybe a developer of an Android app is (1) using Cloud SQL (over firestore?) and (2) has monitoring that would alert them effectively and (3) determine the cause is Google and send a support ticket. As an exercise, which is likely to reach the SRE first?
  - danielscrubs 4y ago
    > They have automated monitoring that would alert SREs before a support ticket likely would. Yes, but how likely is this unlikely situation? It took 12 hours to debug and fix after all.
    - verdverm 4y ago
      It was the dashboard, applications depending on the database service were not impacted. You're comparing apples and oranges and ignoring both a much larger context and the nuances
      - danielscrubs 4y ago
        And most of those customers are paying customers? All I'm saying is that some outages are to be expected from a young company and they are doing excellent work.
        verdverm 4y ago
        Are they doing excellent work? We all have raved about various startups and how great they were, until the externalities became widely understood. Is there something that excludes OpenAI from this possibility? My near term concern is that these language models are going to cause a host of issues for society. Botnets, misinformation, students using for homework, influence operations... I believe trust in information is a bigger concern than global warming. As an aside, what is OpenAI's relative contribution to CO2 emissions compared with other companies of similar size?
- riku_iki 4y ago
  > Android developers are likely to use Cloud SQL "likely" is your hypothesis. They are two different orgs, and internal services can use different infra for analysis.
xdavidliu 4y ago
presumably the data scientists at OpenAI would also not be able to directly lead to higher availability of ChatGPT; it's just an order of magnitude comparison.
- verdverm 4y ago
  Depends on the reason for the outage. If it is an application error, like OOM or other performance issue, then yes, they might be directly tied to the root cause and solution. If it's an infra issue, then maybe not so much. I once spent an entire weekend swapping shifts trying to keep the main server of an AI company up because the devs deployed on friday and were unreachable to fix their memory error. The best thing that happened is the boss came in Monday and froze feature development for a sprint, inventing the "tech debt sprint" that became a quarterly activity. Everyone loved it, uptime became better, and even application development got easier. Size of a company doesn't matter, the magnitude of product offerings aligns with head count. Team size is what matters and is likely a much closer ratio between the companies. Google does have head count and dedicated teams to solve common problems and build solutions at the platform layer. Likely OpenAPI is benefiting from Google's efforts here (from the open source and knowledge sharing which are bar none imho). From an OpenAPI job opening: > The Engineering team wraps a massive fleet of GPUs in scalable, robust, infrastructure powered by Kubernetes, Go, Python, Terraform, Kafka, Postgres, and Snowflake. Our APIs are powered by Python Flask and OpenAPI with a React frontend.
  - htrp 4y ago
    I get everything except Snowflake.... Anyone care to elaborate on why they have snowflake in their stack?
    - disgruntledphd2 4y ago
      For data transformations and analysis, because snowflake is a pretty rocking SQL database if you're cloud hosted.

verdverm 4y ago

How Google, and most likely all businesses, should think about outages is more complex than being up or down. Is it just part of the product, like Slack threads yesterday? Is it 1/2 or 1/1M requests or operations? The Google SRE book is an excellent work: https://sre.google/sre-book/introduction/ https://sre.google/sre-book/introduction/ Also some perspective, if you look at the OpenAPI outages for Jan 2023, the up time is worse than Google Cloud SQL. https://status.openai.com/uptime https://status.openai.com/uptime

danielscrubs 4y ago
Agreed, Google's public incident reports could be improved with those numbers. Some chapters in the book include "Being On-Call", "Effective Troubleshooting" & "Postmortem Culture: Learning from Failure". Which signals to me, a normal work culture and time will improve stability in most companies.

kerpotgh 4y ago

[dead]

lossolo 4y ago

We don't know anything about these outages and their nature. Maybe OpenAI outage was caused by a typo somewhere in code and Google had rolled out system wide monitoring + SQL change that had cascading issues and required many not obvious steps and processing hundrerds of TBs of data. I mean I had outages that were 60 seconds long with magnitude less people than OpenAI and serving more req/s than they do. What does it tell you? Nothing at all.

brookst 4y ago
Internal people should care about why, but customers/users really don’t. We expect openai / Google to take steps to balance likelihood of outage with impact of outage, and plan accordingly. So what this tells us, based on very limited data, is that this level out outage happens to even the biggest of companies.
- lossolo 4y ago
  OpenAI whole infra is magnitudes less complex compared to Google cloud and their infra, OpenAI is not creating cloud for thousands/millions of customers with different use cases, they use Azure to host their software, that's why I find that comparison irrelevant because it's comparing apples to oranges. > plan accordingly You can't plan for every major issue unless you solved the halting problem and unless you as customer want to pay magnitudes more for services you use. For every major client facing issue you probably have 10s/100s/1000s (depending on the system complexity) incidents that were prevented and you know nothing about.

riku_iki 4y ago

> The total number of employees OpenAI have are 375 they outsource infra to MS?

naet 4y ago

Not sure what "perspective" you are trying to put this in by offering these comparison points between OpenAI and Google. Are you trying to say that OpenAI is more reliable than Google? For additional perspective, OpenAI has a very small number of highly similar offerings (neural net with API access), while Google has a huge host of very different offerings, from web indexing and search, to email, to file hosting, to video streaming, to cloud compute... etc. Google also has a vastly larger user pool by at least a couple orders of magnitude. Google's core services are extremely solid even at ridiculous scale and have few outages, any of which would be considered major news. Google also operates all this at a profit, while OpenAI works at a deficit. Google has had to scale larger than nearly any other service while maintaining profitability while OpenAI is more or less free to throw more compute power to solve problems, at any cost, to build valuation. Google has written entire programming languages to help them keep up at an unprecedented scale. Comparing a single minor Google offering going down to an OpenAI outage isn't a fair comparison to either company. Yes, Google has, by your numbers, about 400 times the number of OpenAI employees. I'd be willing to bet that a single large Google service like YouTube handles more than 400 times the amount of compute, data, and traffic that OpenAI does. I wouldn't draw a comparison between the size and efficacy of the employees, but again, Google operates at a very different scale, and has operated at that scale very well. Also anecdotally, half the time I've tried to use ChatGPT it's "at capacity" or throws an internal error, and I've also seen Dall-E unavailable even though I've barely tried to use it, so I wouldn't say that OpenAI service has been ironclad this whole time.

IMTDb 4y ago

To put in perspective, my hello world service has less than 1 employee working on it, has been launched 10 years ago and has roughly 5 minutes of downtime per year when I restart the server. I am available for hire to teach a thing or two to Google (and OpenAI). /s

ironick09 4y ago

The is the worst comparison in the history of comparisons.

nindalf 4y ago

> The total number of employees of Google are 156 500 This is a non-sequitur. This statement has nothing to do with the outage. It’s just thrown in there to pad the number of words in the comment.