7 ms·

With all of these outages, you have to wonder if it's a lack of skilled engineers on their part, or if they simply don't have enough GPUs to keep the lights on

With all of these outages, you have to wonder if it's a lack of skilled engineers on their part, or if they simply don't have enough GPUs to keep the lights on all the time.

cloudking 4y ago
It's a reflection of demand, I imagine ChatGPT is still growing with the media attention + word of mouth. I've referred at least 30 friends to try it out, and many are still using it.
- wyre 4y ago
  ChatGPT is currently the 4th most read page on English Wikipedia right now, earlier I noticed it was 2nd. This is the first time I’ve noticed it on their ranking list.
bleuuuu 4y ago
[dead]
KyeRussell 4y ago
Computers are hard.
pm90 4y ago
A friend that interviewed with them shared that they were a bit too smug about “running at google scale with much leaner but highly skilled staff” (paraphrasing here). Perhaps it starts becoming clear why you need that kind of staffing…
thundergolfer 4y ago
The have first-class platform and systems engineers. Do you think one of the most desireable and well funded software shops in the world wouldn’t be able to get skilled engineers?
- lmm 4y ago
  If they're that well funded, skilled engineers are more likely to be the bottleneck than compute availability.
  - dontreact 4y ago
    Serving a request to one of their APIs requires orders of magnitude more compute than your typical web service
    - la64710 4y ago
      Really ? How do you know? Have they shared any credible data around it? Based on experience of BERT , yes maybe to get the best experience or to serve millions of users you need to run any model on compute intensive infrastructure , BUT if you just want to run for yourself and do some small testing you can very well download it from huggingface and elsewhere and run it on your laptop.
      - Rebelgecko 4y ago
        I've tried a few flavors of GPT on my laptop and a single request used orders of magnitude more CPU and RAM than running a "SELECT * FROM WHEREVER" SQL query
      - TOMDM 4y ago
        Pretty much any laptop would take hours to get anything significant done with GPT3 as the model would need to be batched in and out of memory from disk. The amount of memory required to run these models is immense. If you want a comparison, try running the largest version of the open source BLOOM model yourself.
      - localhost 4y ago
        GPT3 is ~175B parameters. At float16 precision, that's 350GB of weights. BLOOM-176B is about the same size. Here's one person's experience; a token is ~0.75 words. "The Python code in this tutorial generates one token every 3 minutes on a computer with an i5 11gen processor, 16GB of RAM, and a Samsung 980 PRO NVME..." [1] https://towardsdatascience.com/run-bloom-the-largest-open-access-ai-model-on-your-desktop-computer-f48e1e2a9a32 https://towardsdatascience.com/run-bloom-the-largest-open-ac...
ygouzerh 4y ago
Knowing all the increase in term of load that they are having these two last months, 99.53 % of uptime on three months is actually quite impressive.
tstrimple 4y ago
In every company I've worked at that does ML work, the data scientists building the models don't have a strong software delivery background. They aren't building a production system with fail safes and redundancy. Their focus is the model itself. Consequently, when they stand up services in front of these models they are not even close to "production grade". I'd hope that a company focused on ML (versus an enterprise just dipping their toes into ML) would have solved these problems, but I wonder if it's related.
d2049 4y ago
Many companies have skilled engineers and enough hardware, and still have outages. It's an expected part of running any software service. Outages can be minimized, but the idea that they can be eliminated with enough people and computing resources is not how things work, from my experience.

With all of these outages, you have to wonder if it's a lack of skilled engineers on their part, or if they simply don't have enough GPUs to keep the lights on all the time.

cloudking 4y ago

It's a reflection of demand, I imagine ChatGPT is still growing with the media attention + word of mouth. I've referred at least 30 friends to try it out, and many are still using it.

wyre 4y ago
ChatGPT is currently the 4th most read page on English Wikipedia right now, earlier I noticed it was 2nd. This is the first time I’ve noticed it on their ranking list.

bleuuuu 4y ago

[dead]

KyeRussell 4y ago

Computers are hard.

pm90 4y ago

A friend that interviewed with them shared that they were a bit too smug about “running at google scale with much leaner but highly skilled staff” (paraphrasing here). Perhaps it starts becoming clear why you need that kind of staffing…

thundergolfer 4y ago

The have first-class platform and systems engineers. Do you think one of the most desireable and well funded software shops in the world wouldn’t be able to get skilled engineers?

lmm 4y ago
If they're that well funded, skilled engineers are more likely to be the bottleneck than compute availability.
- dontreact 4y ago
  Serving a request to one of their APIs requires orders of magnitude more compute than your typical web service
  - la64710 4y ago
    Really ? How do you know? Have they shared any credible data around it? Based on experience of BERT , yes maybe to get the best experience or to serve millions of users you need to run any model on compute intensive infrastructure , BUT if you just want to run for yourself and do some small testing you can very well download it from huggingface and elsewhere and run it on your laptop.
    - Rebelgecko 4y ago
      I've tried a few flavors of GPT on my laptop and a single request used orders of magnitude more CPU and RAM than running a "SELECT * FROM WHEREVER" SQL query
    - TOMDM 4y ago
      Pretty much any laptop would take hours to get anything significant done with GPT3 as the model would need to be batched in and out of memory from disk. The amount of memory required to run these models is immense. If you want a comparison, try running the largest version of the open source BLOOM model yourself.
    - localhost 4y ago
      GPT3 is ~175B parameters. At float16 precision, that's 350GB of weights. BLOOM-176B is about the same size. Here's one person's experience; a token is ~0.75 words. "The Python code in this tutorial generates one token every 3 minutes on a computer with an i5 11gen processor, 16GB of RAM, and a Samsung 980 PRO NVME..." [1] https://towardsdatascience.com/run-bloom-the-largest-open-access-ai-model-on-your-desktop-computer-f48e1e2a9a32 https://towardsdatascience.com/run-bloom-the-largest-open-ac...

ygouzerh 4y ago

Knowing all the increase in term of load that they are having these two last months, 99.53 % of uptime on three months is actually quite impressive.

tstrimple 4y ago

In every company I've worked at that does ML work, the data scientists building the models don't have a strong software delivery background. They aren't building a production system with fail safes and redundancy. Their focus is the model itself. Consequently, when they stand up services in front of these models they are not even close to "production grade". I'd hope that a company focused on ML (versus an enterprise just dipping their toes into ML) would have solved these problems, but I wonder if it's related.

d2049 4y ago

Many companies have skilled engineers and enough hardware, and still have outages. It's an expected part of running any software service. Outages can be minimized, but the idea that they can be eliminated with enough people and computing resources is not how things work, from my experience.