5 ms·

Ok I guess the team with the largest LLM workload in the world and billions in funding won't understand how to optimise a chip for the exact workload they have

Ok I guess the team with the largest LLM workload in the world and billions in funding won't understand how to optimise a chip for the exact workload they have and near future ones.

jrflowers 3y ago
Exactly. Present success means the ability to forecast what’s needed for future success — see the Pierce-Arrow Motor Car Company and their dominance in the market to this very day
- financypants 3y ago
  This person is not saying success -> more success. I think they’re just pointing out that Altman is smart and is surrounded by smart people and a company that understands the demand because they make up the majority of the demand (and they have a strong thesis).
  - startupsfail 3y ago
    Is he raising for OpenAI or for another venture? If he is using deep knowledge from OpenAI to raise money for another venture, this sounds wrong.
    - Atotalnoob 3y ago
      He is rich and powerful, of course it isn’t wrong /s
      - startupsfail 3y ago
        Or broke and powerful? Because of spending a fortune on WorldCoin, working at a nonprofit and heavily investing into early AI startups?
  - deleted 3y ago
    [deleted]
  - aorloff 3y ago
    No way OpenAI makes up even a plurality of chip demand
    - financypants 3y ago
      Perhaps. I have no idea and am not purporting to know.
    - latency-guy2 3y ago
      OpenAI not itself, but Microsoft is. For 2022 and 2023, Microsoft bought a significant portion of NVIDIA's available hardware. They spent quite a bit of 2023 trying to figure out how to even power the multiple fleets of GPUs. Just now with the mild to expected wild adoption of Azure OpenAI are they getting around to servicing all their (potential) customers.
      - ffsm8 3y ago
        [citation needed] Seriously, this is am outlandish claim just from looking at Microsoft and Nvidias market cap. I am sure that Microsoft is gonna be one of Nvidias largest customers, but I sincerely doubt it's even a double digit percentage of their revenue.
        latency-guy2 3y ago
        All of this is public information, its estimated Microsoft bought ~150k H100s from old reports, we also know today that Meta actually bought 500k units To reach double digit revenue of NVIDIA's 2023 at $26.97 billion, you'd only need to hit ~$2.7B in sales. H100's are priced anywhere between $20k - $35k, so required to purchase ~77k - ~135k units. That is singularly H100s, Microsoft also offers lower compute, and they have the rest of Azure to service with a variety of solutions. Being at #1 or #2 market cap worldwide is not a farfetched position to be a significant controller of chips, especially since they directly work in the space as a platform.
      - verdverm 3y ago
        This ignores Google's in house chips and their internal usage. They've been at this much longer. I doubt we have the visibility to know how they compare in terms of available flops and the unit costs
- highfrequency 3y ago
  Can you elaborate on this? Per ChatGPT: > Using Pierce-Arrow Motor Car Company as an example of such success is historically inaccurate. Pierce-Arrow was an American automobile manufacturer based in Buffalo, New York, which was known for producing luxury cars. It was indeed a dominant and prestigious brand in the early 20th century. However, the company did not manage to maintain its success and ultimately failed to adapt to changing market conditions. It faced financial difficulties during the Great Depression and eventually went bankrupt in 1938. Pierce-Arrow's inability to forecast and adapt to the economic changes and shifts in consumer preferences of the time led to its decline.
  - heromal 3y ago
    From the very answer ChatGPT gave you, it's evident that GP is saying that current success does not imply future success, using that company as an example. What needs elaboration?
    - highfrequency 3y ago
      It's pretty clear he is trying to make the opposite point, see "dominance in this market to this very day"
hashxyz 3y ago
The workload they have is already optimized for something like an Nvidia GPU.
- xyzzy123 3y ago
  I apologise if my response was a little snarky. Even granted that OpenAI are not able to build a chip that is competitive with NVidia's latest GPUs for running LLMs right away (which is an opinion - not backed by any direct evidence, but I agree that it is plausible as they are going up against a lot of prior R&D) is it not possible that: a) The unit economics could be so much better that the result is still a major win, e.g. 50% of the performance at 20% of the price. b) OpenAI is decoupled from existing supply constraints and is able to grow faster and deliver more value. A "worse" chip that you can actually get (in insane volume) may be strategically better than a "superior" chip that is limiting your growth. c) That the plan might include some elements you are not expecting - at the $trillions investment level they might be looking at doing some surprising things e.g. (I am just making this up but there are a lot of possibilities) buy a memory manufacturer and work directly on increasing memory bandwidth.
  - LamaOfRuin 3y ago
    From a lay observer point of view of the semiconductor industry of the last two decades, it seems entirely implausible they could do that quickly without just buying a company that was already working on it. And then, unless that company was big enough to already have a significant defensive patent portfolio, it's likely their efforts would be stymied in court for years if it was remotely successful. The idea that even with expertise, the wins would be so much over what other companies that have hired/bought these companies have been designing for the last 10 years based on very similar requirements (the ones that wrote so much of the foundational research) also seems implausible. c) It's not actually possible to plan investments at that level with anything more than a very vague direction you're aiming. If it is long term, then everything is changing in unpredictable ways before you get even 25% there, but if you throw so much money at the problem in order to try to solve it much more quickly you are disrupting global economic and geopolitical forces in ways that also can't be planned for.
parl_match 3y ago
Correct. They're an LLM team, not chip designers.
- __loam 3y ago
  "I know how machine learning and statistical computing works, therefore I am an expert in hardware design" fallacy.
  - aik 3y ago
    I am guessing an incredibly talented team that is incredibly networked and incredibly well funded and proven agile in the tech hub of the world can find hardware experts. Don’t know why anyone would bet against that.
    - kurthr 3y ago
      We would have heard if they had hired/bought the size of team necessary to design a system large enough to be a significant impact. Modern (eve sub 28nm much less 2nm) design is hugely complex and the range of things that an AI compute engine needs to do are very broad. Perhaps they could design a core and license it out? I'm trying to come up with a way they can do something significant without 100 people. Just the memory and serial connections are complex enough ignoring the GPU or heat/power issues.
    - __loam 3y ago
      It took apple like 10 years to go from their first chips to actually using them in laptops, and they are literally the most well capitalized company on the planet. Sorry if I'm skeptical that some relative up starts with a billion in compute from Microsoft can compete with trillion dollar companies that have been around for decades.
    - parl_match 3y ago
      I wouldn't bet against it but that approach has a remarkably low rate of success. We hear about the winners - survivorship bias is real.
    - 3y ago
jszymborski 3y ago
While vertical integration is a great boon for a company, it's hard to pull off. Being an expert in industry X doesn't mean you'll do great in industry Y, even if they are complementary. Training and designing LLMs doesn't mean you understand the semiconductors business.
- sangnoir 3y ago
  Vertical Integration? It may not be an OpenAI project, going by the reporting when he was ousted. I wont be surprised if the plans are for a Muskian incestuous/I-swear-its-not-self-dealing setup, wirh Altman being the CEO of both entities
pas 3y ago
It makes sense to ASIC-ify the thing to get lower latencies and make the whole thing cheaper, so MS can run GPT-(n+1) cheaper. But this bet only pays off if the LLM industry gets into the mature stage where costs dominate, not innovation.

Ok I guess the team with the largest LLM workload in the world and billions in funding won't understand how to optimise a chip for the exact workload they have and near future ones.

jrflowers 3y ago

Exactly. Present success means the ability to forecast what’s needed for future success — see the Pierce-Arrow Motor Car Company and their dominance in the market to this very day

financypants 3y ago
This person is not saying success -> more success. I think they’re just pointing out that Altman is smart and is surrounded by smart people and a company that understands the demand because they make up the majority of the demand (and they have a strong thesis).
- startupsfail 3y ago
  Is he raising for OpenAI or for another venture? If he is using deep knowledge from OpenAI to raise money for another venture, this sounds wrong.
  - Atotalnoob 3y ago
    He is rich and powerful, of course it isn’t wrong /s
    - startupsfail 3y ago
      Or broke and powerful? Because of spending a fortune on WorldCoin, working at a nonprofit and heavily investing into early AI startups?
- deleted 3y ago
  [deleted]
- aorloff 3y ago
  No way OpenAI makes up even a plurality of chip demand
  - financypants 3y ago
    Perhaps. I have no idea and am not purporting to know.
  - latency-guy2 3y ago
    OpenAI not itself, but Microsoft is. For 2022 and 2023, Microsoft bought a significant portion of NVIDIA's available hardware. They spent quite a bit of 2023 trying to figure out how to even power the multiple fleets of GPUs. Just now with the mild to expected wild adoption of Azure OpenAI are they getting around to servicing all their (potential) customers.
    - ffsm8 3y ago
      [citation needed] Seriously, this is am outlandish claim just from looking at Microsoft and Nvidias market cap. I am sure that Microsoft is gonna be one of Nvidias largest customers, but I sincerely doubt it's even a double digit percentage of their revenue.
      - latency-guy2 3y ago
        All of this is public information, its estimated Microsoft bought ~150k H100s from old reports, we also know today that Meta actually bought 500k units To reach double digit revenue of NVIDIA's 2023 at $26.97 billion, you'd only need to hit ~$2.7B in sales. H100's are priced anywhere between $20k - $35k, so required to purchase ~77k - ~135k units. That is singularly H100s, Microsoft also offers lower compute, and they have the rest of Azure to service with a variety of solutions. Being at #1 or #2 market cap worldwide is not a farfetched position to be a significant controller of chips, especially since they directly work in the space as a platform.
    - verdverm 3y ago
      This ignores Google's in house chips and their internal usage. They've been at this much longer. I doubt we have the visibility to know how they compare in terms of available flops and the unit costs
highfrequency 3y ago
Can you elaborate on this? Per ChatGPT: > Using Pierce-Arrow Motor Car Company as an example of such success is historically inaccurate. Pierce-Arrow was an American automobile manufacturer based in Buffalo, New York, which was known for producing luxury cars. It was indeed a dominant and prestigious brand in the early 20th century. However, the company did not manage to maintain its success and ultimately failed to adapt to changing market conditions. It faced financial difficulties during the Great Depression and eventually went bankrupt in 1938. Pierce-Arrow's inability to forecast and adapt to the economic changes and shifts in consumer preferences of the time led to its decline.
- heromal 3y ago
  From the very answer ChatGPT gave you, it's evident that GP is saying that current success does not imply future success, using that company as an example. What needs elaboration?
  - highfrequency 3y ago
    It's pretty clear he is trying to make the opposite point, see "dominance in this market to this very day"

hashxyz 3y ago

The workload they have is already optimized for something like an Nvidia GPU.

xyzzy123 3y ago
I apologise if my response was a little snarky. Even granted that OpenAI are not able to build a chip that is competitive with NVidia's latest GPUs for running LLMs right away (which is an opinion - not backed by any direct evidence, but I agree that it is plausible as they are going up against a lot of prior R&D) is it not possible that: a) The unit economics could be so much better that the result is still a major win, e.g. 50% of the performance at 20% of the price. b) OpenAI is decoupled from existing supply constraints and is able to grow faster and deliver more value. A "worse" chip that you can actually get (in insane volume) may be strategically better than a "superior" chip that is limiting your growth. c) That the plan might include some elements you are not expecting - at the $trillions investment level they might be looking at doing some surprising things e.g. (I am just making this up but there are a lot of possibilities) buy a memory manufacturer and work directly on increasing memory bandwidth.
- LamaOfRuin 3y ago
  From a lay observer point of view of the semiconductor industry of the last two decades, it seems entirely implausible they could do that quickly without just buying a company that was already working on it. And then, unless that company was big enough to already have a significant defensive patent portfolio, it's likely their efforts would be stymied in court for years if it was remotely successful. The idea that even with expertise, the wins would be so much over what other companies that have hired/bought these companies have been designing for the last 10 years based on very similar requirements (the ones that wrote so much of the foundational research) also seems implausible. c) It's not actually possible to plan investments at that level with anything more than a very vague direction you're aiming. If it is long term, then everything is changing in unpredictable ways before you get even 25% there, but if you throw so much money at the problem in order to try to solve it much more quickly you are disrupting global economic and geopolitical forces in ways that also can't be planned for.

parl_match 3y ago

Correct. They're an LLM team, not chip designers.

__loam 3y ago
"I know how machine learning and statistical computing works, therefore I am an expert in hardware design" fallacy.
- aik 3y ago
  I am guessing an incredibly talented team that is incredibly networked and incredibly well funded and proven agile in the tech hub of the world can find hardware experts. Don’t know why anyone would bet against that.
  - kurthr 3y ago
    We would have heard if they had hired/bought the size of team necessary to design a system large enough to be a significant impact. Modern (eve sub 28nm much less 2nm) design is hugely complex and the range of things that an AI compute engine needs to do are very broad. Perhaps they could design a core and license it out? I'm trying to come up with a way they can do something significant without 100 people. Just the memory and serial connections are complex enough ignoring the GPU or heat/power issues.
  - __loam 3y ago
    It took apple like 10 years to go from their first chips to actually using them in laptops, and they are literally the most well capitalized company on the planet. Sorry if I'm skeptical that some relative up starts with a billion in compute from Microsoft can compete with trillion dollar companies that have been around for decades.
  - parl_match 3y ago
    I wouldn't bet against it but that approach has a remarkably low rate of success. We hear about the winners - survivorship bias is real.
  - 3y ago

jszymborski 3y ago

While vertical integration is a great boon for a company, it's hard to pull off. Being an expert in industry X doesn't mean you'll do great in industry Y, even if they are complementary. Training and designing LLMs doesn't mean you understand the semiconductors business.

sangnoir 3y ago
Vertical Integration? It may not be an OpenAI project, going by the reporting when he was ousted. I wont be surprised if the plans are for a Muskian incestuous/I-swear-its-not-self-dealing setup, wirh Altman being the CEO of both entities

pas 3y ago

It makes sense to ASIC-ify the thing to get lower latencies and make the whole thing cheaper, so MS can run GPT-(n+1) cheaper. But this bet only pays off if the LLM industry gets into the mature stage where costs dominate, not innovation.