6 ms·

According to Jensen it takes about 8000 H100s running for 90 days to train a 1.8 Trillion param MoE GPT-4 scale model. Meta has about 350,000 of these GPUs and

According to Jensen it takes about 8000 H100s running for 90 days to train a 1.8 Trillion param MoE GPT-4 scale model. Meta has about 350,000 of these GPUs and a whole bunch of A100s. This means the ability to train 50 GPT-4 scale models every 90 days or 200 such models per year. This level of overkill suggests to me that the core models will be commoditized to oblivion, making the actual profit margins from AI-centric companies close to 0, especially if Microsoft and Meta keep giving away these models for free. This is actually terrible for investors, but amazing for builders (ironically). The real value methinks is actually over the control of proprietary data used for training which is the single most important factor for model output quality. And this is actually as much an issue for copyright lawyers rather than software engineers once the big regulatory hammers start dropping to protect American workers.

VirusNewbie 2y ago
Llama isn't even in the same stratosphere as the big models when it comes to coding, logic, and other interesting tasks that I think are commercially viable.
laweijfmvo 2y ago
A lot of those GPUs are for their 3B users to run inferencing, no?
- LarsDu88 2y ago
  True that, but I think in a very short amount of time, using dedicated general purpose GPUs just for inferencing is going to be mega overkill. If there's dedicated inferencing silicon (like say the thing created by Groq), all those GPUs will be power sucking liabilities, and then the REAL singularity superintelligence level training can begin.
  - campers 2y ago
    Etched is another dedicated inference hardware company that recently announced their product. It only works for transformer based models, but is ~20x faster than a H100
- benreesman 2y ago
  It’s been a very long time since I had any inside baseball, but I very much doubt that Hopper gear is in the hot inference path. The precisions and mantissa/exponent ratios you want for inference are just different to a mixed-precision, fault tolerant, model and data parallel pipeline. Hopper is for training mega-huge attention decoders: TF32, bfloat16, hot paths to the SRAM end of the cache hierarchy with cache coherency semantics that you can reason about. Parity gear for fault tolerance, it’s just a different game.
Kon-Peki 2y ago
It's like someone thinking that they are SOOOO smart, they are going to get rich selling shovels in the gold rush. So they overpay for the land, they overpay for the factory, they overpay for their sales staff. And then someone else starts giving away shovels for free.
- Terr_ 2y ago
  > And then someone else starts giving away shovels for free. And their business model is shovel-fleet logistics and maintenance... :p
  - TeMPOraL 2y ago
    And/or exploiting the legal infrastructure around intellectual property rights to make sure only hobbyists and geologists can use the shovels without paying through the nose or getting sued into oblivion.
    - freehorse 2y ago
      If your company grows to 700 million monthly active users, then most probably you can make your own AI department and train your own models. I guess people's aspirations are very high in this space, but let's be realistic.
  - woah 2y ago
    The platform for shovel fleet logistics startups
    - ugh123 2y ago
      SaaS (shoveling as a service)
  - jona-f 2y ago
    Their business model is of course tracking all the shovels and then selling the locations of all the gold.
- 2y ago
apitman 2y ago
I thought AI is supposed to put all the lawyers out of work.
- threeseed 2y ago
  Lexis+ AI and Ask Practical Law AI systems produced incorrect information more than 17% of the time, while Westlaw’s AI-Assisted Research hallucinated more than 34% of the time: https://hai.stanford.edu/news/ai-trial-legal-models-hallucinate-1-out-6-or-more-benchmarking-queries https://hai.stanford.edu/news/ai-trial-legal-models-hallucin...
  - falcor84 2y ago
    Just out of curiosity, what's the human lawyer baseline on that?
    - csa 2y ago
      > Just out of curiosity, what's the human lawyer baseline on that? Largely depends on how much money the client has.
    - dbish 2y ago
      it's the self-driving car problem. Humans aren't perfect either but people like to ignore that.
      - deleted 2y ago
        [deleted]
      - Terr_ 2y ago
        True, they're similar... But what's also similar is that people make the mistake of focusing on differences in failure rates while glossing over failure modes. Human imperfections are a family of failure-modes which have a gajillion years of experience in detecting, analyzing, preventing, and repairing. Quirks in ML models... not so much. A quick thought-experiment to illustrate the difference: Imagine there's a self-driving car that is exactly half as likely to cause death or injury than a human driver. That's a good failure rate. The twist is that its major failure mode is totally alien, where units attempt to inexplicably chase-murder random pedestrians. It would be difficult to get people to accept that tradeoff.
vasco 2y ago
The infinitely expanding AI-generated metaverse isn't going to render itself, at least in the case of meta I think that might be one of the only pieces missing.
alberth 2y ago
> making the actual profit margins from AI-centric companies close to 0 The same thinking stopped many legacy tech companies from becoming a “cloud” company ~20 years ago. Fast forward to today and the margin for cloud compute is still obscene. And they all wish in hindsight they got into the cloud business way sooner than they ultimately did.
- lmm 2y ago
  Are the second-tier cloud companies really seeing big margins? Why is it not competed away to zero like airlines?
  - mbb70 2y ago
    There is essentially zero cost for a user to switch airlines. The cost to switch clouds is astronomical for any decent sized org.
    - throwaway2037 2y ago
      I like the sentiment of your post. I mostly agree. If you use OpenShift, doesn't that help to reduce the cost of switching cloud providers?
      - coredog64 2y ago
        Not really. What happens when you run a cloud on top of your cloud is that you don’t get to use any of their differentiating features and that winds up costing you money. Plus you have to pay for your own control plane when that’s already baked into the cloud provider’s charge model.
        throwaway2037 2y ago
        > Plus you have to pay for your own control plane when that’s already baked into the cloud provider’s charge model. When you say "control plane" does this mean Kubernetes?
Willish42 2y ago
> once the big regulatory hammers start dropping to protect American workers Have we been living in the same universe the last 10 years? I don't see this ever happening. Related recent news (literally posted yesterday) https://www.axios.com/2024/07/02/chevron-scotus-biden-cyber-regulations https://www.axios.com/2024/07/02/chevron-scotus-biden-cyber-...
- hnthrow289570 2y ago
  The only upside is state-level minimum wage increases. The federal minimum wage is still a complete joke at $7.25 an hour. But there's bigger fish to fry for American politics and worker obsolescence is not really top of mind for anyone.
- LarsDu88 2y ago
  I think people wildly underestimate how protectionist people - particularly educated software engineers and PhDs will get once an AI model directly impacts their source of wealth. Red state blue collar workers got their candidate to pass tariffs. What happens when both blue state white collar workers and red state blue collar workers need to contest with AI. Perhaps not within the next 10 years, but certainly within 20 years! And if you think 20 years is a long time... 2004 was when Halo 2 came out
  - dmix 2y ago
    It's not going to stop it though even if they try though. You can't stop technical progress like this any more than you can stop piracy. But agreed, between the unions with political pull and "AI safety" grifters I suspect there could be some level of regulatory risk, particularly for the megacorps in California. I doubt it will be some national thing in the US absent a major political upheaval. Definitely possible in the EU which will probably just be a price passed on to customers or reduced access, but that's nothing new for them.
    - techostritch 2y ago
      I keep seeing people say you can’t stop progress (social, technical, etc.) but has this really been tested? There seems to be a lot of political upheaval at least being threatened on the near future, and depending on the forces that come into power I imagine they may be willing to do a lot to protect that power. Tucker Carlson at one point said if FSD was going to take away trucking jobs we should stop that with regulation.
shoggouth 2y ago
In my opinion Elsevier and others charging for access to academic publications has held back the advancement of humanity to a lower exponential acceleration into the future at a considerable factor. Think of how cancer could have been cured a decade ago if information was allowed to flow freely from the 50's forward - if anyone could have read scientific publications for free. I have no respect for people that want to protect the moat around information that could be used to advance humanity.
- OJFord 2y ago
  As much as I'd love open access to academic publications and don't think the current model is great: > Think of how cancer could have been cured a decade ago if information was allowed to flow freely from the 50's forward might be a bit fanciful? Unless you're referring to something particular I'm unaware of. The people best equipped and trained to deliver a cure for cancer (and then some, since it tends not to be particularly field-restricred) do have access. I think the loss is more likely in engineering (to the publication's science), cheaper methods, more reliably manufacturable versions of lab prototypes, etc. I doubt there are many people capable of cancer research breakthroughs who don't have access to cancer research, personally. (And to be clear: I'm not capable of it.)
  - enjoylife 2y ago
    I’ll add that even if the papers we all wanted were more freely accesible, the replication and completeness of their described methods would be another source of slowdown.
    - robwwilliams 2y ago
      Main problem is still just getting good quantitative data and metadata. Most biomedical researchers are motivated to “tell stories”. Few of us care about generating huge mineable data sets.
  - instagib 2y ago
    All of the engineering companies I’ve worked for have not paid for IEEE or any journals. I have to go to the library and maintain membership for IEEE myself then request reimbursement. The schools I’ve worked with have access to everything I’ve needed. They didn’t advertise it but it’s also free for students.
naveen99 2y ago
Propriety data is not necessary for training intelligence. Wikipedia, pubmed, arxiv, Reddit and github are probably sufficient. And babies don’t even use that. I agree though that the returns on hardware rapidly diminish.
kristjansson 2y ago
per Zuckerberg[0], ~half of their H100s were for Reels content recommendation: > I think it was because we were working on Reels. We always want to have enough capacity to build something that we can't quite see on the horizon yet. ... So let's order enough GPUs to do what we need to do on Reels and ranking content and feed. But let's also double that. So there's an immense capacity inside Meta, but the _whole_ fleet isn't available for LLM training. [0]: https://www.dwarkeshpatel.com/p/mark-zuckerberg?open=false#§transcript https://www.dwarkeshpatel.com/p/mark-zuckerberg?open=false#§...
- matthewdgreen 2y ago
  Surely they’re using some of that hardware to overcome Apple’s attempts to deprive them of targeted advertising data.
forgot-im-old 2y ago
What percentage of GPUs are being used for training versus inference?
sailingparrot 2y ago
> This means the ability to train 50 GPT-4 scale models every 90 days or 200 such models per year. Not anywhere close to that. Those 350k GPUs you talk about aren't linked together. They also definitely aren't all H100s. To train a GPT-4 scale model you need a single cluster, where all the GPUs are tightly linked together. At the scale of 20k+ GPUs, the price you pay in networking to link those GPUs is basically almost the same as the price of those GPUs themselves. It's really hard and expensive to do. FB has maybe 2 such clusters, not more than that. And I'm somewhat confident one of those cluster is an A100 cluster. So they can train maybe 6 GPT-4 every 90 days.
- LarsDu88 2y ago
  Ok, I might have misread some rumored ballpark figures. And most of the GPUs will be used for inference rather than training. Still 6 GPT-4's every 90 days is pretty amazing.
- LarsDu88 2y ago
  I had to take a second look at this: https://www.datacenterdynamics.com/en/news/meta-to-operate-600000-gpus-by-year-end/ https://www.datacenterdynamics.com/en/news/meta-to-operate-6... 340,000 H100s 600,000 H100 equivalents (perhaps AMD Instinct cards?) On top of the hundreds of thousands of legacy A100s. And I'm certain the order for B100s will be big. Very big. Even the philanthropic org Chan-Zuckerberg institute current rocks 1000 H100s, probably none used for inference. They are going ALL OUT
  - dmix 2y ago
    > Even the philanthropic org Chan-Zuckerberg institute current rocks 1000 H100s, probably none used for inference. What do they use them for?
    - ChainOfFools 2y ago
      tax writeoffs
  - onion2k 2y ago
itkovian_ 2y ago
Everything is if scaling keeps making the models better. If it does you don't train 50 gpt4s, you have the best model.
weitendorf 2y ago
There are more kinds of AI-centric companies than just foundation models. Making that equivalency is akin to equating internet companies during the dotcom bubble with just websites like pets.com. Now one semi-skilled person in a couple days can make websites that entire teams back then would have taken months to build, but that doesn't mean google.com and thefacebook.com are easily commodotized or bad businesses just because they're websites.
boloust 2y ago
Why are we assuming we're topping out at a GPT-4 scale model?
eru 2y ago
> The real value methinks is actually over the control of proprietary data used for training which is the single most important factor for model output quality. Maybe. But we've barely scratched the surface of being more economical with data. I remember back in the old days, there was lots of work on eg dropout and data augmentation etc. We haven't seen too much of that with the like of ChatGPT yet. I'm also curious to see what the future of multimodal models holds: you can create almost arbitrarily amounts of extra data by pointing a webcam at the world, especially when combined with a robot, or letting your models also play StarCraft or Diplomacy against each other.
babyshake 2y ago
> once the big regulatory hammers start dropping to protect American workers. The US Supreme Court seems determined to make sure that big regulatory hammers are not going to be dropping, from what I can tell.
lkdfjlkdfjlg 2y ago
> but amazing for builders (ironically). Could you expand on this? Who are "the builders" here? You mean the model developers? I don't see how this situation can be "amazing" for the builders - developers will just get a wage out of their work.
bufferoverflow 2y ago
> this means the ability to train 50 GPT-4 scale models every 90 days or 200 such models per year. What it actually means is that they are training next gen models that are 50X larger. And, considering MS and OpenAI are planning to build a $100 billion AI training computer, these 350K GPUs is just a tiny portion of what they are planning. This isn't an overkill. This is the current plan: throw as much compute as possible and hope intelligence scales with compute.

VirusNewbie 2y ago

Llama isn't even in the same stratosphere as the big models when it comes to coding, logic, and other interesting tasks that I think are commercially viable.

laweijfmvo 2y ago

A lot of those GPUs are for their 3B users to run inferencing, no?

LarsDu88 2y ago
True that, but I think in a very short amount of time, using dedicated general purpose GPUs just for inferencing is going to be mega overkill. If there's dedicated inferencing silicon (like say the thing created by Groq), all those GPUs will be power sucking liabilities, and then the REAL singularity superintelligence level training can begin.
- campers 2y ago
  Etched is another dedicated inference hardware company that recently announced their product. It only works for transformer based models, but is ~20x faster than a H100
benreesman 2y ago
It’s been a very long time since I had any inside baseball, but I very much doubt that Hopper gear is in the hot inference path. The precisions and mantissa/exponent ratios you want for inference are just different to a mixed-precision, fault tolerant, model and data parallel pipeline. Hopper is for training mega-huge attention decoders: TF32, bfloat16, hot paths to the SRAM end of the cache hierarchy with cache coherency semantics that you can reason about. Parity gear for fault tolerance, it’s just a different game.

Kon-Peki 2y ago

It's like someone thinking that they are SOOOO smart, they are going to get rich selling shovels in the gold rush. So they overpay for the land, they overpay for the factory, they overpay for their sales staff. And then someone else starts giving away shovels for free.

Terr_ 2y ago
> And then someone else starts giving away shovels for free. And their business model is shovel-fleet logistics and maintenance... :p
- TeMPOraL 2y ago
  And/or exploiting the legal infrastructure around intellectual property rights to make sure only hobbyists and geologists can use the shovels without paying through the nose or getting sued into oblivion.
  - freehorse 2y ago
    If your company grows to 700 million monthly active users, then most probably you can make your own AI department and train your own models. I guess people's aspirations are very high in this space, but let's be realistic.
- woah 2y ago
  The platform for shovel fleet logistics startups
  - ugh123 2y ago
    SaaS (shoveling as a service)
- jona-f 2y ago
  Their business model is of course tracking all the shovels and then selling the locations of all the gold.
2y ago

apitman 2y ago

I thought AI is supposed to put all the lawyers out of work.

threeseed 2y ago
Lexis+ AI and Ask Practical Law AI systems produced incorrect information more than 17% of the time, while Westlaw’s AI-Assisted Research hallucinated more than 34% of the time: https://hai.stanford.edu/news/ai-trial-legal-models-hallucinate-1-out-6-or-more-benchmarking-queries https://hai.stanford.edu/news/ai-trial-legal-models-hallucin...
- falcor84 2y ago
  Just out of curiosity, what's the human lawyer baseline on that?
  - csa 2y ago
    > Just out of curiosity, what's the human lawyer baseline on that? Largely depends on how much money the client has.
  - dbish 2y ago
    it's the self-driving car problem. Humans aren't perfect either but people like to ignore that.
    - deleted 2y ago
      [deleted]
    - Terr_ 2y ago
      True, they're similar... But what's also similar is that people make the mistake of focusing on differences in failure rates while glossing over failure modes. Human imperfections are a family of failure-modes which have a gajillion years of experience in detecting, analyzing, preventing, and repairing. Quirks in ML models... not so much. A quick thought-experiment to illustrate the difference: Imagine there's a self-driving car that is exactly half as likely to cause death or injury than a human driver. That's a good failure rate. The twist is that its major failure mode is totally alien, where units attempt to inexplicably chase-murder random pedestrians. It would be difficult to get people to accept that tradeoff.

vasco 2y ago

The infinitely expanding AI-generated metaverse isn't going to render itself, at least in the case of meta I think that might be one of the only pieces missing.

alberth 2y ago

> making the actual profit margins from AI-centric companies close to 0 The same thinking stopped many legacy tech companies from becoming a “cloud” company ~20 years ago. Fast forward to today and the margin for cloud compute is still obscene. And they all wish in hindsight they got into the cloud business way sooner than they ultimately did.

lmm 2y ago
Are the second-tier cloud companies really seeing big margins? Why is it not competed away to zero like airlines?
- mbb70 2y ago
  There is essentially zero cost for a user to switch airlines. The cost to switch clouds is astronomical for any decent sized org.
  - throwaway2037 2y ago
    I like the sentiment of your post. I mostly agree. If you use OpenShift, doesn't that help to reduce the cost of switching cloud providers?
    - coredog64 2y ago
      Not really. What happens when you run a cloud on top of your cloud is that you don’t get to use any of their differentiating features and that winds up costing you money. Plus you have to pay for your own control plane when that’s already baked into the cloud provider’s charge model.
      - throwaway2037 2y ago
        > Plus you have to pay for your own control plane when that’s already baked into the cloud provider’s charge model. When you say "control plane" does this mean Kubernetes?

Willish42 2y ago

> once the big regulatory hammers start dropping to protect American workers Have we been living in the same universe the last 10 years? I don't see this ever happening. Related recent news (literally posted yesterday) https://www.axios.com/2024/07/02/chevron-scotus-biden-cyber-regulations https://www.axios.com/2024/07/02/chevron-scotus-biden-cyber-...

hnthrow289570 2y ago
The only upside is state-level minimum wage increases. The federal minimum wage is still a complete joke at $7.25 an hour. But there's bigger fish to fry for American politics and worker obsolescence is not really top of mind for anyone.
LarsDu88 2y ago
I think people wildly underestimate how protectionist people - particularly educated software engineers and PhDs will get once an AI model directly impacts their source of wealth. Red state blue collar workers got their candidate to pass tariffs. What happens when both blue state white collar workers and red state blue collar workers need to contest with AI. Perhaps not within the next 10 years, but certainly within 20 years! And if you think 20 years is a long time... 2004 was when Halo 2 came out
- dmix 2y ago
  It's not going to stop it though even if they try though. You can't stop technical progress like this any more than you can stop piracy. But agreed, between the unions with political pull and "AI safety" grifters I suspect there could be some level of regulatory risk, particularly for the megacorps in California. I doubt it will be some national thing in the US absent a major political upheaval. Definitely possible in the EU which will probably just be a price passed on to customers or reduced access, but that's nothing new for them.
  - techostritch 2y ago
    I keep seeing people say you can’t stop progress (social, technical, etc.) but has this really been tested? There seems to be a lot of political upheaval at least being threatened on the near future, and depending on the forces that come into power I imagine they may be willing to do a lot to protect that power. Tucker Carlson at one point said if FSD was going to take away trucking jobs we should stop that with regulation.

shoggouth 2y ago

In my opinion Elsevier and others charging for access to academic publications has held back the advancement of humanity to a lower exponential acceleration into the future at a considerable factor. Think of how cancer could have been cured a decade ago if information was allowed to flow freely from the 50's forward - if anyone could have read scientific publications for free. I have no respect for people that want to protect the moat around information that could be used to advance humanity.

OJFord 2y ago
As much as I'd love open access to academic publications and don't think the current model is great: > Think of how cancer could have been cured a decade ago if information was allowed to flow freely from the 50's forward might be a bit fanciful? Unless you're referring to something particular I'm unaware of. The people best equipped and trained to deliver a cure for cancer (and then some, since it tends not to be particularly field-restricred) do have access. I think the loss is more likely in engineering (to the publication's science), cheaper methods, more reliably manufacturable versions of lab prototypes, etc. I doubt there are many people capable of cancer research breakthroughs who don't have access to cancer research, personally. (And to be clear: I'm not capable of it.)
- enjoylife 2y ago
  I’ll add that even if the papers we all wanted were more freely accesible, the replication and completeness of their described methods would be another source of slowdown.
  - robwwilliams 2y ago
    Main problem is still just getting good quantitative data and metadata. Most biomedical researchers are motivated to “tell stories”. Few of us care about generating huge mineable data sets.
- instagib 2y ago
  All of the engineering companies I’ve worked for have not paid for IEEE or any journals. I have to go to the library and maintain membership for IEEE myself then request reimbursement. The schools I’ve worked with have access to everything I’ve needed. They didn’t advertise it but it’s also free for students.

naveen99 2y ago

Propriety data is not necessary for training intelligence. Wikipedia, pubmed, arxiv, Reddit and github are probably sufficient. And babies don’t even use that. I agree though that the returns on hardware rapidly diminish.

kristjansson 2y ago

per Zuckerberg[0], ~half of their H100s were for Reels content recommendation: > I think it was because we were working on Reels. We always want to have enough capacity to build something that we can't quite see on the horizon yet. ... So let's order enough GPUs to do what we need to do on Reels and ranking content and feed. But let's also double that. So there's an immense capacity inside Meta, but the _whole_ fleet isn't available for LLM training. [0]: https://www.dwarkeshpatel.com/p/mark-zuckerberg?open=false#§transcript https://www.dwarkeshpatel.com/p/mark-zuckerberg?open=false#§...

matthewdgreen 2y ago
Surely they’re using some of that hardware to overcome Apple’s attempts to deprive them of targeted advertising data.

forgot-im-old 2y ago

What percentage of GPUs are being used for training versus inference?

sailingparrot 2y ago

> This means the ability to train 50 GPT-4 scale models every 90 days or 200 such models per year. Not anywhere close to that. Those 350k GPUs you talk about aren't linked together. They also definitely aren't all H100s. To train a GPT-4 scale model you need a single cluster, where all the GPUs are tightly linked together. At the scale of 20k+ GPUs, the price you pay in networking to link those GPUs is basically almost the same as the price of those GPUs themselves. It's really hard and expensive to do. FB has maybe 2 such clusters, not more than that. And I'm somewhat confident one of those cluster is an A100 cluster. So they can train maybe 6 GPT-4 every 90 days.

LarsDu88 2y ago
Ok, I might have misread some rumored ballpark figures. And most of the GPUs will be used for inference rather than training. Still 6 GPT-4's every 90 days is pretty amazing.
LarsDu88 2y ago
I had to take a second look at this: https://www.datacenterdynamics.com/en/news/meta-to-operate-600000-gpus-by-year-end/ https://www.datacenterdynamics.com/en/news/meta-to-operate-6... 340,000 H100s 600,000 H100 equivalents (perhaps AMD Instinct cards?) On top of the hundreds of thousands of legacy A100s. And I'm certain the order for B100s will be big. Very big. Even the philanthropic org Chan-Zuckerberg institute current rocks 1000 H100s, probably none used for inference. They are going ALL OUT
- dmix 2y ago
  > Even the philanthropic org Chan-Zuckerberg institute current rocks 1000 H100s, probably none used for inference. What do they use them for?
  - ChainOfFools 2y ago
    tax writeoffs
- onion2k 2y ago

itkovian_ 2y ago

Everything is if scaling keeps making the models better. If it does you don't train 50 gpt4s, you have the best model.

weitendorf 2y ago

There are more kinds of AI-centric companies than just foundation models. Making that equivalency is akin to equating internet companies during the dotcom bubble with just websites like pets.com. Now one semi-skilled person in a couple days can make websites that entire teams back then would have taken months to build, but that doesn't mean google.com and thefacebook.com are easily commodotized or bad businesses just because they're websites.

boloust 2y ago

Why are we assuming we're topping out at a GPT-4 scale model?

eru 2y ago

> The real value methinks is actually over the control of proprietary data used for training which is the single most important factor for model output quality. Maybe. But we've barely scratched the surface of being more economical with data. I remember back in the old days, there was lots of work on eg dropout and data augmentation etc. We haven't seen too much of that with the like of ChatGPT yet. I'm also curious to see what the future of multimodal models holds: you can create almost arbitrarily amounts of extra data by pointing a webcam at the world, especially when combined with a robot, or letting your models also play StarCraft or Diplomacy against each other.

babyshake 2y ago

> once the big regulatory hammers start dropping to protect American workers. The US Supreme Court seems determined to make sure that big regulatory hammers are not going to be dropping, from what I can tell.

lkdfjlkdfjlg 2y ago

> but amazing for builders (ironically). Could you expand on this? Who are "the builders" here? You mean the model developers? I don't see how this situation can be "amazing" for the builders - developers will just get a wage out of their work.

bufferoverflow 2y ago

> this means the ability to train 50 GPT-4 scale models every 90 days or 200 such models per year. What it actually means is that they are training next gen models that are 50X larger. And, considering MS and OpenAI are planning to build a $100 billion AI training computer, these 350K GPUs is just a tiny portion of what they are planning. This isn't an overkill. This is the current plan: throw as much compute as possible and hope intelligence scales with compute.