7 ms·

SemiAnalysis said it last week and AFAIK it wasn't denied. https://newsletter.semianalysis.com/p/tpuv7-google-takes-a-swing-at-the#:~:text=OpenAI%E2%80%99s%20l

by alecco 11mo ago

SemiAnalysis said it last week and AFAIK it wasn't denied. https://newsletter.semianalysis.com/p/tpuv7-google-takes-a-swing-at-the#:~:text=OpenAI%E2%80%99s%20leading%20researchers%20have%20not%20completed%20a%20successful%20full%2Dscale%20pre%2Dtraining%20run%20that%20was%20broadly%20deployed%20for%20a%20new%20frontier%20model%20since%20GPT%2D4o%20in%20May%202024%2C%20highlighting%20the%20significant%20technical%20hurdle%20that%20Google%E2%80%99s%20TPU%20fleet%20has%20managed%20to%20overcome. https://newsletter.semianalysis.com/p/tpuv7-google-takes-a-s...

CamperBob2 11mo ago
That is.... actually a seriously meaty article from a blog I've never heard of. Thanks for the pointer.
- seatac76 11mo ago
  Semi analysis is great, they typically do semiconductors but reporting is top notch.
  - lanstin 11mo ago
    Wow, that was a good article. So much detail from financial to optical linking to build various data flow topologies. Makes me less aghast at the $10M salaries for the masters of these techniques.
- CSMastermind 11mo ago
  Semianalysis is great, def recommend following
- ipnon 11mo ago
  Dylan Patel founded Semianalysis and he has a great interview with Satya Nadella on Dwarkesh Patel's podcast.
- Numerlor 11mo ago
  This article about them got published just yesterday... https://news.ycombinator.com/item?id=46124883 https://news.ycombinator.com/item?id=46124883 There's a lot of misleading information in what they publish, plagiarism, and I believe some information that wouldn't be possible to get without breaking NDAs
  - girvo 11mo ago
    > I believe some information that wouldn't be possible to get without breaking NDAs …why would I care about this in the slightest?
rahimnathwani 11mo ago
Dylan Patel joined Dwarkesh recently to interview Satya Nadella: https://www.dwarkesh.com/p/satya-nadella-2 https://www.dwarkesh.com/p/satya-nadella-2
- embedding-shape 11mo ago
  And this is relevant how? That interview is 1.5 hours, not something you just casually drop a link to and say "here, listen to this to even understand what point I was trying to make"
  - rahimnathwani 11mo ago
    Sorry, this was meant to be a reply to this comment: https://news.ycombinator.com/item?id=46127942 https://news.ycombinator.com/item?id=46127942 I was trying to make the point that SemiAnalysis is semi-famous.
    - tim333 11mo ago
      The video is interesting showing microsoft's latest data center and Nadella talking about them. I prefered the youtube version https://youtu.be/8-boBsWcr5A https://youtu.be/8-boBsWcr5A
  - kovezd 11mo ago
    You can now ask Gemini, about a video. Very useful!
    - andai 11mo ago
      I have a few lines of "download subtitles with yt-dlp", "remove the VTT crap", and "shove it into llm with a summarization prompt and/or my question appended", but I mostly use Gemini for that now. (And I use it for basically nothing else, oddly enough. They just have the monopoly on access to YouTube transcripts ;)
    - embedding-shape 11mo ago
RossBencina 11mo ago
The SemiAnalysis article that you linked to stated: "OpenAI’s leading researchers have not completed a successful full-scale pre-training run that was broadly deployed for a new frontier model since GPT-4o in May 2024, highlighting the significant technical hurdle that Google’s TPU fleet has managed to overcome." Given the overall quality of the article, that is an uncharacteristically convoluted sentence. At the risk of stating the obvious, "that was broadly deployed" (or not) is contingent on many factors, most of which are not of the GPU vs. TPU technical variety.
- nbardy 11mo ago
  This is misleading. They had 4.5 which was a new scaled up training run. It was a huge model and only served to pro users, but the biggest models are always used as teacher models for smaller models. Thats how you do distillation. It would be stupid to not use the biggest model you have in distillation and a waste since they have the weights. The would have taken some time to calculate the efficiency gains of pretraining vs RL. Resumed the GPT-4.5 for whatever budget made sense and then spent the rest on RL. Sure they chose to not serve the large base models anymore for cost reasons. But I’d guess Google is doing the same. Gemini 2.5 samples very fast and seems way to small to be their base pre train. The efficiency gains in pertaining scale with model scale so it makes sense to train the largest model possible. But then the models end up super sparse and oversized and make little sense to serve in inference without distillation. In RL the efficiency is very different because you have to inference sample the model to draw online samples. So small models start to make more sense to scale. Big model => distill => RL Makes the most theoretical sense for training now days for efficient spending. So they already did train a big model 4.5. Not using it would have been absurd and they have a known recipe they could return scaling on if the returns were justified.
  - copedetector 11mo ago
    [flagged]
  - barrell 11mo ago
    My understanding of 4.5 was that it was released long, long after the initial training run finished. It also had an older cutoff date than the newer 4o models
deleted 11mo ago
[deleted]
binkHN 11mo ago
This is a really great breakdown. With TPUs seemingly more efficient and costing less overall, how does this play for Nvidia? What's to stop them from entering the TPU race with their $5 trillion valuation?
- sysguest 11mo ago
  $5 trillion valuation doesn't mean it has $5 trillion cash in pocket -- so "it depends"
- dragonwriter 11mo ago
  > What's to stop them from entering the TPU race with their $5 trillion valuation? Valuation isn’t available money; they'd have to raise more money in the current, probably tighter for them, investment environment to enter the TPU race, since the money they have already raised that that valuation is based on is already needed to provide runway for what they are already doing without putting money into the TPU race
- matwood 11mo ago
  As others mentioned, 5T isn't money available to NVDA. It could leverage that to buy a TPU company in an all stock deal though. The bigger issue is that entering a 'race' implies a race to the bottom. I've noted this before, but one of NVDA's biggest risks is that its primary customers are also technical, also make hardware, also have money, and clearly see NVDA's margin (70% gross!!, 50%+ profit) as something they want to eliminate. Google was first to get there (not a surprise), but Meta is also working on its own hardware along with Amazon. This isn't a doom post for NVDA the company, but its stock price is riding a knifes edge. Any margin or growth contraction will not be a good day for their stock or the S&P.
  - sigmoid10 11mo ago
    Making the hardware is actually the easy part. Everyone and their uncle who had some cash have tried by now: Microsoft, Meta, Tesla, Huawei, Amazon, Intel - the list goes on and on. But Nvidia is not a chip company. Huang himself said they are mostly a software company. And that is how they were able to build a gigantic moat. Because noone else has even come close on the software side. Google is the only one who has had some success on this side, because they also spent tons of money and time on software refinement by now, while all the other chips vanished into obscurity.

by alecco 11mo ago

CamperBob2 11mo ago

That is.... actually a seriously meaty article from a blog I've never heard of. Thanks for the pointer.

seatac76 11mo ago
Semi analysis is great, they typically do semiconductors but reporting is top notch.
- lanstin 11mo ago
  Wow, that was a good article. So much detail from financial to optical linking to build various data flow topologies. Makes me less aghast at the $10M salaries for the masters of these techniques.
CSMastermind 11mo ago
Semianalysis is great, def recommend following
ipnon 11mo ago
Dylan Patel founded Semianalysis and he has a great interview with Satya Nadella on Dwarkesh Patel's podcast.
Numerlor 11mo ago
This article about them got published just yesterday... https://news.ycombinator.com/item?id=46124883 https://news.ycombinator.com/item?id=46124883 There's a lot of misleading information in what they publish, plagiarism, and I believe some information that wouldn't be possible to get without breaking NDAs
- girvo 11mo ago
  > I believe some information that wouldn't be possible to get without breaking NDAs …why would I care about this in the slightest?

rahimnathwani 11mo ago

Dylan Patel joined Dwarkesh recently to interview Satya Nadella: https://www.dwarkesh.com/p/satya-nadella-2 https://www.dwarkesh.com/p/satya-nadella-2

embedding-shape 11mo ago
And this is relevant how? That interview is 1.5 hours, not something you just casually drop a link to and say "here, listen to this to even understand what point I was trying to make"
- rahimnathwani 11mo ago
  Sorry, this was meant to be a reply to this comment: https://news.ycombinator.com/item?id=46127942 https://news.ycombinator.com/item?id=46127942 I was trying to make the point that SemiAnalysis is semi-famous.
  - tim333 11mo ago
    The video is interesting showing microsoft's latest data center and Nadella talking about them. I prefered the youtube version https://youtu.be/8-boBsWcr5A https://youtu.be/8-boBsWcr5A
- kovezd 11mo ago
  You can now ask Gemini, about a video. Very useful!
  - andai 11mo ago
    I have a few lines of "download subtitles with yt-dlp", "remove the VTT crap", and "shove it into llm with a summarization prompt and/or my question appended", but I mostly use Gemini for that now. (And I use it for basically nothing else, oddly enough. They just have the monopoly on access to YouTube transcripts ;)
  - embedding-shape 11mo ago

RossBencina 11mo ago

The SemiAnalysis article that you linked to stated: "OpenAI’s leading researchers have not completed a successful full-scale pre-training run that was broadly deployed for a new frontier model since GPT-4o in May 2024, highlighting the significant technical hurdle that Google’s TPU fleet has managed to overcome." Given the overall quality of the article, that is an uncharacteristically convoluted sentence. At the risk of stating the obvious, "that was broadly deployed" (or not) is contingent on many factors, most of which are not of the GPU vs. TPU technical variety.

nbardy 11mo ago
This is misleading. They had 4.5 which was a new scaled up training run. It was a huge model and only served to pro users, but the biggest models are always used as teacher models for smaller models. Thats how you do distillation. It would be stupid to not use the biggest model you have in distillation and a waste since they have the weights. The would have taken some time to calculate the efficiency gains of pretraining vs RL. Resumed the GPT-4.5 for whatever budget made sense and then spent the rest on RL. Sure they chose to not serve the large base models anymore for cost reasons. But I’d guess Google is doing the same. Gemini 2.5 samples very fast and seems way to small to be their base pre train. The efficiency gains in pertaining scale with model scale so it makes sense to train the largest model possible. But then the models end up super sparse and oversized and make little sense to serve in inference without distillation. In RL the efficiency is very different because you have to inference sample the model to draw online samples. So small models start to make more sense to scale. Big model => distill => RL Makes the most theoretical sense for training now days for efficient spending. So they already did train a big model 4.5. Not using it would have been absurd and they have a known recipe they could return scaling on if the returns were justified.
- copedetector 11mo ago
  [flagged]
- barrell 11mo ago
  My understanding of 4.5 was that it was released long, long after the initial training run finished. It also had an older cutoff date than the newer 4o models

deleted 11mo ago

[deleted]

binkHN 11mo ago

This is a really great breakdown. With TPUs seemingly more efficient and costing less overall, how does this play for Nvidia? What's to stop them from entering the TPU race with their $5 trillion valuation?

sysguest 11mo ago
$5 trillion valuation doesn't mean it has $5 trillion cash in pocket -- so "it depends"
dragonwriter 11mo ago
> What's to stop them from entering the TPU race with their $5 trillion valuation? Valuation isn’t available money; they'd have to raise more money in the current, probably tighter for them, investment environment to enter the TPU race, since the money they have already raised that that valuation is based on is already needed to provide runway for what they are already doing without putting money into the TPU race
matwood 11mo ago
As others mentioned, 5T isn't money available to NVDA. It could leverage that to buy a TPU company in an all stock deal though. The bigger issue is that entering a 'race' implies a race to the bottom. I've noted this before, but one of NVDA's biggest risks is that its primary customers are also technical, also make hardware, also have money, and clearly see NVDA's margin (70% gross!!, 50%+ profit) as something they want to eliminate. Google was first to get there (not a surprise), but Meta is also working on its own hardware along with Amazon. This isn't a doom post for NVDA the company, but its stock price is riding a knifes edge. Any margin or growth contraction will not be a good day for their stock or the S&P.
- sigmoid10 11mo ago
  Making the hardware is actually the easy part. Everyone and their uncle who had some cash have tried by now: Microsoft, Meta, Tesla, Huawei, Amazon, Intel - the list goes on and on. But Nvidia is not a chip company. Huang himself said they are mostly a software company. And that is how they were able to build a gigantic moat. Because noone else has even come close on the software side. Google is the only one who has had some success on this side, because they also spent tons of money and time on software refinement by now, while all the other chips vanished into obscurity.