3 ms·

> there is no indication Google intends to make their chips available outside of GCP. 1. You can buy mini versions of their chips through Coral (coral.ai). But

> there is no indication Google intends to make their chips available outside of GCP. 1. You can buy mini versions of their chips through Coral (coral.ai). But yea, they’d never sell them externally as long as there exists a higher-margin advantage to selling software on top of them, and chips have supply constraints. 2. Google can sell VMs with the tensor chips attached, like GPUs. Most organizations with budgets that’d impact things will be using the cloud. If Apple/MSFT/AWS/Goog/Meta start serious building their own chips, NVidia could be left out of the top end.

haldujai 3y ago
> Google can sell VMs with the tensor chips attached, like GPUs. They have already been doing this for quite a while now and even when offered free via TRC barely anyone uses TPUs. There is nothing to suggest that Google as an organization is shifting focus to be the HPC cloud provider for the world. As it stands TPU cloud access really seems ancillary to their own internal needs. > If Apple/MSFT/AWS/Goog/Meta start serious building their own chips, NVidia could be left out of the top end. That's a big "if", especially within two years, given that this chip design/manufacturing isn't really a core business interest for any of those companies (other than Google which has massive internal need and potentially Apple who have never indicated interest in being a cloud provider). They certainly could compete with Nvidia for the top-end, but it would be really hard and how much would the vertical integration actually benefit their bottom line? A 2048 GPU SuperPOD is what, like 30M? There's also the risk that the not-always-friendly DoJ gets anti-trusty if a cloud provider has a massive advantage and is locking the HW in their walled garden.
- fragmede 3y ago
  > barely anyone uses TPUs What are you basing that on? I'm not aware of GCP having released any numbers on their usage.
  - haldujai 3y ago
    Anecdotal data warning but for context my research is in medical informatics and I've quite extensively followed publications on transformers dating back to the early BERT variants (including non-medical). I'm making that statement as my experience (easily several hundreds of publications read or reviewed over 3 years) is that it is very uncommon to see TPU's mentioned or TRC acknowledged in any non-Google transformer paper (especially major publications) dating back to the early BERT family of models despite the fact that Google is very generous with research credits (they'll give out preemptible v3-32s and v3-64s for 14 days with little question, presumably upgraded now as I haven't asked for credits in a while). Fully acknowledge this isn't quality evidence to back my claim and I'm happy to be proven wrong but I'm very confident a literature review would support this as when I tried to use TPUs myself I couldn't find much. This doesn't account for industry use, there is probably a non-insignificant amount of enterprise customers still using AutoML (I can think of a few at least) which I believe uses the TPU cloud but I would be surprised if many use TPU nodes directly outside of Jax shops like cohere and anyone still using TF. PyTorch XLA has just breaks too much otherwise and when I last tried to use it in January of this year there was still quite a significant throughput reduction on TPUs. Additionally when using nodes there is a steeper learning curve on the ops side (VM, storage, Stackdriver logging) that make working with them harder than spinning up a A100x8 which is relatively cheap, cheaper than the GCP learning curve for sure.
    - sangnoir 3y ago
      > Anecdotal data warning but for context my research is in medical informatics Isn't Medical Informatics inherently biased against the cloud? That's my uninformed guess as an outsider.
- bagacrap 3y ago
  It seems like one of these two things must be true: A) Nvidia's TAM is not really what the stock is priced foe B) Google will try to enter this market and compete Either way NVDA looks perilously pricey, not that that is very predictive of anything (see TSLA).
brutuscat 3y ago
So what, are they pathological layers? https://youtu.be/XVcKLetqf3U https://youtu.be/XVcKLetqf3U The Intel® Data Center GPU Max Series outperforms Nvidia H100 PCIe card by an average of 30% on diverse workloads1, while independent software vendor Ansys shows a 50% speedup for the Max Series GPU over H100 on AI-accelerated HPC applications.2 The Xeon Max Series CPU, the only x86 processor with high bandwidth memory, exhibits a 65% improvement over AMD’s Genoa processor on the High Performance Conjugate Gradients (HPCG) benchmark1, using less power. High memory bandwidth has been noted as among the most desired features for HPC customers.3 4th Gen Intel Xeon Scalable processors – the most widely used in HPC – deliver a 50% average speedup over AMD’s Milan4, and energy company BP’s newest 4th Gen Xeon HPC cluster provides an 8x increase in performance over its previous-generation processors with improved energy efficiency.2 The Gaudi2 deep learning accelerator performs competitively on deep learning training and inference, with up to 2.4x faster performance than Nvidia A100. https://www.intel.com/content/www/us/en/newsroom/news/intel-delivers-ai-accelerated-hpc-performance.html https://www.intel.com/content/www/us/en/newsroom/news/intel-...

haldujai 3y ago

> Google can sell VMs with the tensor chips attached, like GPUs. They have already been doing this for quite a while now and even when offered free via TRC barely anyone uses TPUs. There is nothing to suggest that Google as an organization is shifting focus to be the HPC cloud provider for the world. As it stands TPU cloud access really seems ancillary to their own internal needs. > If Apple/MSFT/AWS/Goog/Meta start serious building their own chips, NVidia could be left out of the top end. That's a big "if", especially within two years, given that this chip design/manufacturing isn't really a core business interest for any of those companies (other than Google which has massive internal need and potentially Apple who have never indicated interest in being a cloud provider). They certainly could compete with Nvidia for the top-end, but it would be really hard and how much would the vertical integration actually benefit their bottom line? A 2048 GPU SuperPOD is what, like 30M? There's also the risk that the not-always-friendly DoJ gets anti-trusty if a cloud provider has a massive advantage and is locking the HW in their walled garden.

fragmede 3y ago
> barely anyone uses TPUs What are you basing that on? I'm not aware of GCP having released any numbers on their usage.
- haldujai 3y ago
  Anecdotal data warning but for context my research is in medical informatics and I've quite extensively followed publications on transformers dating back to the early BERT variants (including non-medical). I'm making that statement as my experience (easily several hundreds of publications read or reviewed over 3 years) is that it is very uncommon to see TPU's mentioned or TRC acknowledged in any non-Google transformer paper (especially major publications) dating back to the early BERT family of models despite the fact that Google is very generous with research credits (they'll give out preemptible v3-32s and v3-64s for 14 days with little question, presumably upgraded now as I haven't asked for credits in a while). Fully acknowledge this isn't quality evidence to back my claim and I'm happy to be proven wrong but I'm very confident a literature review would support this as when I tried to use TPUs myself I couldn't find much. This doesn't account for industry use, there is probably a non-insignificant amount of enterprise customers still using AutoML (I can think of a few at least) which I believe uses the TPU cloud but I would be surprised if many use TPU nodes directly outside of Jax shops like cohere and anyone still using TF. PyTorch XLA has just breaks too much otherwise and when I last tried to use it in January of this year there was still quite a significant throughput reduction on TPUs. Additionally when using nodes there is a steeper learning curve on the ops side (VM, storage, Stackdriver logging) that make working with them harder than spinning up a A100x8 which is relatively cheap, cheaper than the GCP learning curve for sure.
  - sangnoir 3y ago
    > Anecdotal data warning but for context my research is in medical informatics Isn't Medical Informatics inherently biased against the cloud? That's my uninformed guess as an outsider.
bagacrap 3y ago
It seems like one of these two things must be true: A) Nvidia's TAM is not really what the stock is priced foe B) Google will try to enter this market and compete Either way NVDA looks perilously pricey, not that that is very predictive of anything (see TSLA).

brutuscat 3y ago

So what, are they pathological layers? https://youtu.be/XVcKLetqf3U https://youtu.be/XVcKLetqf3U The Intel® Data Center GPU Max Series outperforms Nvidia H100 PCIe card by an average of 30% on diverse workloads1, while independent software vendor Ansys shows a 50% speedup for the Max Series GPU over H100 on AI-accelerated HPC applications.2 The Xeon Max Series CPU, the only x86 processor with high bandwidth memory, exhibits a 65% improvement over AMD’s Genoa processor on the High Performance Conjugate Gradients (HPCG) benchmark1, using less power. High memory bandwidth has been noted as among the most desired features for HPC customers.3 4th Gen Intel Xeon Scalable processors – the most widely used in HPC – deliver a 50% average speedup over AMD’s Milan4, and energy company BP’s newest 4th Gen Xeon HPC cluster provides an 8x increase in performance over its previous-generation processors with improved energy efficiency.2 The Gaudi2 deep learning accelerator performs competitively on deep learning training and inference, with up to 2.4x faster performance than Nvidia A100. https://www.intel.com/content/www/us/en/newsroom/news/intel-delivers-ai-accelerated-hpc-performance.html https://www.intel.com/content/www/us/en/newsroom/news/intel-...