5 ms·
Does this mean google just deprecated TPUs? Not surprised.
by bushbaba 3y ago
Does this mean google just deprecated TPUs? Not surprised.
- endisneigh 3y agoWhy would you assume that?
- cubefox 3y agoEither GPUs are better for most AI tasks or TPUs. Both being overall approximately equally good is very unlikely.
- panarky 3y ago> most AI tasks Different workloads require different infrastructure. Can your workload saturate the TPU without getting throttled by memory or network? Great! Use TPUs and reduce training cost. But if your TPUs are idle 70% of the time because the constraint is getting data to them ... "A3 represents the first production-level deployment of its GPU-to-GPU data interface, which allows for sharing data at 200 Gbps while bypassing the host CPU. This interface, which Google calls the Infrastructure Processing Unit (IPU), results in a 10x uplift in available network bandwidth for A3 virtual machines (VM) compared to A2 VMs."
- cubefox 3y agoThat's why I said "most" and "overall". Of course TPUs will have a niche. But it looks like the vast majority of money spent on ML compute is converging on GPUs.
- vlovich123 3y agoBut you could do an equivalent TPU<>TPU interlink. Surely that can’t be the reason.
- cavisne 3y agoTPU's have a TPU-TPU interconnect that is faster and lower latency than any GPU cluster [1]. That said this is a huge leap for GPU's on GCP. For A100's SOTA is 1.6tbit per host over Infiniband (which azure and some smaller gpu clouds provided), AWS had 400-800 Gbit and GCP had .... ~100gbit. SOTA seems to be 3.2Tbit for H100 clusters so this still seems a bit slow? (Tricky as they don't give us a clear number just 10x). H100's are much more powerful per chip though so at least initially the clusters will be smaller and not network bound. The tricky thing is no one other than Azure of the big providers seems willing to pay Nvidia's margins for RDMA switches, it seems this is still the case. [1] https://arxiv.org/pdf/2304.01433.pdf https://arxiv.org/pdf/2304.01433.pdf
- tim_sw 3y agothis looks like it's for GCP. TPUs are used for most internal workloads. It's available externally but some of the papercuts and devex without the TPU/TF team helping you can be more painful than using Nvidia/CUDA
- cubefox 3y agoTPUs do compete with GPUs for ML tasks, so yes, this is evidence that GPUs are winning. The only alternative I could imagine is that TPUs will "win" at supercomputers exclusivity aimed at inference (as opposed to training). Since TPUs excel at inference. The question is how much ML compute is used for inference as opposed to training. Not much, I guess, otherwise something like TPUs would be more popular.
- sebzim4500 3y agoI've heard estimates that the amount of compute used to train GPT-4 is equivalent to 8 months of usage and most models are used much less than GPT-4 is, although I guess they are also easier to train.
- hoschicz 3y agoof "usage"? I never bought that claim as it's not clear what usage they mean – on 100x8 months or 10000x8 months?
- saiojd 3y agoSince you seem knowledgable on this topic, what is it that TPUs do differently than GPUs? Why are they better at inference?
- cubefox 3y agoSorry, I actually don't know much about them.
- verdverm 3y agoThey have published various papers and technical reports. The main aim is to make them inhouse and more efficient. Each generation is a little different, like (iirc) v3 is not for training, more for serving at inference time. The use different floating point format and circuits, so they are not good for scientific workloads, iirc again.
- deleted 3y ago[deleted]
- mupuff1234 3y agoHmm no? Clouds offer many competing offerings because different clients have different needs.
- dragonwriter 3y ago> Does this mean google just deprecated TPUs? No, it is the 9,163,584th [0] indication that Google likes to pursue multiple solutions in the same space in parallel with different submarkets, risk profiles, expected payoff terms, or other dimensions. [0] this is a conservative estimate