6 ms·
Touching the Elephant – TPUs
- Simplita 11mo agoThis was a nice breakdown. I always feel most TPU articles skip over the practical parts. This one actually connects the concepts in a way that clicks.
- Zigurd 11mo agoThe extent to which TPU architecture is built for the purpose also doesn't happen in a single design generation. Ironwood is the seventh generation of TPU, and that matters a lot.
- alecco 11mo agoI'm surprised the perspective of China making TPUs at scale in a couple of years is not bigger news. It could be a deadly blow for Google, NVIDIA, and the rest. Combine it with China's nuclear base and labor pool. And the cherry on top, America will train 600k Chinese students as Trump agreed to. The TPUv4 and TPUv6 docs were stolen by a Chinese national in 2022/2023: https://www.cyberhaven.com/blog/lessons-learned-from-the-google-trade-secret-theft-indictment https://www.cyberhaven.com/blog/lessons-learned-from-the-goo... https://www.justice.gov/opa/pr/superseding-indictment-charges-chinese-national-relation-alleged-plan-steal-proprietary-ai https://www.justice.gov/opa/pr/superseding-indictment-charge... And that's just 1 guy that got caught. Who knows how many other cases were there. A Chinese startup is already making clusters of TPUs and has revenue https://www.scmp.com/tech/tech-war/article/3334244/ai-start-offers-local-alternative-googles-tpu-china-seeks-cut-nvidia-reliance https://www.scmp.com/tech/tech-war/article/3334244/ai-start-...
- Workaccount2 11mo agoManufacturing is the hard part. China certainly has the knowledge to build a TPU architecture without needing to steal the plans. What they don't have is the ability to actually build the chips. This is even in spite of also stealing lithography plans. There is a dark art to semiconductor manufacturing that pretty much only TSMC really has the wizards for. Maybe intel and samsung a bit too.
- tomrod 11mo agoLot of retired fab folks in the Austin area if you needed to spin up a local fab. It's really not a dark art, there are plenty of folks that have experience in the industry.
- Workaccount2 11mo agoThis is sort of like saying there are lots of kids in the local community college shop class if you want to spin up an F1 team. The knowledge of making 2008 era chips is not a gating factor for getting a handful of atoms to function as a transistor in current SOTA chips. There are probably 100 people on earth who know how to do this, and the majority of them are in Taiwan. Again, China has literally stolen the plans for EUV lithography, years ago, and still cannot get it to work. Even Samsung and Intel, using the same machines as TSMC, cannot match what they are doing. It's a dark art in the most literal sense. Nevermind that new these cutting edge fabs cost ~$50 Billion each.
- checker659 11mo agoI've always wondered. If you have fuck you money, wouldn't it be possible to build GPUs to do LLM matmul with 2008 technology. Again, assuming energy costs / cooling costs don't matter.
- Zigurd 11mo agoEnergy, cooling, and how much of the building you're taking up do matter. They matter less and in a more manageable way for hyperscalers that have a long established resource management practice in lots of big data centers because they can phase in new technologies as they phase out the old. But it's a lot more daunting to think about building a data center big enough to compete with one full of Blackwell systems there are more than 10 times more performant per watt and per square foot.
- pixl97 11mo agoBuilding the clean rooms at this scale is a limitation in itself. Just getting the factory setup to and the machines put in so they don't generate particulate matter in operation is an art that compares in difficulty to making the chips themselves.
- fullofideas 11mo ago>Combine it with China's nuclear base and labor pool. And the cherry on top, America will train 600k Chinese students as Trump agreed to. I dont understand this part. What has nuclear base got to do with chip manufacturing? And surely, not all 600k students are learning chip design or stealing plans
- pixl97 11mo agoNuclear power is what they are talking about, not weapons.
- alecco 11mo agoI mean they have the power grid to run TPUs at 10x the scale of USA. About students, have you seen the microelectronic labs in American universities lately? A huge chunk are Chinese already. Same with some of the top AI labs.
- dylanowen 11mo agoI assume the nuclear reactors are to power the data centers using the new chips. There have been a few mentions on HN about the US being very behind in building enough power plants to run LLM workloads
- renewiltord 11mo agoWe should ask ourselves: is it worth ruining local communities in order to beat China in the global sphere?
- pstuart 11mo agoThat question was asked and answered years ago and the answer is YES (not me personally, but the people in charge) There are things about China not to be celebrated but one cannot help but admire the way that they invest in their country as a whole. The US is all about "what's in it for me".
- 11mo ago
- llm_nerd 11mo ago>It could be a deadly blow for Google, NVIDIA, and the rest. How would this be a deadly blow to Google? Google makes TPUs for their own services and products, avoiding paying the expensive nvidia tax. If other people make similar products, this has effectively zero impact on Google. nvidia knew their days were numbered, at least in their ownership of the whole market. And China hardly had to steal the great plans for a TPU to make one, and a FMA/MAC unit is actually a surprisingly simple bit of hardware to design. Everyone is adding "TPUs" in their chips - Apple, Qualcomm, Google, AMD, Amazon, Huawei, nvidia (that's what tensor cores are) and everyone else. And that startup isn't the big secret. Huawei already has solutions matching the H20. Once the specific need that can be serviced by an ASIC is clear, everyone starts building it. >America will train 600k Chinese students as Trump agreed to What great advantage do you think this is? America isn't remotely the great gatekeeper on this. If anything, Taiwan + the Netherlands (ASML) are. China would yield infinitely more value in learning manufacturing and fabrication secrets than cloning some specific ASIC.
- lukasb 11mo agoYeah I'm terrified that TPUs will get cheaper, that would be awful.
- pests 11mo agoHalf the article was about the extensive software codependency between TPU's, Borb, lilpunet, their optical switching network, etc. How much of that is manufacturing and not just software and engineering experience, which won't be so easy to copy?
- desideratum 11mo agoThe Scaling ML textbook also has an excellent section on TPUs. https://jax-ml.github.io/scaling-book/tpus/ https://jax-ml.github.io/scaling-book/tpus/
- jauntywundrkind 11mo agoI also enjoyed https://henryhmko.github.io/posts/tpu/tpu.html https://henryhmko.github.io/posts/tpu/tpu.html https://news.ycombinator.com/item?id=44342977 https://news.ycombinator.com/item?id=44342977 . The work that XLA & schedulers are doing here is wildly impressive. This feels so much drastically harder to work with than Itanium must have been. ~400bit VLIW, across extremely diverse execution units. The workload is different, it's not general purpose, but still awe inspiring to know not just that they built the chip but that the software folks can actually use such a wildly weird beast. I wish we saw more industry uptake for XLA. Uptakes not bad, per-se: there's a bunch of different hardware it can target! But what amazing secret sauce, it's open source, and it doesn't feel like there's the industry rally behind it it deserves. It feels like Nvidia is only barely beginning to catch up, to dig a new moat, with the just announced Nvidia Tiles. Such huge overlap. Afaik, please correct if wrong, but XLA isn't at present particularly useful at scheduling across machines, is it? https://github.com/openxla/xla https://github.com/openxla/xla
- desideratum 11mo agoThanks for sharing this. I agree w.r.t. XLA. I've been moving to JAX after many years of using torch and XLA is kind of magic. I think torch.compile has quite a lot of catching up to do. > XLA isn't at present particularly useful at scheduling across machines, I'm not sure if you mean compiler-based distributed optimizations, but JAX does this with XLA: https://docs.jax.dev/en/latest/notebooks/Distributed_arrays_and_automatic_parallelization.html#intro-and-a-quick-example https://docs.jax.dev/en/latest/notebooks/Distributed_arrays_...
- alevskaya 11mo agoI do think it's a lot simpler than the problem Itanium was trying to solve. Neural nets are just way more regular in nature, even with block sparsity, compared to generic consumer pointer-hopping code. I wouldn't call it "easy", but we've found that writing performant NN kernels for a VLIW architecture chip is in practice a lot more straightforward than other architectures. JAX/XLA does offer some really nice tools for doing automated sharding of models across devices, but for really large performance-optimized models we often handle the comms stuff manually, similar in spirit to MPI.
- ddtaylor 11mo agoAre TPUs still stuck to their weird Google bucket thing when using GCP? I hated that.
- daveguy 11mo ago[flagged]
- alivetoad 11mo agoAuthor here. I was expecting some negativity but for whatever reason this one bites more than I imagined. The effort here was earnest and I'm sorry it reads as slop to you. Appreciate the feedback.
- allisdust 11mo agoFor every negative comment, there are 100s of positive comments that never get made. I have spent close to 2 hours on your extremely info dense article and loved every bit of it. Looking forward for the next one.
- alivetoad 11mo agoAppreciate it! Have a couple ideas kicking around for my next write up, but taking a beat to evaluate what went well and all that jazz
- imtringued 11mo agoFor those in the know, the "MXU Cycle Timing" diagram is a thing of beauty.
- alivetoad 11mo agoLove to hear it. I struggled feeling like they looked amateurish, but so long as the ideas are communicated clearly I can live with it
- tomhow 11mo agoModerator here. How much was AI used to write/edit the post?
- alivetoad 11mo ago
- randomtoast 11mo agoI don't think Moore's Law is dead. Starting point: In 1965, the most advanced chips contained roughly 50 to 100 transistors (e.g., early integrated logic). Lets take 1965 -> 2025, which is 60 years. Number of doubling intervals: 60 years / 2 years per doubling = 30 doublings So the theoretical prediction is: Transistors in 2025 (predicted) = 100 × 2^30 ≈ 107 billion transistors The Apple M1 Ultra has 114 billion transistors.
- deleted 11mo ago[deleted]
- KolenCh 11mo agoSome people take Moore’s law in a strong sense: doubling rate is a constant. That is long dead. But if we relax it to be a slowly varying constant, then it is not dead. That constant has been changed (by consensus) for a few times already. Your mistake is to (1) take that constant literally (ie using the strong law) and (2) uses the boundary points to find the “average” effect. The latter is a really flawed argument as it cannot prove it hasn’t been dead (a recent effect) because you haven’t considered it’s change over time.
- _jzlw 11mo ago[flagged]