3 ms·
Google absolutely runs workloads that need supercomputer tight scaling. That's why they built TPUs with ICI networks.
by dekhn 4y ago
Google absolutely runs workloads that need supercomputer tight scaling. That's why they built TPUs with ICI networks.
- nerpderp82 4y agoYou got me on a technicality, but nothing of practicality. This isn't exposed to the larger Google, nor to customers. Google itself as an organism doesn't HPC. It is an ad company that runs ad company systems. Why are there no top-k super computers running on Google?
- HyperSane 4y agowhat is ICI networks?
- dekhn 4y agohttps://dl.acm.org/doi/pdf/10.1145/3360307 https://dl.acm.org/doi/pdf/10.1145/3360307 Very fast (hundreds of gigabits per connection) networks that attach TPUs on the same board, as well as between boards. It's wired up point to point, forming a logical grid (technically a 2D or 3D torus with wraparound links).
- rlupi 4y agoICI is interchip interconnect in TPUv4 pods. You can read more on paper that my collegues in Google platforms recently published https://arxiv.org/abs/2304.01433 https://arxiv.org/abs/2304.01433 https://cloud.google.com/blog/topics/systems/tpu-v4-enables-performance-energy-and-co2e-efficiency-gains https://cloud.google.com/blog/topics/systems/tpu-v4-enables-... I am so happy this paper is finally out :-) I led the SRE work during their NPI. It's a pain when you have to wait 3 years to discuss what you worked on. They are based on OCS, which powers Google datacenter networks https://arxiv.org/abs/2208.10041 https://arxiv.org/abs/2208.10041
- markhahn 4y agoICI is basically nvlink, not like IB.
- dekhn 4y agono, ICI is a fiber optic (or electrical) network between TPUs and it doesn't have any switch functionality. I used to work for Google (on TPUs and ICI). Anyway, the comparisons don't matter that much. The long and short of it is that Google built their own supercomputers (finally) that can do allreduce and alltoall patterns, as well as low-latency broadcast.
- markhahn 4y agomedium isn't the point, and duh, torus networks are distributed-switching.
- dekhn 4y agoDo you have a point? There's nothing technically wrong with saying that TPUs with ICI are a supercomputer- they are very similar in design to the T3E I used decades ago. Please make your point instead of trying to negate whatt I said. Medium matters a lot- because supercomputers are fundamentally constrained by physics in terms of power dissipation, transistor density, and speed of light, all-optical networks have slightly lower latencies than electrical, and also let you build larger systems (longer cables).