some interesting tidbits: > stretched our ML supercomputer scale .. to 4096 TPU v4 nodes > The Google tradition is to write retrospective papers ... TPU v4s a

some interesting tidbits: > stretched our ML supercomputer scale .. to 4096 TPU v4 nodes > The Google tradition is to write retrospective papers ... TPU v4s and A100s deployed in 2020 and both use 7nm technology > The appropriate H100 match would be a successor to TPU v4 deployed in a similar time frame and technology (e.g., in 2023 and 4 nm). > TPU v4 supercomputers [are] the workhorses of large language models (LLMs) like LaMDA, MUM, and PaLM]. These features allowed the 540B parameter PaLM model to sustain a remarkable 57.8% of the peak hardware floating point performance over 50 days while training on TPU v4 supercomputers > Google has deployed dozens of TPU v4 supercomputers for both internal use and for external use via Google Cloud > Moreover, the large size of the TPU v4 supercomputer and its reliance on OCSes looks prescient given that the design began two years before the paper was published that has stoked the enthusiasm for LLMs