Loved this and the detail - thank you. It’s the best inside detail on the engineering work behind these models I’ve ever read. Two things I’m curious about- fi

Loved this and the detail - thank you. It’s the best inside detail on the engineering work behind these models I’ve ever read. Two things I’m curious about- first, what, if any difference would you imagine in training a 400b parameter model? It seems that you have plenty of vram across the cluster, but I want to know what you think. Second, do you think this sort of architecture is the end game for model training? It seems sooo fragile. Are there better shared training mechanisms/architectures? Are there better cluster geometries? Thanks again - great read.