3 ms·
Wormhole does support scale out though via it's built in networking? And there seem to be links for something on top of the board, NVLINK style. And yes, I kno
by buildbot 3y ago
Wormhole does support scale out though via it's built in networking? And there seem to be links for something on top of the board, NVLINK style.
And yes, I know, I've been working on LLM pretraining for about 4 years now, since 2020. The number formats themselves so far are mostly scale invariant or improve with larger scale - you can quantize a larger model and see less performance drop than a smaller model.