3 ms·
It was natively trained in FP4. Probably both to reduce VRAM usage at inference time (fits on a single H100), and to allow better utilization of B200s (which ar
by reissbaker 1y ago
It was natively trained in FP4. Probably both to reduce VRAM usage at inference time (fits on a single H100), and to allow better utilization of B200s (which are especially fast for FP4).
- irthomasthomas 1y agoInteresting, thanks. I didn't know you could even train at FP4 on H100s
- reissbaker 1y agoIt's impressive they got it to work — the lowest I'd heard of this far was native FP8 training.