3 ms·
Is this the sort of thing that a person with curiosity and a 4090 could do? It says he used 8xA100s in the cloud to do this but is it just a matter of the 4090
by ls612 2y ago
Is this the sort of thing that a person with curiosity and a 4090 could do? It says he used 8xA100s in the cloud to do this but is it just a matter of the 4090 going 8x slower or will memory constraints kill the whole endeavour?
- smaddox 2y ago4090 should have enough VRAM for 124M param training. Even at float32 precision, with AdamW optimizer, parameters should only be ~2GB (124M params x 4 bytes per param x ~4 for optimizer weight overhead). So there should be plenty of remaining space for activations.