3 ms·
> another fun fact, according to Twitter rumors GPT-5 is already training on 25,000 A100s w/8 GPUs each, so 4.8m GPU/hours per day. If true that is incredible.
by chasd00 4y ago
> another fun fact, according to Twitter rumors GPT-5 is already training on 25,000 A100s w/8 GPUs each, so 4.8m GPU/hours per day.
If true that is incredible. From what I understand there isn’t a function that relates the increase of training time/effort to performance. For example, twice the training doesn’t mean twice the performance. It could mean no increase or 1000x increase, no one really knows. Does anyone have any idea what to expect in terms of capabilities when this training is finished?
- RockyMcNuts 4y agohttps://twitter.com/abacaj/status/1635837820270002178 https://twitter.com/abacaj/status/1635837820270002178 the bloomberg paper does talk about how they sized the model based on their compute budget and research on time to train ... megatron has 1t parameters i'm intrigued by some of the knowledge graph performance benchmarks and wonder about training it alongside an explicit knowledge graph, instead of building its own implicit knowledge graph, despite the 'bitter lesson' about just letting simple models and massive computation do their thing.