https://twitter.com/abacaj/status/1635837820270002178 https://twitter.com/abacaj/status/1635837820270002178 the bloomberg paper does talk about how they sized

https://twitter.com/abacaj/status/1635837820270002178 https://twitter.com/abacaj/status/1635837820270002178 the bloomberg paper does talk about how they sized the model based on their compute budget and research on time to train ... megatron has 1t parameters i'm intrigued by some of the knowledge graph performance benchmarks and wonder about training it alongside an explicit knowledge graph, instead of building its own implicit knowledge graph, despite the 'bitter lesson' about just letting simple models and massive computation do their thing.