4 ms·
This is my impression as well. Here is an example of the engineering capabilities required to train the modern models: https://github.com/facebookresearch/metas
by oliwary 4y ago
This is my impression as well. Here is an example of the engineering capabilities required to train the modern models:
https://github.com/facebookresearch/metaseq/blob/main/projects/OPT/chronicles/OPT175B_Logbook.pdf https://github.com/facebookresearch/metaseq/blob/main/projec... [PDF]
It's a 114 page document detailing the months of full-time work for multiple engineers that went into training a 175B language model at meta AI.