3 ms·
> It looks like a mid-level implementations of training and inference I’m not familiar with how any of this works but what does state of the art training look
by blackeyeblitzar 3y ago
> It looks like a mid-level implementations of training and inference
I’m not familiar with how any of this works but what does state of the art training look like? Almost no models release their training source code or data sets or pre processing or evaluation code. So is it known what the high level implementation even is?
- spott 3y agohttps://github.com/NVIDIA/Megatron-LM https://github.com/NVIDIA/Megatron-LM This is probably a good baseline to start thinking about LLM training at scale.