3 ms·
In the download script, it skips parts of the model (02 and 83); any ML people have ideas why you'd do that?
by idealmedtech 4y ago
In the download script, it skips parts of the model (02 and 83); any ML people have ideas why you'd do that?
- hansonw 4y agoIt appears the indexing for the model parts is deliberately not contiguous; the 03-82 range represents the main 80 transformer layers. https://github.com/yandex/YaLM-100B/blob/main/megatron_lm/megatron/checkpointing.py#L425-L436 https://github.com/yandex/YaLM-100B/blob/main/megatron_lm/me...
- idealmedtech 4y agoThat makes sense, thanks for clearing it up!