3 ms·

sounds very similar to https://docs.pytorch.org/docs/stable/distributed.fsdp.fully_shard.html#torch.distributed.fsdp.CPUOffloadPolicy https://docs.pytorch.org/d

sounds very similar to https://docs.pytorch.org/docs/stable/distributed.fsdp.fully_shard.html#torch.distributed.fsdp.CPUOffloadPolicy https://docs.pytorch.org/docs/stable/distributed.fsdp.fully_... i wonder how much this could be replicated using only this pytorch primitive

RandyOrion 6mo ago
Check out Fig. 6 in this paper, it shows the comparison between the proposed method and pytorch native FSDP offload method.