Y
HN Search
Hacker News Search
new
|
comments
|
top
|
jobs
easysnap
searching PlanetScale…
1.
▲
2.
▲
3.
▲
4.
▲
5.
▲
6.
▲
3 ms
·
1.
▲
by
easysnap
7y ago
ZeRO is mainly a clever improvement that moves optimizer computation into the 2 phases of Ring-AllReduce. It greatly helps Adam and similar optimizers to reduce per-GPU memory overhead. The naive approach, as used in the well-known Megatron