4 ms·
> 1 bit per weight does this basically correspond to moving each weight either up or down by a fixed amount? I'm a bit surprised you don't at least need a "sta
by radarsat1 2y ago
> 1 bit per weight
does this basically correspond to moving each weight either up or down by a fixed amount? I'm a bit surprised you don't at least need a "stay same" bit, but i suppose it could balance out over multiple iterations.
Interesting that it works at all. Although, thinking on it, I could see it maybe even having a nice regularizing effect where every layer would end up have similar weight magnitudes. (like projecting onto the local n-ball as mentioned in a paper posted recently on HN)
- f_devd 2y agoIt has been more formally studied in signSGD[0], and empirically it's comparable to Adam in terms of behavior. [0]: https://arxiv.org/pdf/1802.04434 https://arxiv.org/pdf/1802.04434
- londons_explore 2y agoThis is for keeping the weight vectors in sync between two machines. The weight vectors themselves are regular floats. But the data exchanged between the machines is 1 bit. Basically, you keep track of changes to the weight vector which hasn't yet been propagated to the other machine. You quantize this to 1 bit per weight (ie. a sign bit) and send it, together with a single scale factor X, accumulating the quantization error for the next sync iteration. You choose X to be the RMS or some similar metric of the accumulated error.