3 ms·
I built this a few years ago: https://github.com/Hello1024/shared-tensor https://github.com/Hello1024/shared-tensor It does updates to weights based on 1 bit
by londons_explore 6y ago
I built this a few years ago:
https://github.com/Hello1024/shared-tensor https://github.com/Hello1024/shared-tensor
It does updates to weights based on 1 bit precision updates each iteration.
It would be fairly trivial to go to less than 1 bit precision too - simply set some threshold (eg 3), and wherever the difference between the weight on the server and the client is greater than 3, transmit a binary "1", else send a binary "0". Then entropy code all the resulting binary.
By adjusting the threshold up and down, you trade off the size of the data to send Vs precision.
- 0-_-0 6y agoI read a paper that did exactly what you describe but of course I can't find it now...