22 ms·
I see this as the future of machine learning. Using a single bit to communicate weight updates during the learning process reduces bandwidth required, and allo
by londons_explore 5y ago
I see this as the future of machine learning.
Using a single bit to communicate weight updates during the learning process reduces bandwidth required, and allows highly parallel training.
I suspect in the future we'll even see methods of sub-1-bit weight updates to further decrease bandwidth requirements to keep massive models approximately in sync between distant learning nodes.
- asxd 5y agoWhat would a sub 1-bit weight update look like? I’m having trouble wrapping my head around the concept of (what I assume would be) a fraction of a bit?
- londons_explore 5y agoImagine we have 2 weights we want to update. For each weight, choose a pseudo-random number known by both sender and receiver. If a "1" bit is received, update both weights with the random number. If a zero is received, don't. After enough iterations, each weight can converge on any value.
- fennecfoxen 5y agoFirst imagine a less-than-bit fact. Suppose you have a set of 1024 computers and you want to pick out a specific computer based on many facts about a single computer. 1 in 4 computers runs Linux and the rest run Windows. Knowing a computer runs Windows is about half a bit of the information necessary to identify the computer. A less-than-bit update can be combined with another such an update to encode a single bit. For instance, my protocol could use a bit to indicate “the computer runs Linux, or it is one of the 256 Windows computers in this list”. You know the computer has a 50% chance of running Windows.