3 ms·
XNOR and DoReFa take pretty big hits though, and they can't be easily trained without full precision, or at least "shadow weights".
by deepnotderp 10y ago
XNOR and DoReFa take pretty big hits though, and they can't be easily trained without full precision, or at least "shadow weights".
- paulsutter 10y agoFor XNORnet, yes, but DoReFa has training with low bitwidth activations[1]. I don't cite these papers as proof that few bits are needed, but as evidence that precision is an open question. [1] "We propose DoReFa-Net, a method to train convolutional neural networks that have low bitwidth weights and activations using low bitwidth parameter gradients. In particular, during backward pass, parameter gradients are stochastically quantized to low bitwidth numbers before being propagated to convolutional layers" (from the abstract)
- p1esk 10y agoDoReFa needs a custom scaling factor for each image when quantizing gradients.
- deepnotderp 10y agoWhat's wrong with that?
- p1esk 10y agoTheir formula for quantizing gradients involves quite a bit of extra computation. It's not clear to me how much it complicates an actual implementation (in software, or hardware). To me, the most interesting question is if we can do training using just 8 bits (weights, activations, and gradients), without all that acrobatics. If so, then we can get another significant (and free!) speed up from GPUs.
- emcq 10y agoWhat's wrong with training in full precision? There's a huge market for efficient inference in robotics, devices, and autonomous vehicles.