7 ms·
Bit is all we need: binary normalized neural networks
- modeless 1y ago> each parameter exists in two forms simultaneously during training: a full-precision 32-bit floating-point value (p) used for gradient updates, and its binarized counterpart (pb) used for forward computations So this is only for inference. Also activations aren't quantized, I think?
- nighthawk454 1y agoYeah, but it’s ’quantization aware’ during training too, which presumably is what allows the quantization at inference to work
- benob 1y agoI wonder if one could store only the binary representation at training and sample a floating point representation (both weights and gradient) during backprop.
- adastra22 1y agoBack propagation on random data that is then thrown away would be pretty useless.
- jampekka 1y ago> Also activations aren't quantized, I think? The very last conclusion: "Future work will focus on the implementation of binary normalization layers using single-bit arrays operations, as well as on quantizing layer activations to 8 or 16-bit precision. These improvements are expected to further enhance the efficiency and performance of the binary neural network models."
- kouteiheika 1y agoYou don't necessarily have to store the parameters in fp32 for gradient updates; I experimented with it and got it working (all parameter full fine-tuning) with parameters being as low as 3-bit (a little bit more than 3-bit, because the block-wise scales were higher precision), which is essentially as low as you can go before "normal" training starts breaking down.
- noosphr 1y agoYes, that's been the downside of these forever. If you use quantized differentiation you can get away with using integers for gradient updates. Explaining how takes a paper and in the end it doesn't even work very well. At university, way back at the end of the last AI winter, I ended up using genetic algorithms to train the models. It was very interesting because weights were trained along with hyper parameters. It was no where near practical because gradient descent is so much better at getting real world results in reasonable time frames - surprisingly because it's more memory efficient.
- AmazingTurtle 1y agoI'm gonna refer to this one here: https://news.ycombinator.com/item?id=45361007 https://news.ycombinator.com/item?id=45361007
- s20n 1y agoAttention Is All You Need - The Beatles ft. Charlie Puth
- mnky9800n 1y agoYes calling your paper this now makes me think your paper has no interesting results. It is kind of the opposite of what’s intended.
- gloomyday 1y agoThis naming trend has been going for 8 years. Incredible.
- IshKebab 1y agoIt's on my naughty list, together with "... considered harmful", "The unreasonable effectiveness of ...", "... for fun and profit", "Falsehoods programmers believe about ...", "The rise and fall of ...".
- fxtentacle 1y agoThese techniques are not new. And the reason why they’re usually not used is on page 9 in the paper. They require about 10x as many training iterations.
- typpilol 1y agoYea I saw that training perplexity and thought hmmm...
- shomp 1y agoTurns out using floats is a feature and not a bug?
- Dylan16807 1y agoNo, I don't think so, in that I don't think anyone has ever called that a bug.
- shomp 1y agoIn the paper summary they did not call it a bug explicitly, but they do say there are 32x improvements in using single bits instead.
- reactordev 1y agoTo memory, sure. At the cost of 32x slower speeds.
- Dylan16807 1y agoThat's an obvious exaggeration. The competition is using smaller weights already, some of which are floating point and some of which aren't. And they use full size floats for training.
- imtringued 1y agoThat means their paper is actually worse than SOTA, which is concerned with training in fp4 natively without full precision [0] for QAT. [0] "full precision" in ML usually means 16 bit floats like bfloat16
- pixelpoet 1y agoThe critical "1" is missing from the title...
- Dylan16807 1y ago"Bit" being singular gets the intent across just fine.
- pixelpoet 1y agoDisagree
- hirako2000 1y agoI also* disagree, otherwise we would say, kilo of meat is enough?
- pixelpoet 1y agoYes, that's the point I was making, and the other person said it's fine without saying how many bits, not me.
- hirako2000 1y agoMy bad, I meant that I disagree with parent, I edited it. I agree with you.
- user823749 1y agoYes, just like in "16 bit integer". No confusion at all.
- jongjong 1y agoThis reminds me if my university days. For one of the assignments, we had to write our own ANN from scratch for handwriting recognition and we implemented a step activation function because that was easier than sigmoid; basically each layer would output one or zero though I guess the weights themselves were scalars. It's just the node outputs which were 1 or 0... But this was convenient because the output of the final layer could be interpreted as a binary which could be converted straight into an ASCII character for comparison and backpropagation.
- meindnoch 1y ago>could be interpreted as a binary which could be converted straight into an ASCII character for comparison and backpropagation. There's nothing to backpropagate with a step function. The derivative is zero everywhere.
- steppi 1y agoIt sounds like jongjong was probably using surrogate gradients. You keep the step activation in the forward pass but replace with a smooth approximation in the backwards pass.
- bjourne 1y agoYeah, but then there is no performance benefit over plain old sgd.
- steppi 1y agoYeah, I think surrogate gradients are usually used to train spiking neural nets where the binary nature is considered an end in itself, for reasons of biological plausibility or something. Not for any performance benefits. It's not an area I really know that much about though.
- nickpsecurity 1y ago
- cindyllm 1y ago[dead]
- forntentacl 1y agoThis paper ignores 50+ years of research in the domain of quantized networks, quantized training algorithms, and reaches wrong conclusions out of sheer ignorance. TLDR abstract of a draft paper I wrote years ago, for those interested in the real limits of quantized networks: We investigate the storage capacity of single‐layer threshold neurons under three synaptic precision regimes—binary (1‐bit), ternary (≈1.585‐bit), and quaternary (2‐bit)—from both information‐theoretic and algorithmic standpoints. While the Gardner bound stipulates maximal loads of α=0.83, 1.5 and 2.0 patterns per weight for the three regimes, practical algorithms only reach α_alg≈0.72, 1.0 and 2.0, respectively. By converting these densities into storage‐efficiency metrics—bits of synaptic memory per stored pattern—we demonstrate that only quaternary weights achieve the theoretical optimum in realistic settings, requiring exactly 1 bit of memory per pattern. Binary and ternary schemes incur 39 % and 58 % overheads, respectively.
- naasking 1y agoIs this actually equivalent to classical forms of quantization though? The paper has extensive discussion of quantization on page 2 and 3. This paper is not just a rehash of earlier work, but pushes the single bit precision to more parts of the system.
- throwaway34609 1y ago[dead]
- thijson 1y agoHow does this compare to: https://arxiv.org/pdf/1811.11431 https://arxiv.org/pdf/1811.11431