3 ms·

Fun fact - single bit neural networks are decision trees. In theory, this means you can 'compile' most neural networks into chains of if-else statements but it

• Almondsetat 8mo ago
  Do you know of any software that does this? Or any papers on the matter? It could be a fun weekend project
  • tomashubelbauer 8mo ago
    Made me think of https://github.com/xoreaxeaxeax/movfuscator https://github.com/xoreaxeaxeax/movfuscator. Would be definitely cool to see it realized even if it would be incredibly impractical (probably).
  • fooker 8mo ago
    I think any quantization approach should work, not an expert on this.
• deleted 8mo ago
  [deleted]
• smokel 8mo ago
  > single bit neural networks are decision trees. I didn't exactly understood what was meant here, so I went out and read a little. There is an interesting paper called "Neural Networks are Decision Trees" [1]. Thing is, this does not imply a nice mapping of neural networks onto decision trees. The trees that correspond to the neural networks are huge. And I get the idea that the paper is stretching the concept of decision trees a bit. Also, I still don't know exactly what you mean, so would you care to elaborate a bit? :) [1] https://arxiv.org/pdf/2210.05189 https://arxiv.org/pdf/2210.05189
  • lioeters 8mo ago
    Closest thing I found was: Single Bit Neural Nets Did Not Work - https://fpga.mit.edu/videos/2023/team04/report.pdf https://fpga.mit.edu/videos/2023/team04/report.pdf > We originally planned to make and train a neural network with single bit activations, weights, and gradients, but unfortunately the neural network did not train very well. We were left with a peculiar looking CPU that we tried adapting to mine bitcoin and run Brainfuck.
  • fooker 8mo ago
    > I still don't know exactly what you mean Straight forward quantization, just to one bit instead of 8 or 16 or 32. Training a one bit neural network from scratch is apparently an unsolved problem though. > The trees that correspond to the neural networks are huge. Yes, if the task is inherently 'fuzzy'. Many neural networks are effectively large decision trees in disguise and those are the ones which have potential with this kind of approach.
    • fc417fc802 8mo ago
      > Training a one bit neural network from scratch is apparently an unsolved problem though. I don't think it's correct to call it unsolved. The established methods are much less efficient than those for "regular" neural nets but they do exist. Also note that the usual approach when going binary is to make the units stochastic. https://en.wikipedia.org/wiki/Boltzmann_machine#Deep_Boltzmann_machine https://en.wikipedia.org/wiki/Boltzmann_machine#Deep_Boltzma...
      • fooker 8mo ago
        Interesting. By unsolved I guess I meant: this looks like it should be easy and efficient but we don't know how to do it yet. Usually this means we are missing some important science in the classification/complexity of problems. I don't know what it could be.
        • fc417fc802 8mo ago
          Perhaps. It's also possible that the approach simply precludes the use of the best tool for the job. Backprop is quite powerful and it just doesn't work in the face of heavy quantization. Whereas if you're already using evolution strategies or a genetic algorithm or similar then I don't expect changing the bit width (or pretty much anything else) to make any difference to the overall training efficiency (which is presumably already abysmal outside of a few specific domains such as RL applied to a sufficiently ambiguous continuous control problem).