10 ms·

This makes me think: the math behind neural nets is kind of arbitrary. The actually important thing is (a) getting a result out, and (b) being able to nudge the

This makes me think: the math behind neural nets is kind of arbitrary. The actually important thing is (a) getting a result out, and (b) being able to nudge the weights to get the result closer to what you want. That doesn't have to be built on "math" or "numbers" but could be built on bits directly so long as the operations satisfy the goals.

throwbadubadu 3y ago
> built on "math" or "numbers" but could be built on bits That reads strange, what is the difference besides the range of numbers?
- Hydraulix989 3y ago
  Maybe bitwise operations instead of arithmetic?
  - riscy 3y ago
    arithmetic is implemented as bitwise operations so i’m still confused. the lower the precision / smaller range of values you choose to represent means fewer bitwise operations are required.
    - Hydraulix989 3y ago
      Maybe something like this: https://arxiv.org/abs/1601.06071 https://arxiv.org/abs/1601.06071
blastonico 3y ago
I would call it applied statistics :-D
david-gpu 3y ago
I used to design low-precision hardware to train AIs. > The actually important thing is (a) getting a result out, and (b) being able to nudge the weights to get the result closer to what you want That is accurate. > That doesn't have to be built on "math" or "numbers" but could be built on bits directly so long as the operations satisfy the goals Some people have published research where the parameters or activations are binary, sometimes in a stochastic fashion. Typically either the AI doesn't converge or it converges to a rather suboptimal solution. Also, just because the weights or activations are binary it doesn't mean there's no math involved. You need some sort of a mathematical model to figure out how to tweak the parameters of the model in order to improve its fitness to the data.
- sigmoid10 3y ago
  The keyword the original comment was missing is differentiability. This is what a) and b) imply because it is how you ensure that you can train your model using gradient descent or whatever. So you need a particular kind of math to train these models using the methods we have today. There's quite a bit of research into other methods for training neural networks that don't come with this constraint, because it's obvious the human brain doesn't work that way. But this is early research and very far from being as useful.
  - stevenhuang 3y ago
    I wouldn't say it's obvious the brain doesn't work this way. See the free energy principle, one of the leading theories of how our brain works. > The free energy principle is a theoretical framework suggesting that the brain reduces surprise or uncertainty by making predictions based on internal models and updating them using sensory input. It highlights the brain's objective of aligning its internal model with the external world to enhance prediction accuracy Making predictions and then correcting for them in some way is a pretty low bar and it's often achieved through differentiability. I wouldn't be surprised if relevant neurological structures exist in our brains that do this.
    - sigmoid10 3y ago
      There are reasons to believe it could work similarly, but they are related to computational efficiency and out lack of knowledge regarding more efficient methods. But I would not bet on our own ignorance as an explanation.
_nalply 3y ago
Probably even 4 bit values suffice. Invent some binary operations that exhibit some smoothness or continuousness such that gradient descent still works, because mostly a small change should have only a small effect. I think mathematicians would say functions that are differentiable.
- tayo42 3y ago
  Quantization is done with 4 bits, unless you meant something else?
  - _nalply 3y ago
    Perhaps quantization with 4 bits is a special case of what I meant. With 4 bits you have 16 different weight values but these values must not be quantized. They could be anything provided gradient descent is possible. Have some combination of binary operations instead of mathematical operations. What these operations do exactly, does not really matter, as long as gradient descent/backpropagation stays possible. In other words, the model will find a way to use the weights correctly.

throwbadubadu 3y ago

> built on "math" or "numbers" but could be built on bits That reads strange, what is the difference besides the range of numbers?

Hydraulix989 3y ago
Maybe bitwise operations instead of arithmetic?
- riscy 3y ago
  arithmetic is implemented as bitwise operations so i’m still confused. the lower the precision / smaller range of values you choose to represent means fewer bitwise operations are required.
  - Hydraulix989 3y ago
    Maybe something like this: https://arxiv.org/abs/1601.06071 https://arxiv.org/abs/1601.06071

blastonico 3y ago

I would call it applied statistics :-D

david-gpu 3y ago

I used to design low-precision hardware to train AIs. > The actually important thing is (a) getting a result out, and (b) being able to nudge the weights to get the result closer to what you want That is accurate. > That doesn't have to be built on "math" or "numbers" but could be built on bits directly so long as the operations satisfy the goals Some people have published research where the parameters or activations are binary, sometimes in a stochastic fashion. Typically either the AI doesn't converge or it converges to a rather suboptimal solution. Also, just because the weights or activations are binary it doesn't mean there's no math involved. You need some sort of a mathematical model to figure out how to tweak the parameters of the model in order to improve its fitness to the data.

sigmoid10 3y ago
The keyword the original comment was missing is differentiability. This is what a) and b) imply because it is how you ensure that you can train your model using gradient descent or whatever. So you need a particular kind of math to train these models using the methods we have today. There's quite a bit of research into other methods for training neural networks that don't come with this constraint, because it's obvious the human brain doesn't work that way. But this is early research and very far from being as useful.
- stevenhuang 3y ago
  I wouldn't say it's obvious the brain doesn't work this way. See the free energy principle, one of the leading theories of how our brain works. > The free energy principle is a theoretical framework suggesting that the brain reduces surprise or uncertainty by making predictions based on internal models and updating them using sensory input. It highlights the brain's objective of aligning its internal model with the external world to enhance prediction accuracy Making predictions and then correcting for them in some way is a pretty low bar and it's often achieved through differentiability. I wouldn't be surprised if relevant neurological structures exist in our brains that do this.
  - sigmoid10 3y ago
    There are reasons to believe it could work similarly, but they are related to computational efficiency and out lack of knowledge regarding more efficient methods. But I would not bet on our own ignorance as an explanation.

_nalply 3y ago

Probably even 4 bit values suffice. Invent some binary operations that exhibit some smoothness or continuousness such that gradient descent still works, because mostly a small change should have only a small effect. I think mathematicians would say functions that are differentiable.

tayo42 3y ago
Quantization is done with 4 bits, unless you meant something else?
- _nalply 3y ago
  Perhaps quantization with 4 bits is a special case of what I meant. With 4 bits you have 16 different weight values but these values must not be quantized. They could be anything provided gradient descent is possible. Have some combination of binary operations instead of mathematical operations. What these operations do exactly, does not really matter, as long as gradient descent/backpropagation stays possible. In other words, the model will find a way to use the weights correctly.