4 ms·

Is this because averages are weighed less (less sensitive) as the total sample size increases?

londons_explore 4y ago
Yes. In a dense everything to everything neural network layer, the number of 'inputs' to a node is proportional to the square root of the number of weights. Therefore, assuming quantization noise is uncorrelated, as the number of weights doubles, the number of inputs goes up by sqrt(2), and the (normalized) noise goes down by a factor of 2*(sqrt(2)). So, as a rule of thumb, you can remove 1 bit of precision of the weights for every 4x increase in the number of weights. All this assumes weights and activations are uncorrelated random variables - which may not hold true.
- londons_explore 4y ago
  Something is wrong with this math... by your logic I could scale the network up big enough that I could quantize the weights down to zero bits...
  - Robotbeat 4y ago
    Rules of thumb typically are just first order approximations which by definition are not guaranteed to hold far from their point of interest (or point of tangency).
  - sebzim4500 4y ago
    Having fewer than 1 bit per weight is not absurd. E.g. you can use 2 bits to represent 3 'weights' if you insist that at most one of the weights is allowed to exist. If you try to order nodes so that adjacent nodes are uncorrelated the performance loss might be manageable. People are already doing stuff like this (see sparsification) so it is conceivable to me that this is just what networks will look like in a few years.
    - magicalhippo 4y ago
      > If you try to order nodes so that adjacent nodes are uncorrelated the performance loss might be manageable. shower thought In graphics we use barycentric coordinates to encode the position within an arbitrary triangle using two coordinates (u,v), with the third being constrained to be 1-u-v. If you order nodes to be correlated, could you use a similar trick to encode three weights for the price of two?
      - hgsgm 4y ago
        Yes, it's the same thing.

Yes. In a dense everything to everything neural network layer, the number of 'inputs' to a node is proportional to the square root of the number of weights. Therefore, assuming quantization noise is uncorrelated, as the number of weights doubles, the number of inputs goes up by sqrt(2), and the (normalized) noise goes down by a factor of 2*(sqrt(2)). So, as a rule of thumb, you can remove 1 bit of precision of the weights for every 4x increase in the number of weights. All this assumes weights and activations are uncorrelated random variables - which may not hold true.

londons_explore 4y ago
Something is wrong with this math... by your logic I could scale the network up big enough that I could quantize the weights down to zero bits...
- Robotbeat 4y ago
  Rules of thumb typically are just first order approximations which by definition are not guaranteed to hold far from their point of interest (or point of tangency).
- sebzim4500 4y ago
  Having fewer than 1 bit per weight is not absurd. E.g. you can use 2 bits to represent 3 'weights' if you insist that at most one of the weights is allowed to exist. If you try to order nodes so that adjacent nodes are uncorrelated the performance loss might be manageable. People are already doing stuff like this (see sparsification) so it is conceivable to me that this is just what networks will look like in a few years.
  - magicalhippo 4y ago
    > If you try to order nodes so that adjacent nodes are uncorrelated the performance loss might be manageable. shower thought In graphics we use barycentric coordinates to encode the position within an arbitrary triangle using two coordinates (u,v), with the third being constrained to be 1-u-v. If you order nodes to be correlated, could you use a similar trick to encode three weights for the price of two?
    - hgsgm 4y ago
      Yes, it's the same thing.