3 ms·

I'm always surprised that the symbolic math paradigms aren't used more. I suppose most cost functions just aren't continuous? That said, seems most of what is

by taeric 4y ago

I'm always surprised that the symbolic math paradigms aren't used more. I suppose most cost functions just aren't continuous? That said, seems most of what is important in today's code is the ability to drive a gpu. Don't know why that couldn't be done from a lisp.

pgorczak 4y ago
Neural nets do use extremely long compositions of nonlinear functions whose gradients need to be computed over and over. I guess because chain rule (automatic differentiation) works so well, most of the libraries in use are only “a little” symbolic in that they can automatically figure out first derivatives for symbolic input and output variables.
- taeric 4y ago
  I thought automatic differentiation is a little different? Will have to look again. I thought the norm for a while was to manually enter the gradient, if possible, or to use a calculated one at each step.
  - mirker 4y ago
    The norm is to use library functions like addition, multiply, power, etc., which have hardcoded derivatives. Then the derivative is computed “symbolically” by composing these base derivatives, which is essentially an exercise of implementing calculator logic. I use scare quotes because there are numerical problems with some functions that have to be hacked around with more manual derivatives, like the cross-entropy/softmax function. In those cases, people suggest to use a different library function special-cased out for the situation, which is not really “symbolic” in the typical mathematical sense.
ianbicking 4y ago
I'm not sure if anything has or will happen with this, but this seemed like an interesting symbolic extension of neural nets: https://syncedreview.com/2020/11/19/facebook-building-automatic-differentiation-system-for-kotlin-programming-language/ https://syncedreview.com/2020/11/19/facebook-building-automa... At least my take on it is: what if you simulate a domain (physics being the most tractable) then can you train using differentials of the model itself and not just differentiating the internal neuron estimators? Lots of open questions... like what's the driving force that asks for an effect that calls for differentiating the model? We kind of expect the training process to just recreate everything internally. Now it has me thinking about "coprocessors"... a lot of the domain-specific math is given to the NN as input features, but what if the NN could select inputs to those algorithm? That would give the NN the ability to speculate. Of course you'd need to also differentiate those algorithms. Now that I'm thinking about it, I guess this is just another activation function, just one that is domain-specific. Are there approaches that use eclectic activation functions? Like in each layer 1-10 is one activation function, 11-20 is another, and then you finish off with tanh or relu or some other genetic activation function.

by taeric 4y ago

pgorczak 4y ago

Neural nets do use extremely long compositions of nonlinear functions whose gradients need to be computed over and over. I guess because chain rule (automatic differentiation) works so well, most of the libraries in use are only “a little” symbolic in that they can automatically figure out first derivatives for symbolic input and output variables.

taeric 4y ago
I thought automatic differentiation is a little different? Will have to look again. I thought the norm for a while was to manually enter the gradient, if possible, or to use a calculated one at each step.
- mirker 4y ago
  The norm is to use library functions like addition, multiply, power, etc., which have hardcoded derivatives. Then the derivative is computed “symbolically” by composing these base derivatives, which is essentially an exercise of implementing calculator logic. I use scare quotes because there are numerical problems with some functions that have to be hacked around with more manual derivatives, like the cross-entropy/softmax function. In those cases, people suggest to use a different library function special-cased out for the situation, which is not really “symbolic” in the typical mathematical sense.

ianbicking 4y ago

I'm not sure if anything has or will happen with this, but this seemed like an interesting symbolic extension of neural nets: https://syncedreview.com/2020/11/19/facebook-building-automatic-differentiation-system-for-kotlin-programming-language/ https://syncedreview.com/2020/11/19/facebook-building-automa... At least my take on it is: what if you simulate a domain (physics being the most tractable) then can you train using differentials of the model itself and not just differentiating the internal neuron estimators? Lots of open questions... like what's the driving force that asks for an effect that calls for differentiating the model? We kind of expect the training process to just recreate everything internally. Now it has me thinking about "coprocessors"... a lot of the domain-specific math is given to the NN as input features, but what if the NN could select inputs to those algorithm? That would give the NN the ability to speculate. Of course you'd need to also differentiate those algorithms. Now that I'm thinking about it, I guess this is just another activation function, just one that is domain-specific. Are there approaches that use eclectic activation functions? Like in each layer 1-10 is one activation function, 11-20 is another, and then you finish off with tanh or relu or some other genetic activation function.