I'm not sure if anything has or will happen with this, but this seemed like an interesting symbolic extension of neural nets: https://syncedreview.com/2020/11/1

I'm not sure if anything has or will happen with this, but this seemed like an interesting symbolic extension of neural nets: https://syncedreview.com/2020/11/19/facebook-building-automatic-differentiation-system-for-kotlin-programming-language/ https://syncedreview.com/2020/11/19/facebook-building-automa... At least my take on it is: what if you simulate a domain (physics being the most tractable) then can you train using differentials of the model itself and not just differentiating the internal neuron estimators? Lots of open questions... like what's the driving force that asks for an effect that calls for differentiating the model? We kind of expect the training process to just recreate everything internally. Now it has me thinking about "coprocessors"... a lot of the domain-specific math is given to the NN as input features, but what if the NN could select inputs to those algorithm? That would give the NN the ability to speculate. Of course you'd need to also differentiate those algorithms. Now that I'm thinking about it, I guess this is just another activation function, just one that is domain-specific. Are there approaches that use eclectic activation functions? Like in each layer 1-10 is one activation function, 11-20 is another, and then you finish off with tanh or relu or some other genetic activation function.