I just recently stumbled upon a practical example of the first item in your list: try approximating sin(FREQ_SCALE * input) using a neural network and gradient

I just recently stumbled upon a practical example of the first item in your list: try approximating sin(FREQ_SCALE * input) using a neural network and gradient descent for various FREQ_SCALE and ranges of input. You will find it to be completely powerless even though you can easily construct a single parameter "NN" with sin activation, that reproduces the function exactly when the parameter is initialized with FREQ_SCALE. The problem extends to any periodic functions. I am working on a blog post about it.