4 ms·

> Just try training a neural network to compute sin(x)! This is also the classic demo case for LSTMs, I have a notebook open right now that as a LSTM learning

> Just try training a neural network to compute sin(x)! This is also the classic demo case for LSTMs, I have a notebook open right now that as a LSTM learning the sine function quite well with 32 dimensional state vector. However the authors point still stands. Neural Networks are not great at computing arbitrary functions where the output is an unbounded real number. The sine function is still limited to range that is bounded to [0,1]. A better example of where NNs really fail is when trying to learn tricky to implement the normal quantile function (inverse CDF). This would be an excellent place for function approximation because a.) generating training data is easy (just run random numbers into the CDF and reverse these arguments into a NN) b.) manually writing the quantile function from scratch is a pain since it involves the inverse error function which is very annoying to implement from scratch. You can learn the standard quantile function for mean=0, sd=1, however if you try to generalize this to taking not only the desired quantile but an arbitrary mean and standard deviation you will not learn anything useful. It's a bit of a shame that neural networks are weak in this area because it would be incredible to have a good tool to approximate inverse functions in general. The fact that we almost never see neural networks being used as a tool for this type of work is evidence of this limitation. In general if you're problem can't be modeled where the output is some vector of probabilities it's not a great fit for NNs.

steppi 4y ago
It's a bit of a shame that neural networks are weak in this area because it would be incredible to have a good tool to approximate inverse functions in general. The fact that we almost never see neural networks being used as a tool for this type of work is evidence of this limitation. It's funny you say. I haven't actually used NNs much in my research but my background is in math and in my spare time I'm a maintainer for SciPy working on special and statistical functions, often the exact kind of stuff you mentioned. I might take up your challenge and write a blog post about it or something if I get any success. Anyway, I wasn’t trying to invalidate the author’s general point, just point out a fun fact about his example.
- PheonixPharts 4y ago
  > I might take up your challenge and write a blog post about it or something if I get any success. Success or failure, I would really enjoy seeing that write up! I would be even more excited to be proven wrong. The promise of "universal function approximator" is very temping. My personal dream would be to have it so one could essentially run scipy in reverse and learn the entire library with a NN. Even in the case I gave, the idea that you could learn an arbitrary quantile function means you could also arbitrarily learn a sampler for any distribution, since all you have to do compose a uniform sampler with whatever quantile function you learned. Of course for this example I'm using "solved" using a similar approach with variational inference (pyro has a write up on it: https://pyro.ai/examples/svi_part_i.html https://pyro.ai/examples/svi_part_i.html, you might find David Blei's "Variational Inference: A Review for Statisticians" useful as well https://arxiv.org/abs/1601.00670 https://arxiv.org/abs/1601.00670)
  - steppi 4y ago
    What a coincidence. Pyro has basically become my go to for Bayesian learning and I know those docs in and out. I’ve also skimmed that David Blei article before. Small world. I’m not very confident of success but it seems like an interesting path to explore.
mumblemumble 4y ago
> LSTM learning the sine function quite well with 32 dimensional state vector. I'd say that, from "LSTM" and "32 dimensional state vector" alone, the author's point still stands. That may not be a huge network by contemporary deep learning standards, but it's still a pretty darned expensive way to compute sine. That's 100% in line with the original assertion that ANNs can learn any function, but they can't necessarily do so efficiently.
Der_Einzige 4y ago
Is it that neural networks are weak or is it that gradient descent/backprop doesn't do well for finding good local solutions for these kinds of problems?
- steppi 4y ago
  My understanding is that it's because extrapolation is difficult in general. Extrapolating the behavior of a periodic function like sine is one thing, but extrapolating the behavior of a 1d function that tends to infinity is another challenge altogether. I happen to be very familiar with the implementation of the inverse CDF of the normal distribution though. My plan is to cook a lot of prior knowledge into the NN. Even if I get it working, the end result will likely seem more like sleight of hand than something truly impressive.

steppi 4y ago

It's a bit of a shame that neural networks are weak in this area because it would be incredible to have a good tool to approximate inverse functions in general. The fact that we almost never see neural networks being used as a tool for this type of work is evidence of this limitation. It's funny you say. I haven't actually used NNs much in my research but my background is in math and in my spare time I'm a maintainer for SciPy working on special and statistical functions, often the exact kind of stuff you mentioned. I might take up your challenge and write a blog post about it or something if I get any success. Anyway, I wasn’t trying to invalidate the author’s general point, just point out a fun fact about his example.

PheonixPharts 4y ago
> I might take up your challenge and write a blog post about it or something if I get any success. Success or failure, I would really enjoy seeing that write up! I would be even more excited to be proven wrong. The promise of "universal function approximator" is very temping. My personal dream would be to have it so one could essentially run scipy in reverse and learn the entire library with a NN. Even in the case I gave, the idea that you could learn an arbitrary quantile function means you could also arbitrarily learn a sampler for any distribution, since all you have to do compose a uniform sampler with whatever quantile function you learned. Of course for this example I'm using "solved" using a similar approach with variational inference (pyro has a write up on it: https://pyro.ai/examples/svi_part_i.html https://pyro.ai/examples/svi_part_i.html, you might find David Blei's "Variational Inference: A Review for Statisticians" useful as well https://arxiv.org/abs/1601.00670 https://arxiv.org/abs/1601.00670)
- steppi 4y ago
  What a coincidence. Pyro has basically become my go to for Bayesian learning and I know those docs in and out. I’ve also skimmed that David Blei article before. Small world. I’m not very confident of success but it seems like an interesting path to explore.

mumblemumble 4y ago

> LSTM learning the sine function quite well with 32 dimensional state vector. I'd say that, from "LSTM" and "32 dimensional state vector" alone, the author's point still stands. That may not be a huge network by contemporary deep learning standards, but it's still a pretty darned expensive way to compute sine. That's 100% in line with the original assertion that ANNs can learn any function, but they can't necessarily do so efficiently.

Der_Einzige 4y ago

Is it that neural networks are weak or is it that gradient descent/backprop doesn't do well for finding good local solutions for these kinds of problems?

steppi 4y ago
My understanding is that it's because extrapolation is difficult in general. Extrapolating the behavior of a periodic function like sine is one thing, but extrapolating the behavior of a 1d function that tends to infinity is another challenge altogether. I happen to be very familiar with the implementation of the inverse CDF of the normal distribution though. My plan is to cook a lot of prior knowledge into the NN. Even if I get it working, the end result will likely seem more like sleight of hand than something truly impressive.