2 ms·

It's a bit of a shame that neural networks are weak in this area because it would be incredible to have a good tool to approximate inverse functions in general.

by steppi 4y ago

It's a bit of a shame that neural networks are weak in this area because it would be incredible to have a good tool to approximate inverse functions in general. The fact that we almost never see neural networks being used as a tool for this type of work is evidence of this limitation. It's funny you say. I haven't actually used NNs much in my research but my background is in math and in my spare time I'm a maintainer for SciPy working on special and statistical functions, often the exact kind of stuff you mentioned. I might take up your challenge and write a blog post about it or something if I get any success. Anyway, I wasn’t trying to invalidate the author’s general point, just point out a fun fact about his example.

PheonixPharts 4y ago
> I might take up your challenge and write a blog post about it or something if I get any success. Success or failure, I would really enjoy seeing that write up! I would be even more excited to be proven wrong. The promise of "universal function approximator" is very temping. My personal dream would be to have it so one could essentially run scipy in reverse and learn the entire library with a NN. Even in the case I gave, the idea that you could learn an arbitrary quantile function means you could also arbitrarily learn a sampler for any distribution, since all you have to do compose a uniform sampler with whatever quantile function you learned. Of course for this example I'm using "solved" using a similar approach with variational inference (pyro has a write up on it: https://pyro.ai/examples/svi_part_i.html https://pyro.ai/examples/svi_part_i.html, you might find David Blei's "Variational Inference: A Review for Statisticians" useful as well https://arxiv.org/abs/1601.00670 https://arxiv.org/abs/1601.00670)
- steppi 4y ago
  What a coincidence. Pyro has basically become my go to for Bayesian learning and I know those docs in and out. I’ve also skimmed that David Blei article before. Small world. I’m not very confident of success but it seems like an interesting path to explore.

by steppi 4y ago

PheonixPharts 4y ago

> I might take up your challenge and write a blog post about it or something if I get any success. Success or failure, I would really enjoy seeing that write up! I would be even more excited to be proven wrong. The promise of "universal function approximator" is very temping. My personal dream would be to have it so one could essentially run scipy in reverse and learn the entire library with a NN. Even in the case I gave, the idea that you could learn an arbitrary quantile function means you could also arbitrarily learn a sampler for any distribution, since all you have to do compose a uniform sampler with whatever quantile function you learned. Of course for this example I'm using "solved" using a similar approach with variational inference (pyro has a write up on it: https://pyro.ai/examples/svi_part_i.html https://pyro.ai/examples/svi_part_i.html, you might find David Blei's "Variational Inference: A Review for Statisticians" useful as well https://arxiv.org/abs/1601.00670 https://arxiv.org/abs/1601.00670)

steppi 4y ago
What a coincidence. Pyro has basically become my go to for Bayesian learning and I know those docs in and out. I’ve also skimmed that David Blei article before. Small world. I’m not very confident of success but it seems like an interesting path to explore.