3 ms·

> There is often no function to find in solving a problem, ie., there is no mapping from ImageSpace -> DogCatSpace. Ie., most things are genuine ambiguitites --

> There is often no function to find in solving a problem, ie., there is no mapping from ImageSpace -> DogCatSpace. Ie., most things are genuine ambiguitites --- a stick in a water appears bent, indistinguishably from an actually bent stick in some other transparent fluid. That's tangential imo. There is a function that maps from image space to dog/cat/don't know space. A sentient being that get the "don't know" can get more info to resolve the ambiguity (or rephrase the question). A universal function approximator can still make itself useful even if all it can do it say it doesn't know. This is a question of problem setup. NNs are bad at extrapolating, including e.g. trivially to periodic functions. This is a limitation if you thought they could do that, but again a question of understanding what they do. Hype, as you say, leads some people to believe NNs are magic, leading to mismatched expectation. A universal interpolate-only function approximator is still pretty useful though, just maybe disappointing if you understood that to imply sentience

mjburgess 5y ago
Sure, we could see `animal = f(image)` as a partial function and lift it into some total function space, `maybe_animal = f(image)`. Is our reasoning here actually this total function though? We really do always need to keep in mind that animal reasoning will terminate, animals will act "guided by other reasoning", and resume the original reasoning process. Action really messes up this neat "totalising option" for partial functions. If i'm not sure if "fluffy" is a dog or a cat, i might wait longer for it to move; I might throw something at it; I might ask its owner. This isnt as simple as "don't know", since reasoning is kinda time-bound and time-parameterised, my very measuring process is sensitive to my own confidence in what-something-is. Is this really "f(image) = dont-know"? I dont think so. I think it's more like, "judgement = body-brain-state(t, reasoning-goal, {action policies}, ...large-number-of-other-things)". This gets to my issue. In my view what animals are doing is better described by a dynamical equation of state (like a wavefunction), as the whole system is operating under its own dynamical evolution (including, eg., what fluffy does in response to your puzzlement). I dont see Dog|Cat|DontKnow as the answer here. I dont think it's the actual total function which corresponds to our judgement, though it probably is total -- we just end up with "DontKnow" in all the cases actual intellkigence is requireds.... the very think we're aiming to model.
bannedbybros 5y ago
sdenton4 5y ago
"NNs are bad at extrapolating, including e.g. trivially to periodic functions." WaveNet and its descendents are the obvious counter example here. It is excellent at learning to generate periodic and nearly periodic functions...
- version_five 5y ago
  I know I've seen papers where they use sinusoidal nonlinearities to learn periodic functions. I felt like that's a bit of a hack though (not necessarily a bad thing)- you're bringing domain knowledge in, which if you allow makes it easy to extend to periodic functions. The failure of a vanilla nn to learn periodicity is I think a specific failure to extrapolate, which is the bigger problem. I'll take a look at the architecture you mention. Edit: looked it up, I see it's an auto regressive model, like pixel CNN in 1D. I do know pixelCNN, I hadn't really considered the connection to periodic functions but I see what you're saying. In a sense any AR model is extrapolating, but not in the sense I mean: it has seen training examples of the next point predicted from the last points, it's not extending a relationship it learned in training to something new. Anyway, thanks for pointing the model out
  - sdenton4 5y ago
    There's ample literature showing that neutral vocoders generalize to new speakers not seen in training. IME, once you train on around ten speakers the models generalize quite well to new speakers, and in practice training sets for production models are often using hundreds of speakers for training. There are also non autoregressive GAN vocoders which work great, with a tiny fraction of the compute of the original WaveNet. Soundstream is a good example of this. My own work on bird song separation uses non AR models for generating separation masks. It works great and generalizes to new habitats and species without difficulty. https://ai.googleblog.com/2022/01/separating-birdsong-in-wild-for.html?m=1 https://ai.googleblog.com/2022/01/separating-birdsong-in-wil... Calling architecture variants a hack is risking no true Scotsman arguments... The only true neutral networks are the ones that don't work, perhaps?
- rileyphone 5y ago
  I think the approach of [0] is closer to what’s going on in our brains, basically evolving symbolic equations of partial derivatives until we get something good enough. Really fascinating and succinct paper. 0. https://cdanfort.w3.uvm.edu/courses/237/schmidt-lipson-2009.pdf https://cdanfort.w3.uvm.edu/courses/237/schmidt-lipson-2009....

mjburgess 5y ago

Sure, we could see `animal = f(image)` as a partial function and lift it into some total function space, `maybe_animal = f(image)`. Is our reasoning here actually this total function though? We really do always need to keep in mind that animal reasoning will terminate, animals will act "guided by other reasoning", and resume the original reasoning process. Action really messes up this neat "totalising option" for partial functions. If i'm not sure if "fluffy" is a dog or a cat, i might wait longer for it to move; I might throw something at it; I might ask its owner. This isnt as simple as "don't know", since reasoning is kinda time-bound and time-parameterised, my very measuring process is sensitive to my own confidence in what-something-is. Is this really "f(image) = dont-know"? I dont think so. I think it's more like, "judgement = body-brain-state(t, reasoning-goal, {action policies}, ...large-number-of-other-things)". This gets to my issue. In my view what animals are doing is better described by a dynamical equation of state (like a wavefunction), as the whole system is operating under its own dynamical evolution (including, eg., what fluffy does in response to your puzzlement). I dont see Dog|Cat|DontKnow as the answer here. I dont think it's the actual total function which corresponds to our judgement, though it probably is total -- we just end up with "DontKnow" in all the cases actual intellkigence is requireds.... the very think we're aiming to model.

bannedbybros 5y ago

sdenton4 5y ago

"NNs are bad at extrapolating, including e.g. trivially to periodic functions." WaveNet and its descendents are the obvious counter example here. It is excellent at learning to generate periodic and nearly periodic functions...

version_five 5y ago
I know I've seen papers where they use sinusoidal nonlinearities to learn periodic functions. I felt like that's a bit of a hack though (not necessarily a bad thing)- you're bringing domain knowledge in, which if you allow makes it easy to extend to periodic functions. The failure of a vanilla nn to learn periodicity is I think a specific failure to extrapolate, which is the bigger problem. I'll take a look at the architecture you mention. Edit: looked it up, I see it's an auto regressive model, like pixel CNN in 1D. I do know pixelCNN, I hadn't really considered the connection to periodic functions but I see what you're saying. In a sense any AR model is extrapolating, but not in the sense I mean: it has seen training examples of the next point predicted from the last points, it's not extending a relationship it learned in training to something new. Anyway, thanks for pointing the model out
- sdenton4 5y ago
  There's ample literature showing that neutral vocoders generalize to new speakers not seen in training. IME, once you train on around ten speakers the models generalize quite well to new speakers, and in practice training sets for production models are often using hundreds of speakers for training. There are also non autoregressive GAN vocoders which work great, with a tiny fraction of the compute of the original WaveNet. Soundstream is a good example of this. My own work on bird song separation uses non AR models for generating separation masks. It works great and generalizes to new habitats and species without difficulty. https://ai.googleblog.com/2022/01/separating-birdsong-in-wild-for.html?m=1 https://ai.googleblog.com/2022/01/separating-birdsong-in-wil... Calling architecture variants a hack is risking no true Scotsman arguments... The only true neutral networks are the ones that don't work, perhaps?
rileyphone 5y ago
I think the approach of [0] is closer to what’s going on in our brains, basically evolving symbolic equations of partial derivatives until we get something good enough. Really fascinating and succinct paper. 0. https://cdanfort.w3.uvm.edu/courses/237/schmidt-lipson-2009.pdf https://cdanfort.w3.uvm.edu/courses/237/schmidt-lipson-2009....