5 ms·

Novel paper. But it seems like a lot of the excitement is because of the fusion of two buzz-words, one from the 80s and 90s and another from the 2010s. So, the

by shas3 8y ago

Novel paper. But it seems like a lot of the excitement is because of the fusion of two buzz-words, one from the 80s and 90s and another from the 2010s. So, the output is a bunch of stuff coming out of a kinda simple dynamical system. Chaotic for sure, but still simple. Deep learning (and more generally, recurrent neural nets, LSTMs, and derivatives thereof) has been shown capable of learning much more complex nonlinear systems, including human perception. Given this, I think the OP paper is a low-hanging fruit. It was only a matter of time before someone figured out a way to learn specific chaotic nonlinear dynamical systems. Nice work nonetheless.

IAmEveryone 8y ago
I'm not sure if human perception is a chaotic system. Chaos is defined as "small change in input -> large change in output". Perception is actually the opposite, with small input changes (changes in light, different angles,...) leading to fundamentally unchanged perception ("It's a tree").
- jessriedel 8y ago
  First, sensitivity to initial conditions is a necessary but not sufficient property for chaos. Some sort of folding/mixing is also necessary, which can be gauranteed by a bounded state space. Second, it's definitely true that information processing systems, if they are to be reliable enough to be useful, are not going to be chaotic throughout their state space. They need to return the same output given a certain input. But I'd imagine there are also at least a few noisy regions.
- perl4ever 8y ago
  I always thought that a perception like "it's a tree" that remains stable could possibly be an attractor in a chaotic system. If you look at the trajectory of a particle around an attractor, its position is very unpredictable after a while, but which attractor it is orbiting is not so random or unstable. Thinking of the visual of the Lorenz attractor.
- selimthegrim 8y ago
  You need also ergodicity and mixing, as jessriedel states, meaning respectively that you explore every area of your phase space with equal probability, and that two trajectories that begin arbtrarily close together do in fact diverge instead of sticking together, at infinite time
diehunde 8y ago
That's like saying, "it was only a matter of time someone figured out time traveling", when it happens. The whole point of the investigation was about chaotic systems, regardless of whether they're complex or not.
- heavenlyblue 8y ago
  Nope, given our current theories time travel is practically impossible. On the other hand we know that chaotic systems are functions and that matrices can approximate any function.
  - chestervonwinch 8y ago
    > matrices can approximate any function I'm not sure what point you're trying to make. Matrices only perform linear transformations, so matrices only approximate functions linearly, which in general, is a terrible approximation globally.
    - ekun 8y ago
      Especially, if you have complex systems where discretization and linearization aren't computationally achievable and/or numerically accurate ... like predicting global weather patterns or even very small experiments. I think about the phrase: All models are wrong; some models are useful.
  - dekhn 8y ago
    ITYMeant "A multilayer perceptron is a universal function approximator.
  - diehunde 8y ago
    I was pretty sure someone was going to focus on the example I gave instead of the idea :), very predictable. Chaotic systems are not just functions, that's why there's an entire discipline that studies them. There are many ways to approximate functions that have been developed during a long time and all have had trouble approximating chaotic functions.
TTPrograms 8y ago
In the example they give they predict the future of a simulation. If you have a perfect simulation, with perfect observation, why not just run the simulation forwards? Well, the goal is to apply this to the real world, where you possibly have only an approximate model and observations - which are noisy and imprecise. So, instead, try predicting the future based on noisy observations. Due to exponential divergence, it seems unlikely this would work. Looking through the paper, it looks like they do not analyze the performance under noisy observation - they just analyze their ability to estimate the Lyapunov exponents under noise, which is much easier. So the real world application (in terms of forward forecasting) seems like it's limited to cases where chaotic divergence between simulation and real world is due to simulation model errors rather than observation error - the latter is still a fundamental limit. Otherwise, this demonstrates that DNNs can be trained to solve diff eqs well, which is fairly well-trodden work. As a result, claiming that this can enhance ex. weather prediction is highly misleading at this point, as nothing has demonstrated any prediction performance improvements under noisy observation, which is really the fundamental (and practical) limit in chaotic systems. While it still may be possible to use DNNs to do that (by ex. learning how to most accurately estimate the components of state with the largest Lyapunov constants) I don't think this work demonstrates the feasibility of that yet. The language in the paper is more reserved, in that they primarily claim that they can use the DNN to learn a dynamic model even with observation noise and then that model can be used to estimate Lyapunov constants accurately. That's much more reasonable.

by shas3 8y ago

IAmEveryone 8y ago

I'm not sure if human perception is a chaotic system. Chaos is defined as "small change in input -> large change in output". Perception is actually the opposite, with small input changes (changes in light, different angles,...) leading to fundamentally unchanged perception ("It's a tree").

jessriedel 8y ago
First, sensitivity to initial conditions is a necessary but not sufficient property for chaos. Some sort of folding/mixing is also necessary, which can be gauranteed by a bounded state space. Second, it's definitely true that information processing systems, if they are to be reliable enough to be useful, are not going to be chaotic throughout their state space. They need to return the same output given a certain input. But I'd imagine there are also at least a few noisy regions.
perl4ever 8y ago
I always thought that a perception like "it's a tree" that remains stable could possibly be an attractor in a chaotic system. If you look at the trajectory of a particle around an attractor, its position is very unpredictable after a while, but which attractor it is orbiting is not so random or unstable. Thinking of the visual of the Lorenz attractor.
selimthegrim 8y ago
You need also ergodicity and mixing, as jessriedel states, meaning respectively that you explore every area of your phase space with equal probability, and that two trajectories that begin arbtrarily close together do in fact diverge instead of sticking together, at infinite time

diehunde 8y ago

That's like saying, "it was only a matter of time someone figured out time traveling", when it happens. The whole point of the investigation was about chaotic systems, regardless of whether they're complex or not.

heavenlyblue 8y ago
Nope, given our current theories time travel is practically impossible. On the other hand we know that chaotic systems are functions and that matrices can approximate any function.
- chestervonwinch 8y ago
  > matrices can approximate any function I'm not sure what point you're trying to make. Matrices only perform linear transformations, so matrices only approximate functions linearly, which in general, is a terrible approximation globally.
  - ekun 8y ago
    Especially, if you have complex systems where discretization and linearization aren't computationally achievable and/or numerically accurate ... like predicting global weather patterns or even very small experiments. I think about the phrase: All models are wrong; some models are useful.
- dekhn 8y ago
  ITYMeant "A multilayer perceptron is a universal function approximator.
- diehunde 8y ago
  I was pretty sure someone was going to focus on the example I gave instead of the idea :), very predictable. Chaotic systems are not just functions, that's why there's an entire discipline that studies them. There are many ways to approximate functions that have been developed during a long time and all have had trouble approximating chaotic functions.

TTPrograms 8y ago

In the example they give they predict the future of a simulation. If you have a perfect simulation, with perfect observation, why not just run the simulation forwards? Well, the goal is to apply this to the real world, where you possibly have only an approximate model and observations - which are noisy and imprecise. So, instead, try predicting the future based on noisy observations. Due to exponential divergence, it seems unlikely this would work. Looking through the paper, it looks like they do not analyze the performance under noisy observation - they just analyze their ability to estimate the Lyapunov exponents under noise, which is much easier. So the real world application (in terms of forward forecasting) seems like it's limited to cases where chaotic divergence between simulation and real world is due to simulation model errors rather than observation error - the latter is still a fundamental limit. Otherwise, this demonstrates that DNNs can be trained to solve diff eqs well, which is fairly well-trodden work. As a result, claiming that this can enhance ex. weather prediction is highly misleading at this point, as nothing has demonstrated any prediction performance improvements under noisy observation, which is really the fundamental (and practical) limit in chaotic systems. While it still may be possible to use DNNs to do that (by ex. learning how to most accurately estimate the components of state with the largest Lyapunov constants) I don't think this work demonstrates the feasibility of that yet. The language in the paper is more reserved, in that they primarily claim that they can use the DNN to learn a dynamic model even with observation noise and then that model can be used to estimate Lyapunov constants accurately. That's much more reasonable.