4 ms·

Could it be that this behavior is just caused by numerical issues and/or randomness in the calculation, rather than a real property of neural networks?

sigmoid10 3y ago
Fractals are not caused by randomness. Fractals arise from scale invariance and self similarity, which in turn can come from non linear systems and iterative processes. It is very easy to generate fractals in practice and even the most vanilla neural networks trivially fulfill the requirements (at least when you look at training output). In that sense it would be weird not to find fractal structures when you look hard enough.
- tomxor 3y ago
  "If we're built from spirals, while living in a giant spiral, then everything we put our hands to, is infused with the spiral" ... Sorry I couldn't help myself.
- nyrikki 3y ago
  Not exactly, Newton's fractal, which is topological, specifically the wada property, is a boundary condition. It relates to fractal (non-integer) dimensions, which was first described by Mandelbrot in a paper about self similarity. Here is a paper that covers some of that. https://www.minvydasragulskis.com/sites/default/files/publications/ND_2017_v2.pdf https://www.minvydasragulskis.com/sites/default/files/public... In Newton's fractal, no matter how small a circle you can draw, your circle will either contain one root or all the roots. The basins that contain one root are open sets that share a boundary set. Even if you could have perfect information and precision this property holds. This means any change in initial conditions that crosses a boundary will be indeterminate. There is another feature called riddled basins, where every point is arbitrarily close to other basins. This is another situation where even with perfect information and unlimited precision a perturbations would be indeterminate. A positive Laponov exponent which isn't sufficient to prove chaos, but is always positive in the presence of chaos may even be 0 or negative in the above situations. Take the typical predator prey model and add fear and refuge and you hit the riddled basins. Stack four reflective balls in a pyramid and shine different color lights in two sides and you will see the Wada property. Neither of those problems are addressable with the assumption of deterministic effects with finite precision.
  - sigmoid10 3y ago
    Chaos is just another possible consequence of non linear systems. And newtons's fractal is also generated by just another iterative process. That doesn't mean the origin is necessarily random.

sigmoid10 3y ago

Fractals are not caused by randomness. Fractals arise from scale invariance and self similarity, which in turn can come from non linear systems and iterative processes. It is very easy to generate fractals in practice and even the most vanilla neural networks trivially fulfill the requirements (at least when you look at training output). In that sense it would be weird not to find fractal structures when you look hard enough.

tomxor 3y ago
"If we're built from spirals, while living in a giant spiral, then everything we put our hands to, is infused with the spiral" ... Sorry I couldn't help myself.
nyrikki 3y ago
Not exactly, Newton's fractal, which is topological, specifically the wada property, is a boundary condition. It relates to fractal (non-integer) dimensions, which was first described by Mandelbrot in a paper about self similarity. Here is a paper that covers some of that. https://www.minvydasragulskis.com/sites/default/files/publications/ND_2017_v2.pdf https://www.minvydasragulskis.com/sites/default/files/public... In Newton's fractal, no matter how small a circle you can draw, your circle will either contain one root or all the roots. The basins that contain one root are open sets that share a boundary set. Even if you could have perfect information and precision this property holds. This means any change in initial conditions that crosses a boundary will be indeterminate. There is another feature called riddled basins, where every point is arbitrarily close to other basins. This is another situation where even with perfect information and unlimited precision a perturbations would be indeterminate. A positive Laponov exponent which isn't sufficient to prove chaos, but is always positive in the presence of chaos may even be 0 or negative in the above situations. Take the typical predator prey model and add fear and refuge and you hit the riddled basins. Stack four reflective balls in a pyramid and shine different color lights in two sides and you will see the Wada property. Neither of those problems are addressable with the assumption of deterministic effects with finite precision.
- sigmoid10 3y ago
  Chaos is just another possible consequence of non linear systems. And newtons's fractal is also generated by just another iterative process. That doesn't mean the origin is necessarily random.