3 ms·

I think the article is very honest about this just being a fun exploration. They even show how you can get similar patterns with newton's algorithm which is a

• alexmolas 3y ago
  Yes, the author is clear in this regard. But I've seen people interpreting this paper as "training deep networks is chaotic", and I don't think that's the case. I interpret it more as "if you're not careful with your learning rate your training will be chaotic"
  • repelsteeltje 3y ago
    Yes. I immediately had visions of a misconfigured PID controller or similar chaotic emergence encountered in control theory. I was surprised and delighted though, that this type of chaos has so much fractal beauty.
    • nyrikki 3y ago
      Note that not all fractals are chaotic. Inter dimensionality is the defining feature of fractals. A multilayer ANN will have compression that may be fractal like. But in theory a feed forward network could be single layer but will a lot more neurons. I have the feeling this result was due to representation but there are some features that look like riddled basins. Riddled basins do arise in SNN or spikey neutral networks that have continuous output bs the binary output of ANNs. But as all PAC learning is just compression that may be the cause also. I'll be digging into this on the weekend though.
  • breather 3y ago
    With all do respect, why are you commenting here rather than replying to the person giving your comment context?