4 ms·

Random Attractors – Found using Lyapunov Exponents (2001)

137 points by cs702 1y ago

zparky 1y ago
Similar, post on the Henon Attractor 4h ago: https://news.ycombinator.com/item?id=45424223 https://news.ycombinator.com/item?id=45424223
- cs702 1y ago
  Also, from that page: https://towardsdatascience.com/attractors-in-neural-network-circuits-beauty-and-chaos/ https://towardsdatascience.com/attractors-in-neural-network-...
  - dtj1123 1y ago
    In the event that the first post led to this one, I'd be curious to know what the intermediate internet rabbithole consisted of.
AfterHIA 1y ago
These visualizations are beautiful. I'm a musician at heart so I really geek out about bifurcation maps. You get to see the exquisite relationship between chaos and form. It's like nature and math producing visual jazz. Thanks for a kick ass addition cs702!
elcritch 1y ago
This is how I envision LLMs working to some extent. As in that the "logic paths" follow something like this where the markov-chain-esque probabilities jump around the vector space. It reminds me that to get the answer I want that I need to setup the prompt to get me near the right "attractor logic" pathway. Once in a close enough ballpark then they'll bounce to the right path. As a counter, I found that if you add an incorrect statement or fact that lies completely outside the realm of the logic-attractor for a given topic that the output is severally degraded. Well more like a statement or fact that's "orthogonal" to the logic-attractor for a topic. Very much as if it's struggling to stay on the logic-attractor path but the outlier fact causes it to stray. Sometimes less is more.
- cs702 1y ago
  Interesting. Nothing prohibits us from thinking of pretrained LLMs as dynamical systems that take a token state and compute an updated token state: x_{n+1} = LLM(x_n), starting from an initial token state x_0. Surely we can compute trajectories (without sampling, for determinism) and study whether LLMs exhibit chaotic behavior. I don't think I've seen research along those lines before. Has anyone here?
  - elcritch 1y ago
    Looks like @cs702 [1] posted a related link where a NN follows an attractor pattern! I've only skimmed it but it very much looks like what I've been imagining. It'd be cool to see more research into this area. 1: https://news.ycombinator.com/item?id=45427778 https://news.ycombinator.com/item?id=45427778 2: https://towardsdatascience.com/attractors-in-neural-network-circuits-beauty-and-chaos/ https://towardsdatascience.com/attractors-in-neural-network-...
    - cs702 1y ago
      That's for a small and shallow neural network. I was wondering about LLMs specifically.
      - elcritch 1y ago
        Well me too, but it shows that there is some basis for the thinking. It's sort of surprising there's not more exploration into the area.
cantor_S_drug 1y ago
https://paulbourke.net/fractals/lyapunov/ https://paulbourke.net/fractals/lyapunov/ > It may diverge to infinity, for the range (+- 2) used here for each parameter this is the most likely event. These are also easy to detect and discard, indeed they need to be in order to avoid numerical errors. https://superliminal.com/fractals/bbrot/ https://superliminal.com/fractals/bbrot/ The above image shows the overall entire Buddhabrot object. To produce the image only requires some very simple modifications to the traditional mandelbrot rendering technique: Instead of selecting initial points on the real-complex plane one for each pixel, initial points are selected randomly from the image region or larger as needed. Then, each initial point is iterated using the standard mandelbrot function in order to first test whether it escapes from the region near the origin or not. Only those that do escape are then re-iterated in a second, pass. (The ones that don't escape - I.E. which are believed to be within the Mandelbrot Set - are ignored). During re-iteration, I increment a counter for each pixel that it lands on before eventually exiting. Every so often, the current array of "hit counts" is output as a grayscale image. Eventually, successive images barely differ from each other, ultimately converging on the one above. Is it possible to use the Buddhabrot technique on the lyapunov fractals ?
- fractal4d 1y ago
  Seems to me that the images on Bourke's site _are_ produced using the general "Buddhabrot" technique (splatting points onto an image). Although each image appears to only represent a single orbit sequence and the reject condition is inverted so that only stable orbits are shown. I've personally found the technique very versatile and have had a lot of fun playing around with it and exploring different variations. Was excited enough about the whole thing that I created a website for sharing some of my explorations: https://www.fractal4d.net/ https://www.fractal4d.net/ (shameless self-advertisement) With the exception of some Mandelbrot-style images all the rest are produced by splatting complex-valued orbit points onto an image in one way or another.
esafak 1y ago
Is anyone doing anything besides visualizations with this chaos stuff? I liked the article linked below depicting the state space of artificial neurons: https://towardsdatascience.com/attractors-in-neural-network-circuits-beauty-and-chaos/ https://towardsdatascience.com/attractors-in-neural-network-...
- MountDoom 1y ago
  Not really. Fractals and chaos theory were a bit like blockchain in that it was a "new kind of science" and it was supposed to explain everything, and you could buy pop-science books talking about the implications. And then it sort of fizzled out, because while it's interesting and gives us a bit of additional philosophical insights into certain problems, it doesn't do anything especially useful. You can use it to draw cool space-filling shapes.
  - sxzygz 1y ago
    I don’t think you’re remotely correct, but I also don’t know how to dispute your ignorance in any useful way. To @esafak I suggest following @westurner’s post. I like the concept of Stable Manifolds. Classifying types of them is interesting. Group symmetries on the phase space are interesting. Explaining this and more is not work I’m prepared to do here. Use Wikipedia, ask ChatGPT, enrol in a course on Chaos and Fractal Dynamics, etc.
    - MountDoom 1y ago
      I am quite familiar with this space and I will reassert that its by far most significant application is making pretty pictures. The Wikipedia list you're indirectly referencing is basically a fantasy wishlist of the areas where we expected the chaos theory to revolutionize things, with little to show for it. "Chaos theory cryptography", come on.
  - chermi 1y ago
    No. Yes to the overhype, no to the "not really". I don't see how better understanding non-linear systems and global dynamics can be not be considered useful. For starters, better control of nonlinear systems/keeping them from turning them chaotic is incredibly useful. So many hard problems can be approximately reduced to "keep this non-linear system stable." Staying in the "edge of chaos" regime has proven to be an optimal choice for a plethora of problems.
jheitmann 1y ago
There's a book covering this and more from 1993 called "Strange Attractors: Creating Patterns in Chaos" by Julian C. Sprott that's freely available here: https://sprott.physics.wisc.edu/SA.HTM https://sprott.physics.wisc.edu/SA.HTM It's fun (errr... for me at least) to translate the ancient basic code into a modern implementation and play around. The article mentions that it's interesting how the 2d functions can look 3d. That's definitely true. But, there's also no reason why you can't just add on however many dimensions you want and get real many-dimensioned structures with which you can noodle around with visualizations and animations.
- throwaway173738 1y ago
  As an undergraduate I worked with some other Physics students to construct an analog circuit using op amps that modeled one of Sprott’s equations and we confirmed experimentally that the system exhibited chaotic behavior. We also used a transconductance amplifier as a control parameter and swept through the different states (chaotic, period windows) of the circuit. We did not go as far as comparing the experimental and predicted period windows while I was there but it was an interesting project for us. At one point I turned up an article in Physica D describing how to calculate the first Lyapunov exponent using small data sets which we used to compute whether we were in a period window or not.
Sjeiti 1y ago
Back when I had all the time in the world I made this https://attractors.ronvalstar.nl/ https://attractors.ronvalstar.nl/ I never could quite get the hang of Lyapunov exponents though.

137 points by cs702 1y ago

Similar, post on the Henon Attractor 4h ago: https://news.ycombinator.com/item?id=45424223 https://news.ycombinator.com/item?id=45424223

cs702 1y ago
Also, from that page: https://towardsdatascience.com/attractors-in-neural-network-circuits-beauty-and-chaos/ https://towardsdatascience.com/attractors-in-neural-network-...
- dtj1123 1y ago
  In the event that the first post led to this one, I'd be curious to know what the intermediate internet rabbithole consisted of.

AfterHIA 1y ago

These visualizations are beautiful. I'm a musician at heart so I really geek out about bifurcation maps. You get to see the exquisite relationship between chaos and form. It's like nature and math producing visual jazz. Thanks for a kick ass addition cs702!

elcritch 1y ago

This is how I envision LLMs working to some extent. As in that the "logic paths" follow something like this where the markov-chain-esque probabilities jump around the vector space. It reminds me that to get the answer I want that I need to setup the prompt to get me near the right "attractor logic" pathway. Once in a close enough ballpark then they'll bounce to the right path. As a counter, I found that if you add an incorrect statement or fact that lies completely outside the realm of the logic-attractor for a given topic that the output is severally degraded. Well more like a statement or fact that's "orthogonal" to the logic-attractor for a topic. Very much as if it's struggling to stay on the logic-attractor path but the outlier fact causes it to stray. Sometimes less is more.

cs702 1y ago
Interesting. Nothing prohibits us from thinking of pretrained LLMs as dynamical systems that take a token state and compute an updated token state: x_{n+1} = LLM(x_n), starting from an initial token state x_0. Surely we can compute trajectories (without sampling, for determinism) and study whether LLMs exhibit chaotic behavior. I don't think I've seen research along those lines before. Has anyone here?
- elcritch 1y ago
  Looks like @cs702 [1] posted a related link where a NN follows an attractor pattern! I've only skimmed it but it very much looks like what I've been imagining. It'd be cool to see more research into this area. 1: https://news.ycombinator.com/item?id=45427778 https://news.ycombinator.com/item?id=45427778 2: https://towardsdatascience.com/attractors-in-neural-network-circuits-beauty-and-chaos/ https://towardsdatascience.com/attractors-in-neural-network-...
  - cs702 1y ago
    That's for a small and shallow neural network. I was wondering about LLMs specifically.
    - elcritch 1y ago
      Well me too, but it shows that there is some basis for the thinking. It's sort of surprising there's not more exploration into the area.

cantor_S_drug 1y ago

https://paulbourke.net/fractals/lyapunov/ https://paulbourke.net/fractals/lyapunov/ > It may diverge to infinity, for the range (+- 2) used here for each parameter this is the most likely event. These are also easy to detect and discard, indeed they need to be in order to avoid numerical errors. https://superliminal.com/fractals/bbrot/ https://superliminal.com/fractals/bbrot/ The above image shows the overall entire Buddhabrot object. To produce the image only requires some very simple modifications to the traditional mandelbrot rendering technique: Instead of selecting initial points on the real-complex plane one for each pixel, initial points are selected randomly from the image region or larger as needed. Then, each initial point is iterated using the standard mandelbrot function in order to first test whether it escapes from the region near the origin or not. Only those that do escape are then re-iterated in a second, pass. (The ones that don't escape - I.E. which are believed to be within the Mandelbrot Set - are ignored). During re-iteration, I increment a counter for each pixel that it lands on before eventually exiting. Every so often, the current array of "hit counts" is output as a grayscale image. Eventually, successive images barely differ from each other, ultimately converging on the one above. Is it possible to use the Buddhabrot technique on the lyapunov fractals ?

fractal4d 1y ago
Seems to me that the images on Bourke's site _are_ produced using the general "Buddhabrot" technique (splatting points onto an image). Although each image appears to only represent a single orbit sequence and the reject condition is inverted so that only stable orbits are shown. I've personally found the technique very versatile and have had a lot of fun playing around with it and exploring different variations. Was excited enough about the whole thing that I created a website for sharing some of my explorations: https://www.fractal4d.net/ https://www.fractal4d.net/ (shameless self-advertisement) With the exception of some Mandelbrot-style images all the rest are produced by splatting complex-valued orbit points onto an image in one way or another.

esafak 1y ago

Is anyone doing anything besides visualizations with this chaos stuff? I liked the article linked below depicting the state space of artificial neurons: https://towardsdatascience.com/attractors-in-neural-network-circuits-beauty-and-chaos/ https://towardsdatascience.com/attractors-in-neural-network-...

MountDoom 1y ago
Not really. Fractals and chaos theory were a bit like blockchain in that it was a "new kind of science" and it was supposed to explain everything, and you could buy pop-science books talking about the implications. And then it sort of fizzled out, because while it's interesting and gives us a bit of additional philosophical insights into certain problems, it doesn't do anything especially useful. You can use it to draw cool space-filling shapes.
- sxzygz 1y ago
  I don’t think you’re remotely correct, but I also don’t know how to dispute your ignorance in any useful way. To @esafak I suggest following @westurner’s post. I like the concept of Stable Manifolds. Classifying types of them is interesting. Group symmetries on the phase space are interesting. Explaining this and more is not work I’m prepared to do here. Use Wikipedia, ask ChatGPT, enrol in a course on Chaos and Fractal Dynamics, etc.
  - MountDoom 1y ago
    I am quite familiar with this space and I will reassert that its by far most significant application is making pretty pictures. The Wikipedia list you're indirectly referencing is basically a fantasy wishlist of the areas where we expected the chaos theory to revolutionize things, with little to show for it. "Chaos theory cryptography", come on.
- chermi 1y ago
  No. Yes to the overhype, no to the "not really". I don't see how better understanding non-linear systems and global dynamics can be not be considered useful. For starters, better control of nonlinear systems/keeping them from turning them chaotic is incredibly useful. So many hard problems can be approximately reduced to "keep this non-linear system stable." Staying in the "edge of chaos" regime has proven to be an optimal choice for a plethora of problems.

jheitmann 1y ago

There's a book covering this and more from 1993 called "Strange Attractors: Creating Patterns in Chaos" by Julian C. Sprott that's freely available here: https://sprott.physics.wisc.edu/SA.HTM https://sprott.physics.wisc.edu/SA.HTM It's fun (errr... for me at least) to translate the ancient basic code into a modern implementation and play around. The article mentions that it's interesting how the 2d functions can look 3d. That's definitely true. But, there's also no reason why you can't just add on however many dimensions you want and get real many-dimensioned structures with which you can noodle around with visualizations and animations.

throwaway173738 1y ago
As an undergraduate I worked with some other Physics students to construct an analog circuit using op amps that modeled one of Sprott’s equations and we confirmed experimentally that the system exhibited chaotic behavior. We also used a transconductance amplifier as a control parameter and swept through the different states (chaotic, period windows) of the circuit. We did not go as far as comparing the experimental and predicted period windows while I was there but it was an interesting project for us. At one point I turned up an article in Physica D describing how to calculate the first Lyapunov exponent using small data sets which we used to compute whether we were in a period window or not.

Sjeiti 1y ago

Back when I had all the time in the world I made this https://attractors.ronvalstar.nl/ https://attractors.ronvalstar.nl/ I never could quite get the hang of Lyapunov exponents though.