3 ms·
Interesting. Nothing prohibits us from thinking of pretrained LLMs as dynamical systems that take a token state and compute an updated token state: x_{n+1} = LL
by cs702 1y ago
Interesting. Nothing prohibits us from thinking of pretrained LLMs as dynamical systems that take a token state and compute an updated token state: x_{n+1} = LLM(x_n), starting from an initial token state x_0. Surely we can compute trajectories (without sampling, for determinism) and study whether LLMs exhibit chaotic behavior. I don't think I've seen research along those lines before. Has anyone here?
- elcritch 1y agoLooks like @cs702 [1] posted a related link where a NN follows an attractor pattern! I've only skimmed it but it very much looks like what I've been imagining. It'd be cool to see more research into this area. 1: https://news.ycombinator.com/item?id=45427778 https://news.ycombinator.com/item?id=45427778 2: https://towardsdatascience.com/attractors-in-neural-network-circuits-beauty-and-chaos/ https://towardsdatascience.com/attractors-in-neural-network-...