6 ms·
Why neural networks struggle with the Game of Life (2020)
- leric 2y agoA good example of irreducible complexity
- yumong 2y ago[flagged]
- rustcleaner 2y agobased Simple systems built on simple rules creating universally complete computation behaviors are both unintuitive to and underrated by common man.
- nottorp 2y ago> Interestingly, no matter how complex a grid becomes, you can predict the state of each cell in the next timestep with the same rules. How is that interesting? It's the definition of the Game of Life... it's not like it's a natural system that you don't know the full rules for...
- brookst 2y agoIt’s interesting the way Go is interesting: very simple rules can produce extreme complexity. At least, I think that’s interesting.
- yumong 2y agoStephen Wolfram pushes this to the extreme. Surprised that nobody mentioned him yet.
- nottorp 2y agoI wouldn't compare the game of life that has exactly one "next state" with something like Go.
- brookst 2y agoIt’s interesting that you wouldn’t, yet I would.They aren’t isomorphic, for sure. Go’s complexity comes from two players alternately picking one out of a very large number of options. GoL’s complexity comes from a very large number of nodes “picking” between two states. That’s not precise, just illustrating that there is some symmetry of simplicity/complexity, at least to my eyes.
- SkyBelow 2y agoWhy not? We could even think of both as collections of 3d structures showing all valid structures possible for a board of size n by n. There are some differences, every single 3d Conway structure has a unique top layer, while Go does not. But that seems like an overall minor difference. There are many more Go shapes than Conway shapes given the same N, but both are already so numerous that I'm not sure that is a difference worth stopping the comparison.
- bell-cot 2y agoFrom a quick skim, then string search for "interesting" - I'd say that word is fluff, added to keep their audience reading through their dull background intro.
- deleted 2y ago[deleted]
- kozlovsky 2y agoIf we show a neural network some examples from the Game of Life and expect it to master the rules of a cellular automaton, then aren't we asking too much from it? In some ways, this is analogous to expecting that if we show the neural network examples from the physical world, it will automatically derive Newton's three laws. Not every person observing the world around him can independently deduce Newton's laws from scratch, no matter how many examples he sees.
- passwordoops 2y ago"then aren't we asking too much from it" Not according to the hype merchants, hucksters, and VCs who think word models are displaying emergence and we're 6 months from AGI, if only we can have more data
- nottorp 2y agoLet's be snarky a bit: Can you do a neural network that, given a starting position of the game of life, decides if it cycles or not? ;) Ok, not cycles... dies, stabilizes, goes into a loop etc.
- GrantMoyer 2y agoSo Oracle's working on an LLM too, eh?
- DeathArrow 2y agoEverybody and their mom are into LLMs.
- red75prime 2y agoI struggled with the Game of Life too. I was fascinated by it and evolved cell populations on graph paper by hand (yeah, I'm that old). When I've got a computer, I checked my drawings and all of them were wrong.
- zaphar 2y agoI used to do this as a kid on long car trips. I would play GoL on paper. It's a good way to eat up time enjoyably.
- maxbond 2y agoReminds me of Knuth's dragon mosaic, which also contains a mistake. https://www.youtube.com/watch?v=v678Em6qyzk https://www.youtube.com/watch?v=v678Em6qyzk
- mondrian 2y agoReally fun & informative interview, thanks for linking!
- K0balt 2y agoWhat’s not clear to me is if it is non trivial to (a)create a GOL NN that models the game cells directly as neurons (which would seem to be an efficient and effective method) or if it’s just (b) nontrivial to create a transformer architecture model that can model the game state n-turns in the future. I would be very surprised if (a) was not effective, but that (b) is difficult is not surprising, since that is a very nontrivial task that requires intermediary modelling tools to perform for humans (arguably the most advanced NN that we have access to at the moment) (a) is actually a form of (b) in the form of a modelling tool.
- moconnor 2y agoBoth are ~trivial as detailed in the paper and article.
- moconnor 2y agoFor everyone reading neither the article nor the paper: - both show neural networks can learn the game of life just fine - the finding is that to learn the rules reliably the networks need to be very over-parameterised (e.g. many times larger than the minimal size needed for hand-crafted weights to perfectly solve the problem) This is not really a new result nor a surprising one, nor does it say anything about the kinds of functions a neural network can represent. It's an attempt to understand an existing observation: once we have trained a large overparameterized neural network we can often compress it to a smaller one with very little loss. So why can't we learn the smaller one directly? One of the theories referred to in the article and paper is the lottery hypothesis, which states that a large network is a superposition of many small networks and the larger you are the more likely at least one of those gets a "lucky" set of weights and converges quickly to the right solution. There is already interesting evidence for this.
- actionfromafar 2y ago> once we have trained a large overparameterized neural network we can often compress it to a smaller one with very little loss. So why can't we learn the smaller one directly? I feel something similar goes on in us humans. Interesting to think about.
- rmnclmnt 2y agoIndeed, you have to over-engineer something before converge to a leaner solution to a problem
- brookst 2y agoAnd I think this is because the ideal complex system is one where all the subsystems and parts combine to produce adequate reliability. Exponentiation means it is more efficient to start by far exceeding the required reliability and then optimizing the most expensive subsystems/parts. It is less efficient and far more frustrating if multiple things have to be improved to meet requirements.
- ngrilly 2y agoThis paper from 2020 has been peer reviewed: https://openreview.net/forum?id=uKZsVyFKbaj https://openreview.net/forum?id=uKZsVyFKbaj
- rasca 2y agoHere's a nice article about how another team solved Game of Life in LLMs using Claude 3 Opus: https://x.com/ctjlewis/status/1786948443472339247 https://x.com/ctjlewis/status/1786948443472339247 . It's a really nice read.
- deleted 2y ago[deleted]
- scotty79 2y ago> These findings are in line with “The Lottery Ticket Hypothesis,” If the fit was due to a lucky subset of weights you could have train smaller networks many times instead of using many times bigger network. So it must be something more. Like increased opportunity to create best solution out of large number of random lucky parts. I think there should be way more research on neural pruning. After all it's what our brains do to reach the correct architecture and weights during our development.
- hamilyon2 2y ago> In machine learning, one of the popular ways to improve the accuracy of a model that is underperforming is to increase its complexity. And this technique worked with the Game of Life. For those who didn't read the article, the content doesn't support the title.
- phaedrus 2y agoI wonder if anyone has tried to approach the problem from the other end: start with the hand-tuned network and randomize just some of the weights (or all of the weights a small amount), and see at what point the learning algorithm can no longer get back to the correct formulation of the problem. Map the boundary between almost-solved and failure to converge, instead of starting from a random point trying to get to almost-solved.
- phaedrus 2y agoPhilosophically, why should it be the case that aggregations of statistical calculations (one way of viewing the matrix multiplications of ANNs) can approximate intelligence? I think it's because our ability to know reality is inherently statistical. To be clear, I'm not suggesting macro scale, i.e. not quantum, reality itself is probabilistic, only that our ability to interpret perception of it and model it is statistical. That is, an observation or a sensor doesn't actually tell you the state of the world; it is a measurement from which you infer things. Viewed through this standpoint, maybe the Game of Life and other discrete, fully-knowable toy problem worlds aren't as applicable to the problem of general intelligence as we imagine. A way to put this into practice could be to introduce a level of error in both the hand-tuned and learned networks' ability to accurately measure the input states of the Life tableau (and/or introduce some randomness in the application of the Life rules on the simulation), and see whether the superiority of the hand-tuned network persists or if the learned network is more robust in the face of uncertain inputs or fallible rule-applications.
- ellis0n 2y agoAmazing idea! GoL/LLM
- puttycat 2y agoSimilar discussion with a potential solution, in the domain of formal language learning: Bridging the Empirical-Theoretical Gap in Neural Network Formal Language Learning Using Minimum Description Length https://arxiv.org/abs/2402.10013 https://arxiv.org/abs/2402.10013
- gordytammy04 2y ago[dead]