4 ms·
An old story: “What are you doing?”, asked Minsky. “I am training a randomly wired neural net to play Tic-Tac-Toe” Sussman replied. “Why is the net wired ran
by aidenn0 4y ago
An old story:
“What are you doing?”, asked Minsky.
“I am training a randomly wired neural net to play Tic-Tac-Toe” Sussman replied.
“Why is the net wired randomly?”, asked Minsky.
“I do not want it to have any preconceptions of how to play”, Sussman said.
Minsky then shut his eyes.
“Why do you close your eyes?”, Sussman asked his teacher.
“So that the room will be empty.”
At that moment, Sussman was enlightened.
- dekhn 4y agoEvery time I read this I appreciate its sublimity.
- samvher 4y agoMaybe I'm naive - but I don't get it. Closing your eyes doesn't make the room empty. And in the same way not programming preconceptions into the neural net doesn't make the preconceptions go away? I realize explaining a joke or something like this takes away some of the charm (sorry), but would love to get the point :)
- just_testing 4y agoI interpret that as Minsky being... an unpleasurable person, to say the least.
- usefulcat 4y agoI think you’ve exactly described the point.
- PuppyTailWags 4y agoRandomly wiring a neural net doesn't remove preconceptions, it just makes it that you don't know them. Similarly, closing your eyes doesn't make a room empty, it just makes it that you can't see what's there. Minsky is pointing out that Sussman's fundamental assumptions on desiring to remove preconceptions is logically flawed.
- samvher 4y agoGotcha - I read it as Sussman didn't want to program his preconceptions into the network (for which randomness seemed suitable, which is why I was confused). Your explanation makes more sense.
- michaelt 4y agoThere's an old series of - stories? jokes? called 'unix koans' [1] which always end with a master answering a question in a very unclear but profound-sounding way, then the line 'Upon hearing this, [someone] was enlightened.' I never found them laugh-out-loud funny myself. This is probably a reference to those. [1] http://www.catb.org/~esr/writings/unix-koans/ http://www.catb.org/~esr/writings/unix-koans/
- aidenn0 4y agoA Koan (公案) is a concept from Zen Buddhism. Zen made its way into pop-culture (or at least the popular counter-culture movement) in the US in the 1960s via several writers of the Beat Generation. Project MAC at MIT was founded in the early 60s (Gerald Sussman started there in '64 I think)? So a number of faux Koans were in circulation in the AI crowd by about 1970.
- whatshisface 4y agoUnfortunately because it's delivered as a koan we'll never know whether he's talking about the fact that the random weights determine the nearest local minimum or the hyperparameters.
- mistrial9 4y agoalso unfortunately, a "real koan" often relies on conflating expectations+conditioning versus direct experience, where direct experience is shown to invoke an "impossible" result; teaching that the conditioned, subjective mind does not see fundamental aspects of reality, though it thinks it has the answers. These technical mimics of that structure echo a time when a lot of people, relatively, were experimenting with disregarding personal subjectivity in favor of direct experience and deeper Truth. In the modern tech versions, that is rarely if ever part of the story?
- yarky 4y agoThe problem with preconceptions about your parameters is that you might be missing some crazy cool path to your goal, which you might find by randomly exploring your sample space. I remember seeing this same principle in mcmc samplers using uniform priors. Why is this so crazy?
- 95014_refugee 4y agoIt's predicated on the assumption that a random discovery from a zero-comprehension state is more likely to get you to a goal than an evolution from a state that has at least some correctness. More generally, it disingenuously disregards the fact that the definition of the problem brings with it an enormous set of preconceptions. Reductio ad absurdum, you should just start training a model on completely random data in search of some unexpected but useful outcome. Obviously we don't do this; by setting a goal and a context we have already applied constraints, and so this really just devolves into a quantitative argument about the set of initial conditions. (This is the entire point of the Minsky / Sussman koan.)
- yarky 4y ago> from a zero-comprehension state is more likely to get you to a goal than an evolution from a state that has at least some correctness. I get that starting from a point with "some correctness" makes sense if you want to use such information (e.g. a standard starting point). However, such information is a preconceived solution to the problem, which might not be that useful after all. The fact is that you indeed might not at all need such information to find an optimal solution to a given problem. > by setting a goal and a context we have already applied constraints. I might be missing your point here since the goal and constraints must come from the real world problem to solve which is independent from the method to solve the problem. Unless you're describing p-value hacking your wait out, which is a broader problem.
- aidenn0 4y agoWith exploring, the starting state should only affect which local-maximum you end up in. Therefore you need to make an argument that a random starting state is likely to end up in a higher local-maximum than a non-random starting state. There is always a starting state; using a random one only means you don't know what it is.