6 ms·
In the days when Sussman was a novice, Minsky once came to him as he sat hacking at the PDP-6. "What are you doing?", asked Minsky. "I am training a randomly
by Tossrock 11y ago
In the days when Sussman was a novice, Minsky once came to him as he sat hacking at the PDP-6.
"What are you doing?", asked Minsky.
"I am training a randomly wired neural net to play Tic-tac-toe", Sussman replied.
"Why is the net wired randomly?", asked Minsky.
"I do not want it to have any preconceptions of how to play", Sussman said.
Minsky then shut his eyes.
"Why do you close your eyes?" Sussman asked his teacher.
"So that the room will be empty."
At that moment, Sussman was enlightened.
RIP.
- bijection 11y agoFrom the jargon file [1] [1] http://www.catb.org/jargon/html/koans.html#id3141241 http://www.catb.org/jargon/html/koans.html#id3141241
- tarr11 11y agoFound this original version copied from this story (original source is a dead link) [1] "So Sussman began working on a program. Not long after, this odd-looking bald guy came over. Sussman figured the guy was going to boot him out, but instead the man sat down, asking, “Hey, what are you doing?” Sussman talked over his program with the man, Marvin Minsky. At one point in the discussion, Sussman told Minsky that he was using a certain randomizing technique in his program because he didn’t want the machine to have any preconceived notions. Minsky said, “Well, it has them, it’s just that you don’t know what they are.” It was the most profound thing Gerry Sussman had ever heard. And Minsky continued, telling him that the world is built a certain way, and the most important thing we can do with the world is avoid randomness, and figure out ways by which things can be planned. Wisdom like this has its effect on seventeen-year-old freshmen, and from then on Sussman was hooked.]" [1] http://spetharrific.tumblr.com/post/26600309788/sussman-attains-enlightenment http://spetharrific.tumblr.com/post/26600309788/sussman-atta...
- dandrews 11y agoThat sounds like the version told in Levy's Hackers. Maybe someone in HN-land can verify that; I can't find my copy.
- dTal 11y agoI had no idea that koan was a (mostly) true story. Thanks a lot for posting.
- kkylin 11y agoYes, I had once asked GJS about that exactly that story, and he confirmed it (though maybe not the exact words).
- EdwardCoffin 11y agoDanny Hillis: "The first time I met Marvin Minsky: I walked into his office, I was very intimidated. He was sitting there, he was throwing wadded up pieces of paper at a wastebasket across the room, and doing a terrible job of it. He's missing it, they're all falling short. So I watch this for a while. Then he looked up at me and said 'ah! I forgot! It's one half em gee squared!'" - From his talk On Game Software Development, in 2001 I think (from Technetcast.com)
- josteink 11y agoMaybe I'm wrong, but that sounds like he was trying to get the formula for kinetic energy, which would be half em vee squared. Just to be clear: I'm not questioning the story, just the details in the recollection :)
- EdwardCoffin 11y agoI did wonder at the discrepancy at the time, so I re-listened to the audio several times carefully when I was writing the transcript above. To me, he is clearly saying gee rather than vee (could be an artifact of the recording though). I'm sure Hillis knows more physics than I do (though I knew the equations of motion pretty well at one time), but he could easily have just mis-spoke. I didn't pursue this line of thought, but considered it might have been something to do with him deriving an expression for the vertical position in a gravitational field, perhaps in terms of horizontal motion or something.
- evanb 11y agoWill robots inherit the earth? Yes, but they will be our children. --Marvin Minsky http://web.media.mit.edu/~minsky/papers/sciam.inherit.html http://web.media.mit.edu/~minsky/papers/sciam.inherit.html
- gohrt 11y agoIt is now several decades later. Do TensorFlow/CNN builders use random initial configurations, or custom designed stuctures?
- argonaut 11y agoUsually random drawn from a certain distribution (ex: Gaussian with std. deviation 0.001, or a std. deviation dependent on the number of input/output units (Xavier initialization)). For some tasks, you may wish to initialize using a network that was already trained on a different dataset, if you have reason to believe the new training task is similar to the previous task.
- deleted 11y ago[deleted]
- Houshalter 11y agoNN weights need to start random because otherwise two weights with exactly the same value can get "stuck" and be unable to differentiate. Backpropagation relies on starting random patterns that kind of match so that it can fine tune them. But the weights are often initialized to be really close to zero.
- brianpgordon 11y agoGiven the era though, Sussman may have actually been working with a neural net that's not the typical hidden-layer variety. "Randomly wired" could be a statement about the topography of the network, not about the weights.
- argonaut 11y agoThere is no evidence he was actually working with a neural net. https://web.archive.org/web/20120717041345/http://sch57.msk.ru/~khim/hackers/part1/chapter6.html https://web.archive.org/web/20120717041345/http://sch57.msk....
- 11y ago
- georgespencer 11y agoI always loved this TED talk of his https://www.youtube.com/watch?v=RYsTv-ap3XQ https://www.youtube.com/watch?v=RYsTv-ap3XQ
- jes5199 11y agoAmazingly, according to this 1981 interview in the New Yorker, Minsky's first neural net was itself randomly wired! http://www.newyorker.com/magazine/1981/12/14/a-i http://www.newyorker.com/magazine/1981/12/14/a-i "Because of the random wiring, it had a sort of fail-safe characteristic. If one of the neurons wasn’t working, it wouldn’t make much of a difference—and, with nearly three hundred tubes and the thousands of connections we had soldered, there would usually be something wrong somewhere. In those days, even a radio set with twenty tubes tended to fail a lot. I don’t think we ever debugged our machine completely, but that didn’t matter. By having this crazy random design, it was almost sure to work, no matter how you built it."
- areyousure 11y agoHonest question: What does this koan mean? Specifically, what is closing one's eyes analogous to in the neural network? What is the recommended alternative action?
- Scarblac 11y agoSomewhere else is the thread is a closer rendition of how the actual exchange went. Using that, I finally understand the koan: Closing his eyes did not make the room empty. It made him not know which things were where. Randomizing the neural network did not remove all the preconceptions from the network. It made him not know what the network's preconceptions were.