5 ms·

"Switching from experimenting a lot and thinking a little to experimenting a little and thinking a lot was a key turnaround in productivity. When debugging with

by Radim 9y ago

"Switching from experimenting a lot and thinking a little to experimenting a little and thinking a lot was a key turnaround in productivity. When debugging with long iteration times, you really need to pour time into the hypothesis-forming step - thinking about what all the possibilities are, how likely they seem on their own, and how likely they seem in light of everything you’ve seen so far." Not to glorify by-gone days, but isn't there a certain charm to having to wait your turn to plug punch cards into a mainframe? Having a "single shot", with long iteration cycles, indeed does something strange to the way you approach programming and bugs and program design. Anecdote: my long iteration cycles were caused by my mom restricting access to my C64. I filled page after page (paper!) with code, anxious for any chance to try it. I like to think I learned something during these "mental dry runs"… On topic: I wrote a rant about the "Mummy Effect" of trying to reproduce ML papers (which we do regularly): https://rare-technologies.com/mummy-effect-bridging-gap-between-academia-industry/ https://rare-technologies.com/mummy-effect-bridging-gap-betw...

zwieback 9y ago
I'd say this is generally good advice in science & engineering. My EE and ME coworkers are forced to do this because the each cycle is so slow and expensive.
seanmcdirmid 9y ago
Computers have incredible potential in helping us think about and solve problems. Most programming environments have difficulty in showing up that potential however (they are think to program rather than program to think). Especially in machine learning, it is interesting to wonder what kind of tool on a computer would help us solve problems if it isn’t a programming environment or notebook/playground.
j0e1 9y ago
I've been facing similar issues recently and have been slowly drifting towards similar ideas mentioned in the article. Though, coming from a more traditional software dev background, I'm still in the process of internalizing how the popular start-up mantra-'build fast and fail fast' doesn't really work in ML. Also, logging every experiment is another discipline I'm teaching myself. And I fell in love with org-mode in the process.
- deong 9y ago
  When I was doing my PhD, I once resorted to turning on all logging, printing about 60 pages of the resulting output, taping them to the floor of the lab, and crawling around on all fours with a printout of the source code tracing the execution to try and prove to myself that a result I was seeing wasn't due to just a bug in the source code. In an odd way, it's maybe my fondest memory from my dissertation work.
  - deleted 9y ago
    [deleted]
ssivark 9y ago
As they say, a few days in the laboratory might save a few hours in the library! Alternately, a few days of doing might save a few hours of thinking. :-)
- hulahoof 9y ago
  The flavour I'd always heard was a day of coding saves an hour of planning
crististm 9y ago
Hamming of the hamming code fame was saying exactly this in one of his lectures. His computer was slow enough that one iteration in finding optimal trajectory for a missile took several minutes, enough to allow him to get insight into what the 'curious' results meant. (In that particular story, it meant that for a ballistic missile the optimal trajectory was to shoot it straight up and steer it on the way down instead of steering it on an elliptic path from the get go)
- sytelus 9y ago
  Where can we find more about this story?
  - crististm 9y ago
    Look up "you and your research" on youtube. He mentioned that story in a couple of lectures, one time in the one about simulation: https://youtu.be/O5Ml5kPouG8?list=PL2FF649D0C4407B30&t=688 https://youtu.be/O5Ml5kPouG8?list=PL2FF649D0C4407B30&t=688 I recommend the whole course though.
pjmorris 9y ago
In my first programming class (1981) - FORTRAN! - we keypunched cards, handed them to a clerk to run, and waited hours for the results. We learned to be careful and to think things through. But nothing like Knuth... "When I was at Stanford with the AI project [in the late 1960s] one of the things we used to do every Thanksgiving is have a computer programming contest with people on research projects in the Bay area. The prize I think was a turkey. [John] McCarthy used to make up the problems. The one year that Knuth entered this, he won both the fastest time getting the program running and he also won the fastest execution of the algorithm. He did it on the worst system with remote batch called the Wilbur system. And he basically beat the shit out of everyone. And they asked him, "How could you possibly do this?" And he answered, "When I learned to program, you were lucky if you got five minutes with the machine a day. If you wanted to get the program going, it just had to be written right. So people just learned to program like it was carving stone. You sort of have to sidle up to it. That's how I learned to program." [0] [0] http://www.softpanorama.org/People/Knuth/index.shtml http://www.softpanorama.org/People/Knuth/index.shtml
segmondy 9y ago
The REPL is often misused and is now a curse. The idea behind the REPL is for fast iteration in development, but the key thing missing is thinking. Most people write code at the REPL to see what will happen. They don't formulate a hypothesis before hitting the enter key. It's more of a guess driven development methodology, keep trying till it works. If you want to improve your program experience, slow down on the run/compile cycle. Think it through.

by Radim 9y ago

zwieback 9y ago

I'd say this is generally good advice in science & engineering. My EE and ME coworkers are forced to do this because the each cycle is so slow and expensive.

seanmcdirmid 9y ago

Computers have incredible potential in helping us think about and solve problems. Most programming environments have difficulty in showing up that potential however (they are think to program rather than program to think). Especially in machine learning, it is interesting to wonder what kind of tool on a computer would help us solve problems if it isn’t a programming environment or notebook/playground.

j0e1 9y ago

I've been facing similar issues recently and have been slowly drifting towards similar ideas mentioned in the article. Though, coming from a more traditional software dev background, I'm still in the process of internalizing how the popular start-up mantra-'build fast and fail fast' doesn't really work in ML. Also, logging every experiment is another discipline I'm teaching myself. And I fell in love with org-mode in the process.

deong 9y ago
When I was doing my PhD, I once resorted to turning on all logging, printing about 60 pages of the resulting output, taping them to the floor of the lab, and crawling around on all fours with a printout of the source code tracing the execution to try and prove to myself that a result I was seeing wasn't due to just a bug in the source code. In an odd way, it's maybe my fondest memory from my dissertation work.
- deleted 9y ago
  [deleted]

ssivark 9y ago

As they say, a few days in the laboratory might save a few hours in the library! Alternately, a few days of doing might save a few hours of thinking. :-)

hulahoof 9y ago
The flavour I'd always heard was a day of coding saves an hour of planning

crististm 9y ago

Hamming of the hamming code fame was saying exactly this in one of his lectures. His computer was slow enough that one iteration in finding optimal trajectory for a missile took several minutes, enough to allow him to get insight into what the 'curious' results meant. (In that particular story, it meant that for a ballistic missile the optimal trajectory was to shoot it straight up and steer it on the way down instead of steering it on an elliptic path from the get go)

sytelus 9y ago
Where can we find more about this story?
- crististm 9y ago
  Look up "you and your research" on youtube. He mentioned that story in a couple of lectures, one time in the one about simulation: https://youtu.be/O5Ml5kPouG8?list=PL2FF649D0C4407B30&t=688 https://youtu.be/O5Ml5kPouG8?list=PL2FF649D0C4407B30&t=688 I recommend the whole course though.

pjmorris 9y ago

In my first programming class (1981) - FORTRAN! - we keypunched cards, handed them to a clerk to run, and waited hours for the results. We learned to be careful and to think things through. But nothing like Knuth... "When I was at Stanford with the AI project [in the late 1960s] one of the things we used to do every Thanksgiving is have a computer programming contest with people on research projects in the Bay area. The prize I think was a turkey. [John] McCarthy used to make up the problems. The one year that Knuth entered this, he won both the fastest time getting the program running and he also won the fastest execution of the algorithm. He did it on the worst system with remote batch called the Wilbur system. And he basically beat the shit out of everyone. And they asked him, "How could you possibly do this?" And he answered, "When I learned to program, you were lucky if you got five minutes with the machine a day. If you wanted to get the program going, it just had to be written right. So people just learned to program like it was carving stone. You sort of have to sidle up to it. That's how I learned to program." [0] [0] http://www.softpanorama.org/People/Knuth/index.shtml http://www.softpanorama.org/People/Knuth/index.shtml

segmondy 9y ago

The REPL is often misused and is now a curse. The idea behind the REPL is for fast iteration in development, but the key thing missing is thinking. Most people write code at the REPL to see what will happen. They don't formulate a hypothesis before hitting the enter key. It's more of a guess driven development methodology, keep trying till it works. If you want to improve your program experience, slow down on the run/compile cycle. Think it through.