4 ms·

Similar experience here. A shocking number of people simply can't code. I started giving people a "simple" whiteboard programming question: write code, in an

by dkarl 16y ago

Similar experience here. A shocking number of people simply can't code. I started giving people a "simple" whiteboard programming question: write code, in any real language (i.e., not pseudocode,) to print out the permutations of a string, and give me running commentary on what they were doing. The idea was, by watching them complete a simple coding task that would require a couple of iterations of design and a little bit of "debugging" at the end, I could get a feel for how they attack problems. In about a dozen interviews, nobody did it. The problem was way too hard. By the end of the cycle of interviews I had started giving lots of guidance to the candidates: asking them about permutations that their solutions would miss, asking them how many permutations their solution would print, asking how many permutations a string of length 2, 3, or 4 might have, and when they got stuck, telling them what they had right and when they went wrong. Nothing really helped, though. Perhaps it's not really so shocking. Maybe's it's just too hard in a one-hour interview. I thought it would be easy, but then again, I've never had to write code in an interview. What is shocking is how poorly people floundered. I'm pretty sure most of them couldn't have even reversed a singly linked list. In the end, I gave the best marks to the candidates who: 1. Asked whether it was okay for duplicates to appear in the output. 2. Wrote up a roughly correct (syntax errors and off-by-one errors aside) implementation of their first idea for solving the problem and explained how it worked. 3. Understood when I pointed out permutations that their solutions would miss. 4. Were a good sport about being asked to solve the problem. (Actually, nobody who failed this criteria did remotely well at the others, so it's somewhat redundant. I mention it because I was surprised that people would get crabby about being asked to write code in an interview.) Three or four of the candidates, out of about a dozen, managed to meet all four criteria. These were people who passed a phone screen! Even worse, some of the candidates who failed one or more of the four criteria above apparently shined in their other interviews, claiming to have been in charge of designing and implementing significant systems.

lincolnq 16y ago
Solving this problem (permutations) in Haskell helped me understand the list monad, which in turn helped me think about nested-loop algorithms in general and how they should be structured in code. So thanks. It was a fun exercise and I learned something. I am pretty sure I nailed it. But I have been asked questions like this at interviews and screwed them up (e.g., at Facebook, which asked me something very similar and I failed to get it right). The notable thing was that at home, sitting at my desk, I wasn't afraid of sitting silently and figuring it out. It took me about 3 minutes of silence for my brain to organize itself enough to solve the problem. I'm not willing to spend 3 minutes in silence at an interview, and yet if I just jumped in I would go down a lot of wrong alleys, become confused, and screw it up. Is there room at an interview for someone to silently figure something out, without someone waiting on you, asking to "show me your thought process"?
- mechanical_fish 16y ago
  Is there room at an interview for someone to silently figure something out...? Frankly? No. Not for three minutes, anyway. That's why either (a) the permutations question is too hard, or (b) the interviewer cannot necessarily expect anyone to get the right answer, but should instead judge incidental aspects of the process. (This is what dkarl seems to be doing, and that's fine.) If you are asked a hard question, I think it's better to steer on the side of "going down a lot of wrong alleys, out loud". Just be honest about it. If you find yourself lost, go ahead and say "oops, thinking out loud during interviews is not my strong suit and now I'm pretty sure I'm lost". Find your own bugs and laugh at them. Backtrack. Start over. Restate the problem. Sketch a unit test that describes the answer that you can't quite derive yet. Define a simpler problem that you can solve out loud, and solve that first. And, of course, always do the stupidest thing that could possibly work. It occurs to me that the average student spends all kinds of time in school studying how to solve problems quietly and completely at one's desk, and very little time practicing the art of sketching half-baked solutions out loud while waving one's hands. But you can practice that.
  - aristus 16y ago
    "the art of sketching half-baked solutions out loud while waving one's hands" You mean, teaching?
    - mechanical_fish 16y ago
      Precisely.
- dkarl 16y ago
  The people who did best tended to have long silences at first and get more chatty as they went on. The ones who dove right in coding and chatting did poorly. I held that against them as a sign that they would always code first and think about correctness later, which may have been unfair. Maybe I'll make "talk as you go" optional next time, in case it pressures people down the wrong path.
- calibraxis 16y ago
jules 16y ago
Permutations are much harder than reversing a linked list. Actually I'm interested in what solution you had in mind. I can only think of these three: 1) Do it recursively like permutations(xs) = [[x] + p for p in permutations(xs - [x]) for x in xs]. 2) Systematically generate all combinations of letters, only print the ones that are permutations of the original. 3) Systematically generate the permutations in lexicographical order. All three would require pretty significant effort to get correct on a whiteboard in a language like C (without high level list comprehensions & operations).
- waterhouse 16y ago
  Regarding your last sentence, he did say: > write code, in any real language (i.e., not pseudocode,) to print out the permutations of a string
- dkarl 16y ago
  1) is what I was expecting. The solutions I saw were either attempts at 1) or attempts to do 3) using nested loops. There was also one guy who spent an hour trying to solve it with increasingly intricate methods of swapping the letters around in the string, I think because the first algorithmy thing that came to his mind was balancing binary trees. (He kept sketching the string as a tree rooted at the first letter.) Choosing C would be a pretty big fail, though honestly, people didn't struggle much with language issues. I told them it was fine if they didn't remember exact APIs, and nobody got to the point where they were worried about whether the code would parse. One guy did it in C++ using the string class, and I don't think he was at a big disadvantage. It's (evidently) harder to figure out the details of the algorithm than to write out a solution in C++.
  - jules 16y ago
    Having just spent half an hour to figure out (3) by swapping letters around, I think (1) may still be easier even in C because the method is straightforward. Are you using functional languages where you work? Interviewees who use a language with list comprehensions would have a significantly easier time with this question (the solution being two lines of code instead of 10+ lines of intricate letter swapping). Did any of the candidates use Python or ML or Haskell? What is the reason they didn't get it right? It may be harder to figure out idea of the algorithm than the implementation details, but in my experience you often won't even think about the high level algorithm if you are in a C++ mindset, you'll only think about swapping letters around.
Daniel_Newby 16y ago
"... give me running commentary on what they were doing ..." There's your problem! You are asking people to simultaneously (1) design a divide-and-conquer algorithm, (2) write syntactically-correct code on a whiteboard, an unfamiliar mode of writing for most people, and (3) give a comprehensible oral presentation, which creates a lot of social and emotional pressure for many people. Distraction is the death of reasoned thought. Break the interview into single-purpose phases. (1) Help them write down a clear description with some examples. (2) Have them cook up pseudocode or diagrams of how to solve it, with review/debugging from you. (3) Have them write code. (4) Have them explain why it works. As for being crabby, probably a few of them saw, but could not articulate, that the micromanagement set them up to fail. They should handle it better, but it's understandable. If you want to evaluate their willingness to graciously write code, start the interview with a dead simple FizzBuzz style challenge. E.g., write a function that takes an array of integers as input and copies them to an output array in reverse order. Specifically tell them they can ask questions or ask for help, but need not explain anything, that the code will speak for itself.
ajuc 16y ago
I would probably fail at your interview. It is considerably harder than fizz-buzz or reversing linked list (where no thinking is needed, so stress won't matter), and I would be stressed that it takes me too much time to think this out, so I better start some coding and talking. Things can only get worse from that place :)

by dkarl 16y ago

lincolnq 16y ago

Solving this problem (permutations) in Haskell helped me understand the list monad, which in turn helped me think about nested-loop algorithms in general and how they should be structured in code. So thanks. It was a fun exercise and I learned something. I am pretty sure I nailed it. But I have been asked questions like this at interviews and screwed them up (e.g., at Facebook, which asked me something very similar and I failed to get it right). The notable thing was that at home, sitting at my desk, I wasn't afraid of sitting silently and figuring it out. It took me about 3 minutes of silence for my brain to organize itself enough to solve the problem. I'm not willing to spend 3 minutes in silence at an interview, and yet if I just jumped in I would go down a lot of wrong alleys, become confused, and screw it up. Is there room at an interview for someone to silently figure something out, without someone waiting on you, asking to "show me your thought process"?

mechanical_fish 16y ago
Is there room at an interview for someone to silently figure something out...? Frankly? No. Not for three minutes, anyway. That's why either (a) the permutations question is too hard, or (b) the interviewer cannot necessarily expect anyone to get the right answer, but should instead judge incidental aspects of the process. (This is what dkarl seems to be doing, and that's fine.) If you are asked a hard question, I think it's better to steer on the side of "going down a lot of wrong alleys, out loud". Just be honest about it. If you find yourself lost, go ahead and say "oops, thinking out loud during interviews is not my strong suit and now I'm pretty sure I'm lost". Find your own bugs and laugh at them. Backtrack. Start over. Restate the problem. Sketch a unit test that describes the answer that you can't quite derive yet. Define a simpler problem that you can solve out loud, and solve that first. And, of course, always do the stupidest thing that could possibly work. It occurs to me that the average student spends all kinds of time in school studying how to solve problems quietly and completely at one's desk, and very little time practicing the art of sketching half-baked solutions out loud while waving one's hands. But you can practice that.
- aristus 16y ago
  "the art of sketching half-baked solutions out loud while waving one's hands" You mean, teaching?
  - mechanical_fish 16y ago
    Precisely.
dkarl 16y ago
The people who did best tended to have long silences at first and get more chatty as they went on. The ones who dove right in coding and chatting did poorly. I held that against them as a sign that they would always code first and think about correctness later, which may have been unfair. Maybe I'll make "talk as you go" optional next time, in case it pressures people down the wrong path.
calibraxis 16y ago

jules 16y ago

Permutations are much harder than reversing a linked list. Actually I'm interested in what solution you had in mind. I can only think of these three: 1) Do it recursively like permutations(xs) = [[x] + p for p in permutations(xs - [x]) for x in xs]. 2) Systematically generate all combinations of letters, only print the ones that are permutations of the original. 3) Systematically generate the permutations in lexicographical order. All three would require pretty significant effort to get correct on a whiteboard in a language like C (without high level list comprehensions & operations).

waterhouse 16y ago
Regarding your last sentence, he did say: > write code, in any real language (i.e., not pseudocode,) to print out the permutations of a string
dkarl 16y ago
1) is what I was expecting. The solutions I saw were either attempts at 1) or attempts to do 3) using nested loops. There was also one guy who spent an hour trying to solve it with increasingly intricate methods of swapping the letters around in the string, I think because the first algorithmy thing that came to his mind was balancing binary trees. (He kept sketching the string as a tree rooted at the first letter.) Choosing C would be a pretty big fail, though honestly, people didn't struggle much with language issues. I told them it was fine if they didn't remember exact APIs, and nobody got to the point where they were worried about whether the code would parse. One guy did it in C++ using the string class, and I don't think he was at a big disadvantage. It's (evidently) harder to figure out the details of the algorithm than to write out a solution in C++.
- jules 16y ago
  Having just spent half an hour to figure out (3) by swapping letters around, I think (1) may still be easier even in C because the method is straightforward. Are you using functional languages where you work? Interviewees who use a language with list comprehensions would have a significantly easier time with this question (the solution being two lines of code instead of 10+ lines of intricate letter swapping). Did any of the candidates use Python or ML or Haskell? What is the reason they didn't get it right? It may be harder to figure out idea of the algorithm than the implementation details, but in my experience you often won't even think about the high level algorithm if you are in a C++ mindset, you'll only think about swapping letters around.

Daniel_Newby 16y ago

"... give me running commentary on what they were doing ..." There's your problem! You are asking people to simultaneously (1) design a divide-and-conquer algorithm, (2) write syntactically-correct code on a whiteboard, an unfamiliar mode of writing for most people, and (3) give a comprehensible oral presentation, which creates a lot of social and emotional pressure for many people. Distraction is the death of reasoned thought. Break the interview into single-purpose phases. (1) Help them write down a clear description with some examples. (2) Have them cook up pseudocode or diagrams of how to solve it, with review/debugging from you. (3) Have them write code. (4) Have them explain why it works. As for being crabby, probably a few of them saw, but could not articulate, that the micromanagement set them up to fail. They should handle it better, but it's understandable. If you want to evaluate their willingness to graciously write code, start the interview with a dead simple FizzBuzz style challenge. E.g., write a function that takes an array of integers as input and copies them to an output array in reverse order. Specifically tell them they can ask questions or ask for help, but need not explain anything, that the code will speak for itself.

ajuc 16y ago

I would probably fail at your interview. It is considerably harder than fizz-buzz or reversing linked list (where no thinking is needed, so stress won't matter), and I would be stressed that it takes me too much time to think this out, so I better start some coding and talking. Things can only get worse from that place :)