4 ms·

The problem is though how do you evaluate for real work without a portfolio of experience? Real work projects take weeks - perhaps months - to design, build, re

The problem is though how do you evaluate for real work without a portfolio of experience? Real work projects take weeks - perhaps months - to design, build, review and release. How do you test for that, really? I agree leetcode style interviews are artificial, but I think they persist because few people have identified and popularised effective alternatives. At least with leetcode you've shown people in front of you have been prepared to go learn arcane stuff and apply it. It's not good, but it's better than nothing. I was once asked to do the Gilded Rose kata[1] for one ~200 headcount Series D "startup", and it not only resembled real work - it's a refactoring problem - but it also showed their engineering culture. When I joined, I found smart people who weren't leetcode robots, but thoughtful engineers trying to solve tricky engineering problems. I "stole" Gilded Rose to use in other companies when interviewing until I joined my current employer (a FAANG, heavily prescribed process), with great success. I would like to see more katas that are as good at testing real world skills. Also, something I've only ever been interviewed for twice in 25+ years, which I think is underplayed: pair programming and handling code review feedback. Do this more, please. If you hire people not knowing how they're going to respond to a principal engineer telling them "we need you to think about 4 other things that weren't in the original scope given to you by the PM, can you please go deal with them", why are you surprised when toys are subsequently thrown out of metaphorical prams? [1] https://github.com/emilybache/GildedRose-Refactoring-Kata https://github.com/emilybache/GildedRose-Refactoring-Kata

dkdbejwi383 3y ago
Thanks for sharing, I've not come across this kata before. Seems like it'll be a fun one.
jebarker 3y ago
This seems like a better way to evaluate real-world coding skills. Did you use it in a standard 45-60 min interview or as a take home problem? It seems like it'd take a decent amount of time to read the requirements and just introduce the task.
- PaulRobinson 3y ago
  Take home and “please time box it to maybe an hour maximum, you’re probably doing a load of interviews and tech tests at the moment, so you don’t need to finish: I just want to see your approach”. In-person time I used to like to ask a question I got from here: “Tell me about your favourite coding work or side project”. We’d then dig into choices, trade offs, obstacles and how they got around them. My favourite answer to this was “an OpenGL implementation for the X Window protocol which I wrote in Common Lisp”. Just getting through the first layer of “Why [x]?”, took 15 minutes and we spent an hour talking about all the facets. Hired him, was a superb colleague.
  - kaashif 3y ago
    "Please timebox to an hour" is a bit dodgy though. If one candidate really does that, but the rest give it an extra 2 or 3 hours, that 1 hour candidate will probably be at a disadvantage.
    - neon_electro 3y ago
      Thank you for mentioning this. I always find myself going further back and forth with employers who allow a take-home exercise but ask that it be time-boxed nonetheless; it's difficult to try to unpack how they're trying to communicate what kind of time pressure a role experiences, and that if I took longer than their expected time box, does that look bad just because I needed some more time to understand the problem?
      - remich 3y ago
        I don't think most employers are out there looking at the timestamps on your commits to see if you really completed it within the timebox or not -- I certainly wouldn't. I don't think that candidates who spend less time or turn in incomplete take homes are necessarily at a disadvantage. Sooo much can be discerned from a take home even if it's not finished. I can evaluate a candidate's familiarity with modern syntax, how they organize functions, how readable their code is, whether or not they used ChatGPT/CoPilot or copy/pasted from an online tutorial (surprisingly easy to discern when you're evaluating many submissions), and so on. All of that tells me a lot about how well someone functions as an engineer, as well as what level they're operating at, and it doesn't require the completion of the take home problem.
nine_zeros 3y ago
Refactoring is the best kind of test! It has a slight bias for programming languages but the bias can be mitigated by having refactorable libraries in many languages.
- nomel 3y ago
  I'm fairly certain I would not perform well with this, unless the library was trivial. Refactoring is a task that's mostly about reading, understanding context, and a bit of rumination. I think having them design the library, or a fraction of it, from scratch, would give a more momentum.
  - nine_zeros 3y ago
    > I think having them design the library, or a fraction of it, from scratch, would give a more momentum. I have done this as a take home test and I am ok with this as long as it doesn't take more than 5 hours.
    - nomel 3y ago
      I would never allow this. I need to know that they generated the answer. In several video interviews, I've had people copy paste the first google hit, and one used co-programmer, listening and writing the answers in another window. The second was good at coding, but couldn't explain a single line of it. I think it's only safe to assume that anyone with a take home will cheat. But, it's also very hard to fire people, in our group.
MichaelZuo 3y ago
It seems fairly straightforward to just ask verbal questions? For example, if the position is mostly coding C then just get them to explain how some X interacts with some Y and how that interacts with some Z. Maybe with a copy of of K&R and get them to point to page so and so. Do that a few times, maybe use a whiteboard, and I think any experienced C developer will be able to tell who's faking it and who really understands within 2 or 3 iterations. Even if there are no experienced folks in the company, then it will take longer but, after an hour or two it'd be pretty difficult for anyone to fake a solid understanding of hundreds of pages of K&R.
- PaulRobinson 3y ago
  Not the point you’re making but in a C shop, if I interview and somebody pulls out K&R in 2023, they’re not getting hired. I get the wider point you’re making, and yes, a chat can help. Elsewhere in a reply to this I talk about how I personally like to do that, but the problem is, a significant number of people can talk about coding but can’t actually code. I once interviewed somebody who had passed 3 previous calls. Asked them to implement fizzbuzz. Couldn’t. That is a real problem. It’s not myth. As such, I’m not hiring somebody without seen them writing code the same way I wouldn’t hire a designer without seeing their actual personal portfolio.
  - nomel 3y ago
    > a significant number of people can talk about coding but can’t actually code. We removed pseudocode from our pre-screening due to this. Some people are absolutely great with English and pseudocode, but can't write a for loop, in their "preferred" language of their choosing. I'm sure they could be great at programming...eventually.
  - Apocryphon 3y ago
    A Zed Shaw's Learn C the Hard Way fan, I see.
  - MichaelZuo 3y ago
    Asking people face-to-face is a lot different then asking over a phone, it's easily 1/10th the info, or less, so of course the percentage of fakers is much higher.
VoodooJuJu 3y ago
>I think they persist because few people have identified and popularised effective alternatives Astrology persists, but that's not a testament to its efficacy.

dkdbejwi383 3y ago

Thanks for sharing, I've not come across this kata before. Seems like it'll be a fun one.

jebarker 3y ago

This seems like a better way to evaluate real-world coding skills. Did you use it in a standard 45-60 min interview or as a take home problem? It seems like it'd take a decent amount of time to read the requirements and just introduce the task.

PaulRobinson 3y ago
Take home and “please time box it to maybe an hour maximum, you’re probably doing a load of interviews and tech tests at the moment, so you don’t need to finish: I just want to see your approach”. In-person time I used to like to ask a question I got from here: “Tell me about your favourite coding work or side project”. We’d then dig into choices, trade offs, obstacles and how they got around them. My favourite answer to this was “an OpenGL implementation for the X Window protocol which I wrote in Common Lisp”. Just getting through the first layer of “Why [x]?”, took 15 minutes and we spent an hour talking about all the facets. Hired him, was a superb colleague.
- kaashif 3y ago
  "Please timebox to an hour" is a bit dodgy though. If one candidate really does that, but the rest give it an extra 2 or 3 hours, that 1 hour candidate will probably be at a disadvantage.
  - neon_electro 3y ago
    Thank you for mentioning this. I always find myself going further back and forth with employers who allow a take-home exercise but ask that it be time-boxed nonetheless; it's difficult to try to unpack how they're trying to communicate what kind of time pressure a role experiences, and that if I took longer than their expected time box, does that look bad just because I needed some more time to understand the problem?
    - remich 3y ago
      I don't think most employers are out there looking at the timestamps on your commits to see if you really completed it within the timebox or not -- I certainly wouldn't. I don't think that candidates who spend less time or turn in incomplete take homes are necessarily at a disadvantage. Sooo much can be discerned from a take home even if it's not finished. I can evaluate a candidate's familiarity with modern syntax, how they organize functions, how readable their code is, whether or not they used ChatGPT/CoPilot or copy/pasted from an online tutorial (surprisingly easy to discern when you're evaluating many submissions), and so on. All of that tells me a lot about how well someone functions as an engineer, as well as what level they're operating at, and it doesn't require the completion of the take home problem.

nine_zeros 3y ago

Refactoring is the best kind of test! It has a slight bias for programming languages but the bias can be mitigated by having refactorable libraries in many languages.

nomel 3y ago
I'm fairly certain I would not perform well with this, unless the library was trivial. Refactoring is a task that's mostly about reading, understanding context, and a bit of rumination. I think having them design the library, or a fraction of it, from scratch, would give a more momentum.
- nine_zeros 3y ago
  > I think having them design the library, or a fraction of it, from scratch, would give a more momentum. I have done this as a take home test and I am ok with this as long as it doesn't take more than 5 hours.
  - nomel 3y ago
    I would never allow this. I need to know that they generated the answer. In several video interviews, I've had people copy paste the first google hit, and one used co-programmer, listening and writing the answers in another window. The second was good at coding, but couldn't explain a single line of it. I think it's only safe to assume that anyone with a take home will cheat. But, it's also very hard to fire people, in our group.

MichaelZuo 3y ago

It seems fairly straightforward to just ask verbal questions? For example, if the position is mostly coding C then just get them to explain how some X interacts with some Y and how that interacts with some Z. Maybe with a copy of of K&R and get them to point to page so and so. Do that a few times, maybe use a whiteboard, and I think any experienced C developer will be able to tell who's faking it and who really understands within 2 or 3 iterations. Even if there are no experienced folks in the company, then it will take longer but, after an hour or two it'd be pretty difficult for anyone to fake a solid understanding of hundreds of pages of K&R.

PaulRobinson 3y ago
Not the point you’re making but in a C shop, if I interview and somebody pulls out K&R in 2023, they’re not getting hired. I get the wider point you’re making, and yes, a chat can help. Elsewhere in a reply to this I talk about how I personally like to do that, but the problem is, a significant number of people can talk about coding but can’t actually code. I once interviewed somebody who had passed 3 previous calls. Asked them to implement fizzbuzz. Couldn’t. That is a real problem. It’s not myth. As such, I’m not hiring somebody without seen them writing code the same way I wouldn’t hire a designer without seeing their actual personal portfolio.
- nomel 3y ago
  > a significant number of people can talk about coding but can’t actually code. We removed pseudocode from our pre-screening due to this. Some people are absolutely great with English and pseudocode, but can't write a for loop, in their "preferred" language of their choosing. I'm sure they could be great at programming...eventually.
- Apocryphon 3y ago
  A Zed Shaw's Learn C the Hard Way fan, I see.
- MichaelZuo 3y ago
  Asking people face-to-face is a lot different then asking over a phone, it's easily 1/10th the info, or less, so of course the percentage of fakers is much higher.

VoodooJuJu 3y ago

>I think they persist because few people have identified and popularised effective alternatives Astrology persists, but that's not a testament to its efficacy.