3 ms·

"work sample test" I've given this same advice for _years_ for designing technical interviews. Take a problem that you or your team has actually faced in the l

"work sample test" I've given this same advice for _years_ for designing technical interviews. Take a problem that you or your team has actually faced in the last n months. Distill it down to remove tribal knowledge and get it to something you or a capable colleague can hammer out in m minutes. Give the candidate 3m minutes to solve it. Typically this means it should take you 15 minutes to solve and give them 45 minutes. Our biggest hit for a question that followed this for an SRE type role: we had a cron job that was legacy and buggy and caused a couple of issues over 6 months. We went back and created a very similar script with very similar bugs, added some similar data and re-seeding scripts, and told the candidates "customers say that feature $x is not working, and that is ran by this script. Here is how to run it, here is the user's id that can be found in the data seed." We added some other data validation problems. After they fixed it, we asked them what they would do to make this more production safe and dig in on what they have and have not done for improving existing systems. We expected discussions on structured logging, metrics, alerting, testing, code layout, etc. Over time, the roles that you are looking to fill and the weaknesses on the existing team that you want to shore up will change and you will need different interview questions. Maybe more towards distributed systems or more towards testing or observability.

arthurcolle 4y ago
3m -> 30m presumably?
- akerl_ 4y ago
  > get it to something you or a capable colleague can hammer out in m minutes. Give the candidate 3m minutes to solve it I think they're using "m" there as a variable, to frame that you should give the candidate 3 times however long it takes for an insider to solve it.
  - arthurcolle 4y ago
    The usage seems inconsistent. I will check and report back to the Hive
- xarope 4y ago
  > Typically this means it should take you 15 minutes to solve and give them 45 minutes They mean 3 x m where m is how much time you think it would take for you to solve; so the example is: if it takes you 15mins, then give the interviewee 3 x 15 = 45mins
  - arthurcolle 4y ago
    Sorry, missed this. Thanks
- nicbou 4y ago
  3m minutes is already shy of 6 years. I know that tech interviews are long but this is excessive.
bolanyo 4y ago
This is the right answer. Questions of this type test for both experience (what you want from non-junior candidates - the fact that they have learned to program defensively, check edge cases, etc) and common sense (what you want from junior candidates - not making silly simplifying assumptions, thinking divergently about what could go wrong, etc). Importantly, it also allows you to examine fluency. The candidate who can write a code/pseudocode solution as fast as they can type, and explain it verbally at the same time, is much more impressive than the candidate who pauses for a long time before starting haltingly and unconfidently. Fluency in solving leetcode problems just tests for having practiced leetcode, but fluency in writing real code tests for actual productivity. If you have a question like this, you can drill down into performance characteristics just as the question in TFA does. Not necessarily using big-Oh notation, but "what if you have to do this millions of times? what if you have to provide results immediately? what if some of the files are too big to fit into RAM?". But you can also drill down into other aspects of the problem like changing specs, noisy input, constrained resources, observability. FWIW, when I came up with a question like this (with the constraint that it had to represent a problem I had solved that morning), ChatGPT was not able to solve it correctly.
noisy_boy 4y ago
> Our biggest hit for a question that followed this for an SRE type role: we had a cron job that was legacy and buggy and caused a couple of issues over 6 months. We went back and created a very similar script with very similar bugs, added some similar data and re-seeding scripts, and told the candidates "customers say that feature $x is not working, and that is ran by this script. Here is how to run it, here is the user's id that can be found in the data seed." We added some other data validation problems. After they fixed it, we asked them what they would do to make this more production safe and dig in on what they have and have not done for improving existing systems. We expected discussions on structured logging, metrics, alerting, testing, code layout, etc. No wonder it is a hit. It is an actual problem that makes me want to go for it - I want to take an interview like this, instead of the usual of dreading interviews! Keep it up.

arthurcolle 4y ago

3m -> 30m presumably?

akerl_ 4y ago
> get it to something you or a capable colleague can hammer out in m minutes. Give the candidate 3m minutes to solve it I think they're using "m" there as a variable, to frame that you should give the candidate 3 times however long it takes for an insider to solve it.
- arthurcolle 4y ago
  The usage seems inconsistent. I will check and report back to the Hive
xarope 4y ago
> Typically this means it should take you 15 minutes to solve and give them 45 minutes They mean 3 x m where m is how much time you think it would take for you to solve; so the example is: if it takes you 15mins, then give the interviewee 3 x 15 = 45mins
- arthurcolle 4y ago
  Sorry, missed this. Thanks
nicbou 4y ago
3m minutes is already shy of 6 years. I know that tech interviews are long but this is excessive.

bolanyo 4y ago

This is the right answer. Questions of this type test for both experience (what you want from non-junior candidates - the fact that they have learned to program defensively, check edge cases, etc) and common sense (what you want from junior candidates - not making silly simplifying assumptions, thinking divergently about what could go wrong, etc). Importantly, it also allows you to examine fluency. The candidate who can write a code/pseudocode solution as fast as they can type, and explain it verbally at the same time, is much more impressive than the candidate who pauses for a long time before starting haltingly and unconfidently. Fluency in solving leetcode problems just tests for having practiced leetcode, but fluency in writing real code tests for actual productivity. If you have a question like this, you can drill down into performance characteristics just as the question in TFA does. Not necessarily using big-Oh notation, but "what if you have to do this millions of times? what if you have to provide results immediately? what if some of the files are too big to fit into RAM?". But you can also drill down into other aspects of the problem like changing specs, noisy input, constrained resources, observability. FWIW, when I came up with a question like this (with the constraint that it had to represent a problem I had solved that morning), ChatGPT was not able to solve it correctly.

noisy_boy 4y ago

> Our biggest hit for a question that followed this for an SRE type role: we had a cron job that was legacy and buggy and caused a couple of issues over 6 months. We went back and created a very similar script with very similar bugs, added some similar data and re-seeding scripts, and told the candidates "customers say that feature $x is not working, and that is ran by this script. Here is how to run it, here is the user's id that can be found in the data seed." We added some other data validation problems. After they fixed it, we asked them what they would do to make this more production safe and dig in on what they have and have not done for improving existing systems. We expected discussions on structured logging, metrics, alerting, testing, code layout, etc. No wonder it is a hit. It is an actual problem that makes me want to go for it - I want to take an interview like this, instead of the usual of dreading interviews! Keep it up.