7 ms·

> Real-life tests are THE best thing to send job candidates I agree, but only if you're allowed to use references/google/etc and given a reasonable amount of t

> Real-life tests are THE best thing to send job candidates I agree, but only if you're allowed to use references/google/etc and given a reasonable amount of time to accomplish it. I've had a "real-life" test where I wasn't allowed to verify or look up information, or where I'm giving a very short time to execute, and I've always thought those were absurd. The idea that you would ever have to do something like this in a bomb-diffusing type of scenario with no internet and a ticking clock, would only test if you have already solved the problem before, not whether you could investigate a new problem.

davidgerard 12y ago
Yeah, this is an open-book test. Of course, there's tests like user:B5Geek's, where the candidate gets a PC with a slightly-unplugged network cable and has to realise this fact.
- michaelchisari 12y ago
  Yeah, but B5Geek's is a pretty entry-level thing. I've encountered whiteboard tests where you have to recreate an algorithm that someone spent their PhD thesis creating. It seems like a good way to find "geniuses", but it just isn't practical. In a lot of ways, it's a matter of identity crisis for software developers. We're all, industry included, not quite sure whether we're mechanics, carpenters, architects or scientists.
  - davidgerard 12y ago
    True. Testing for sysadmins is easier: you're after a way of thinking. So a competence test ("can you do what you claimed?") with a freeform "keep notes on your thinking along the way" is quite informative. You still need to interview, of course. I confess I don't know how I'd apply this to developers. They pass fizzbuzz, OK - what do you do next?
    - michaelchisari 12y ago
      For developers that have the academics, but not the experience, I've always said that the industry needs to adopt the apprenticeship programs that other trades have. Where you do a mixture of work for the company, and trade schooling run by the industry to teach you the specific skills you need. For example, an IBEW apprentice will do a lot of gruntwork on the job site. Pulling wire, for instance. But they'll get exposed to how things are done, to trade industry expectations, and what a project looks like from start to end. Part of the week, they spend in a classroom learning the specifics of their trade. Not just the high level theory, but the nitty gritty, based on real world applications and problems. This apprenticeship lasts from 2-5 years depending. I don't see why this model wouldn't work really well with programming.
      - porpoisemonkey 12y ago
        Apprenticeship requires a constant stream of mentors. I'm not sure there is an ample supply of professional programmers who would be interested in taking on the responsibility of mentoring in addition to their full-time jobs.
        smoe 12y ago
        It really depends what those professionals are used to. I did an Apprenticeship as Software Developer in Switzerland. That is, 1 year of school followed by 3 years alternating between the company and the school. So once you start at the company you already have a viable amount of knowledge in programming, databases, networking, etc and supposed to be able to work on your own. You don't get constant mentoring at the company, but can ask one of the developers or fellow apprentices when you're stuck with one of the tasks you get assigned. Or for example, any time a developer had a small task that he felt, I couldn't do by myself, or it was too time critical he just asked me to sit beside him and watch, listen and take notes while he did the work. How it works is different in any company but the outcome should be the same: real world experience on real projects with real clients and their time/budget constraints. That means, once your finished and a professional yourself, it isn't a question of interest but a completely normal thing to help out the apprentices. Because that's how you got to the point where you are.
  - aftbit 12y ago
    I've taken a whiteboard test that asked me to come up with an algorithm that was difficult for me. I got the job even though my solution was horribly inefficient (and I knew it), and the interviewer later told me that the point wasn't to see what algorithm I knew - it was to see if I acted like an ass when I didn't know stuff.
    - michaelchisari 12y ago
      > it was to see if I acted like an ass when I didn't know stuff Surely there's more efficient ways of evaluating that.
      - vegggdor 12y ago
        On the plus side, it did take a stab at what algorithms he knew, too.
    - lostcolony 12y ago
      Not sure that's a good test either. When tasked to figure out something as part of a job, by a manager one knows and trusts, who takes feedback of "I don't know how to do this but I will investigate it", a person may act one way, whereas being tasked with someone's PhD thesis by a stranger on an interview may lead to a very different response. I know I am far less tolerant of bullshit interviews than I once was, and would have far fewer qualms vocalizing what I'm thinking - "I don't know. If I saw this in real life I'd check Google, because it reads exactly like the sort of contrived problem used in interviews and thus is well documented, and likely solved quickly by those who have memorized it. I am not one of those. I'll take a stab at it now if you just want to see how I reason through it, but if coming up with an efficient, correct implementation is part of your criteria we can probably save both of us the trouble" - is that considered acting like an ass?
icelancer 12y ago
>I agree, but only if you're allowed to use references/google/etc and given a reasonable amount of time to accomplish it. I've had a "real-life" test where I wasn't allowed to verify or look up information, or where I'm giving a very short time to execute, and I've always thought those were absurd. Employers that do this are ridiculous. As are educators, too. What is this, preparation for when coders are kidnapped by terrorists and forced to recite how to tar/untar stuff on command? (relevant xkcd, of course)
- michaelchisari 12y ago
  > Employers that do this are ridiculous. In my experience, they're usually startups or smaller companies that are trying to "copy google" in their hiring practices, but don't fully think out how they're doing it. It's not a good sign.
  - hueving 12y ago
    Well Google does it too. Their 'gang bang' interviews are notorious for weeding out people that don't perform well under short term pressure like that. "just write an algorithm for a stack on this whiteboard with a big O complexity analysis and memory usage pattern in 15 minutes while we all stare at you. What? You don't work well like this? Well this is how elite Googlers work all of the time so GTFO"
    - redstar504 12y ago
      To be fair, a lot of people are capable of becoming that type of employee after gaining comfort at the company, and after building relationships with their co-workers. To expect someone to do that in front of strangers while under pressure doesn't necessarily correlate with a programmer who can evolve into such a person.
- eyko 12y ago
  > What is this, preparation for when coders are kidnapped by terrorists and forced to recite how to tar/untar stuff on command? :-D you never know!
kamaal 12y ago
For any person who claims to have years of Unix experience, they can easily do anything they want using man pages. Though I agree with you on the general premise though. One must be allowed to look up information they need to solve a problem. >>I've had a "real-life" test where I wasn't allowed to verify or look up information, or where I'm giving a very short time to execute, and I've always thought those were absurd. You certainly haven't worked at major IT services firms here in India. I spent early part of working years (around 2006) coding by purely reading books and discussing things with folks on internal mailing list. Because only managers and levels above were allowed internet access. >>The idea that you would ever have to do something like this in a bomb-diffusing type of scenario with no internet and a ticking clock, would only test if you have already solved the problem before, not whether you could investigate a new problem. This is unfortunately true even for coding interviews. Most algorithm questions are impossible to answer unless you have rote memorized the algorithms and their implementations.
- mjmahone17 12y ago
  If you have to give white-board coding interviews, the goal should be to have a question that's complex, potentially elegant and straightforward. By complex, I mean the question should be difficult enough that you can't easily hold every parameter, case and step in your head: I want to see how you deal with situations that you have to mentally walk through and issues that you might lose track of. By potentially elegant, I mean both brute-force and optimal solutions should be able to fit on a small whiteboard, without requiring more than 2-4 variable declarations and no more than a nested for loop (or some sort of recursion). The optimal solution shouldn't require putting together pieces that they aren't really expected to know/use, i.e. no more complex than arrays, maps, strings and ints, and any other "data structures" the candidate ought to know, given their background. For instance, if they claim to be a crypto expert, they should know how to use a black box that gives them private and public keys. By straightforward I mean, you don't want the candidate to have to have an "aha" moment. If you're asking for them to write a max-flow variant, you'd better be explicit that you don't mind a really brute-force solution, or most of the interview will be spent trying to remember complex algorithms. Is the algorithm one that you can understand by walking through test cases/figuring out what steps you took in your mind to go from input to output? If it takes more than that, you're going to get a few false positives, i.e. "Wow she was great! No one's answered this so fast!" (because she just spent 20 hours doing this on a homework), and many false negatives.
jtheory 12y ago
It's gets a bit complicated when your job candidates may find your test interesting enough that they post a complete walk-through of the solution online. The candidate was kind enough to avoid posting the name of the company (to hopefully hide the walk-through from Google queries for "crack me hiring test company name", but an HN discussion might then arise around the walk-through, in which the company would be named, and well, there goes your test. It's hard to accurately run "take home" tests when hiring because of just this.
- Fuxy 12y ago
  If you're not changing your tests on every interview batch you're doing it wrong. You're asking a potential candidate to spend hours cracking your program but you can't be bothered to change it for each batch? I expect any company applying this interview technique to at least put some effort into it and even then it pays off since a batch could contain hundreds of candidates and you only have to build the program once.
  - aaronharnly 12y ago
    What do you mean by "batch"? Most companies have rolling hiring.
    - warrenm 12y ago
      this is true - but you're [generally] going to have a batch of candidates for any given position
  - chrisbennet 12y ago
    This ^ "You only get out of it, what you put into it." goes both ways - employers should expect to extend some effort if they expect their candidates to extend some effort. I strongly suspected that sometimes goals of the person responsible for hiring are not quite aligned with the company they work for. Example: Company's goal: Hire an exceptional candidate. Lazy or overworked hiring manager's goal: Reduce the size of the stack of 1000's of resume's I have to go through using any means possible that will make it appear like he/she actually did the work to find the candidate. "Lemme see, use keyword search to ditch the ones without 'Github' on their resume'. Now ditch the ones who don't have a college degree. Still some resume's left? OK, how about tossing the ones without a CS degree?" Run a filter with enough conditions and you can get that pile down to nothing pretty fast. If that is still not enough filtering, ask the candidates to do something very time consuming. This a great way to weed out people who are very busy or exceptional at what they do and don't need to jump through your hoops.
eyko 12y ago
I've been in two interviews where they asked me to code without googling or looking at the reference. Personally, I think it's as absurd as writing shell scripts on a computer with no `man`, or without being able to refresh your memory with `<command> --help`. Good luck with that! I was in an interview where they asked me to configure specific firewall rules on a notepad with no access to a terminal, so I couldn't check man iptables for stuff like how to match certain sources/destinations hosts/IPs. And I was not too familiar with the opts (was it --dest, --dst, or -d? and how do calculate the network mask for this range of IPs?). It was also for a junior position so I thought they were taking a piss at me and simply told them so.

davidgerard 12y ago

Yeah, this is an open-book test. Of course, there's tests like user:B5Geek's, where the candidate gets a PC with a slightly-unplugged network cable and has to realise this fact.

michaelchisari 12y ago
Yeah, but B5Geek's is a pretty entry-level thing. I've encountered whiteboard tests where you have to recreate an algorithm that someone spent their PhD thesis creating. It seems like a good way to find "geniuses", but it just isn't practical. In a lot of ways, it's a matter of identity crisis for software developers. We're all, industry included, not quite sure whether we're mechanics, carpenters, architects or scientists.
- davidgerard 12y ago
  True. Testing for sysadmins is easier: you're after a way of thinking. So a competence test ("can you do what you claimed?") with a freeform "keep notes on your thinking along the way" is quite informative. You still need to interview, of course. I confess I don't know how I'd apply this to developers. They pass fizzbuzz, OK - what do you do next?
  - michaelchisari 12y ago
    For developers that have the academics, but not the experience, I've always said that the industry needs to adopt the apprenticeship programs that other trades have. Where you do a mixture of work for the company, and trade schooling run by the industry to teach you the specific skills you need. For example, an IBEW apprentice will do a lot of gruntwork on the job site. Pulling wire, for instance. But they'll get exposed to how things are done, to trade industry expectations, and what a project looks like from start to end. Part of the week, they spend in a classroom learning the specifics of their trade. Not just the high level theory, but the nitty gritty, based on real world applications and problems. This apprenticeship lasts from 2-5 years depending. I don't see why this model wouldn't work really well with programming.
    - porpoisemonkey 12y ago
      Apprenticeship requires a constant stream of mentors. I'm not sure there is an ample supply of professional programmers who would be interested in taking on the responsibility of mentoring in addition to their full-time jobs.
      - smoe 12y ago
        It really depends what those professionals are used to. I did an Apprenticeship as Software Developer in Switzerland. That is, 1 year of school followed by 3 years alternating between the company and the school. So once you start at the company you already have a viable amount of knowledge in programming, databases, networking, etc and supposed to be able to work on your own. You don't get constant mentoring at the company, but can ask one of the developers or fellow apprentices when you're stuck with one of the tasks you get assigned. Or for example, any time a developer had a small task that he felt, I couldn't do by myself, or it was too time critical he just asked me to sit beside him and watch, listen and take notes while he did the work. How it works is different in any company but the outcome should be the same: real world experience on real projects with real clients and their time/budget constraints. That means, once your finished and a professional yourself, it isn't a question of interest but a completely normal thing to help out the apprentices. Because that's how you got to the point where you are.
- aftbit 12y ago
  I've taken a whiteboard test that asked me to come up with an algorithm that was difficult for me. I got the job even though my solution was horribly inefficient (and I knew it), and the interviewer later told me that the point wasn't to see what algorithm I knew - it was to see if I acted like an ass when I didn't know stuff.
  - michaelchisari 12y ago
    > it was to see if I acted like an ass when I didn't know stuff Surely there's more efficient ways of evaluating that.
    - vegggdor 12y ago
      On the plus side, it did take a stab at what algorithms he knew, too.
  - lostcolony 12y ago
    Not sure that's a good test either. When tasked to figure out something as part of a job, by a manager one knows and trusts, who takes feedback of "I don't know how to do this but I will investigate it", a person may act one way, whereas being tasked with someone's PhD thesis by a stranger on an interview may lead to a very different response. I know I am far less tolerant of bullshit interviews than I once was, and would have far fewer qualms vocalizing what I'm thinking - "I don't know. If I saw this in real life I'd check Google, because it reads exactly like the sort of contrived problem used in interviews and thus is well documented, and likely solved quickly by those who have memorized it. I am not one of those. I'll take a stab at it now if you just want to see how I reason through it, but if coming up with an efficient, correct implementation is part of your criteria we can probably save both of us the trouble" - is that considered acting like an ass?

icelancer 12y ago

>I agree, but only if you're allowed to use references/google/etc and given a reasonable amount of time to accomplish it. I've had a "real-life" test where I wasn't allowed to verify or look up information, or where I'm giving a very short time to execute, and I've always thought those were absurd. Employers that do this are ridiculous. As are educators, too. What is this, preparation for when coders are kidnapped by terrorists and forced to recite how to tar/untar stuff on command? (relevant xkcd, of course)

michaelchisari 12y ago
> Employers that do this are ridiculous. In my experience, they're usually startups or smaller companies that are trying to "copy google" in their hiring practices, but don't fully think out how they're doing it. It's not a good sign.
- hueving 12y ago
  Well Google does it too. Their 'gang bang' interviews are notorious for weeding out people that don't perform well under short term pressure like that. "just write an algorithm for a stack on this whiteboard with a big O complexity analysis and memory usage pattern in 15 minutes while we all stare at you. What? You don't work well like this? Well this is how elite Googlers work all of the time so GTFO"
  - redstar504 12y ago
    To be fair, a lot of people are capable of becoming that type of employee after gaining comfort at the company, and after building relationships with their co-workers. To expect someone to do that in front of strangers while under pressure doesn't necessarily correlate with a programmer who can evolve into such a person.
eyko 12y ago
> What is this, preparation for when coders are kidnapped by terrorists and forced to recite how to tar/untar stuff on command? :-D you never know!

kamaal 12y ago

For any person who claims to have years of Unix experience, they can easily do anything they want using man pages. Though I agree with you on the general premise though. One must be allowed to look up information they need to solve a problem. >>I've had a "real-life" test where I wasn't allowed to verify or look up information, or where I'm giving a very short time to execute, and I've always thought those were absurd. You certainly haven't worked at major IT services firms here in India. I spent early part of working years (around 2006) coding by purely reading books and discussing things with folks on internal mailing list. Because only managers and levels above were allowed internet access. >>The idea that you would ever have to do something like this in a bomb-diffusing type of scenario with no internet and a ticking clock, would only test if you have already solved the problem before, not whether you could investigate a new problem. This is unfortunately true even for coding interviews. Most algorithm questions are impossible to answer unless you have rote memorized the algorithms and their implementations.

mjmahone17 12y ago
If you have to give white-board coding interviews, the goal should be to have a question that's complex, potentially elegant and straightforward. By complex, I mean the question should be difficult enough that you can't easily hold every parameter, case and step in your head: I want to see how you deal with situations that you have to mentally walk through and issues that you might lose track of. By potentially elegant, I mean both brute-force and optimal solutions should be able to fit on a small whiteboard, without requiring more than 2-4 variable declarations and no more than a nested for loop (or some sort of recursion). The optimal solution shouldn't require putting together pieces that they aren't really expected to know/use, i.e. no more complex than arrays, maps, strings and ints, and any other "data structures" the candidate ought to know, given their background. For instance, if they claim to be a crypto expert, they should know how to use a black box that gives them private and public keys. By straightforward I mean, you don't want the candidate to have to have an "aha" moment. If you're asking for them to write a max-flow variant, you'd better be explicit that you don't mind a really brute-force solution, or most of the interview will be spent trying to remember complex algorithms. Is the algorithm one that you can understand by walking through test cases/figuring out what steps you took in your mind to go from input to output? If it takes more than that, you're going to get a few false positives, i.e. "Wow she was great! No one's answered this so fast!" (because she just spent 20 hours doing this on a homework), and many false negatives.

jtheory 12y ago

It's gets a bit complicated when your job candidates may find your test interesting enough that they post a complete walk-through of the solution online. The candidate was kind enough to avoid posting the name of the company (to hopefully hide the walk-through from Google queries for "crack me hiring test company name", but an HN discussion might then arise around the walk-through, in which the company would be named, and well, there goes your test. It's hard to accurately run "take home" tests when hiring because of just this.

Fuxy 12y ago
If you're not changing your tests on every interview batch you're doing it wrong. You're asking a potential candidate to spend hours cracking your program but you can't be bothered to change it for each batch? I expect any company applying this interview technique to at least put some effort into it and even then it pays off since a batch could contain hundreds of candidates and you only have to build the program once.
- aaronharnly 12y ago
  What do you mean by "batch"? Most companies have rolling hiring.
  - warrenm 12y ago
    this is true - but you're [generally] going to have a batch of candidates for any given position
- chrisbennet 12y ago
  This ^ "You only get out of it, what you put into it." goes both ways - employers should expect to extend some effort if they expect their candidates to extend some effort. I strongly suspected that sometimes goals of the person responsible for hiring are not quite aligned with the company they work for. Example: Company's goal: Hire an exceptional candidate. Lazy or overworked hiring manager's goal: Reduce the size of the stack of 1000's of resume's I have to go through using any means possible that will make it appear like he/she actually did the work to find the candidate. "Lemme see, use keyword search to ditch the ones without 'Github' on their resume'. Now ditch the ones who don't have a college degree. Still some resume's left? OK, how about tossing the ones without a CS degree?" Run a filter with enough conditions and you can get that pile down to nothing pretty fast. If that is still not enough filtering, ask the candidates to do something very time consuming. This a great way to weed out people who are very busy or exceptional at what they do and don't need to jump through your hoops.

eyko 12y ago

I've been in two interviews where they asked me to code without googling or looking at the reference. Personally, I think it's as absurd as writing shell scripts on a computer with no `man`, or without being able to refresh your memory with `<command> --help`. Good luck with that! I was in an interview where they asked me to configure specific firewall rules on a notepad with no access to a terminal, so I couldn't check man iptables for stuff like how to match certain sources/destinations hosts/IPs. And I was not too familiar with the opts (was it --dest, --dst, or -d? and how do calculate the network mask for this range of IPs?). It was also for a junior position so I thought they were taking a piss at me and simply told them so.