6 ms·

I recently ran an interview process for a relatively senior eng role at a tiny startup. Because I believe different interview methods work better for different

I recently ran an interview process for a relatively senior eng role at a tiny startup. Because I believe different interview methods work better for different people, I offered everyone a choice: 1. Do a takehome test, targeted to take about 4 hours but with no actual time limit. This was a non-algorithmic project that was just a stripped-down version of what I'd spent the last month on in actual work. 2. Do an onsite pairing exercise in 2 hours. This would be a version of #1, but more of "see how far we get in 2 hours." 3. Submit a code sample of pre-existing work. Based on the ire I've seen takehome tests get, I figured we'd get a good spread between all three, but amazingly, ~90-95% of candidates chose the takehome test. That matches my preference as a candidate as well. I don't know if this generalizes beyond this company/role, but it was an interesting datapoint - I was very surprised to find that most people preferred it!

bhhaskin 2y ago
Why would you even do any of that for a senior role? I wouldn't waste my time with it, and it shows you don't know how to interview/evaluate for a senior position.
- 0xffff2 2y ago
  I've come around to the idea that all people who will be expected to write code as part of their job need to be evaluated for their ability to write code as part of the interview process. I've seen too many people write convincing and impressive resumes without a single ounce of technical ability to their name.
  - bhhaskin 2y ago
    You're hiring for a senior role, not a junior or mid. Your candidates aren't fresh out of school or just starting their careers. They already have plenty of work experience. They already know how to code and have been doing it for years. A conversation will tell you much more about their approaches to problems solving, team work, and mentoring than a coding challenge will. Looking at their GitHub, their work experience and references will tell you the rest. It's like asking a doctor to go over basic human anatomy.
    - 0xffff2 2y ago
      >They already know how to code and have been doing it for years. I'm telling you from my own experience that this is observably not true. Twice now I've been burned by internal hires who objectively had years of experience (easy to verify when they have been working within my division) and yet couldn't code their way out of a paper bag. And tangentially, with some of the health care experiences I've had I very well might try to administer a basic anatomy quiz to potential doctors if I thought I could get away with it.
    - WillDaSilva 2y ago
      Have you ever tried to hire developers? I have, many times. It's shocking how incapable most people with a history of working as senior developers for 10+ years are at basic programming questions. I'm talking about stuff at the level of Fizz Buzz, or barely more difficult, being solved in a language of their choice with ample time. Through all of the technical interviews I've run on behalf of multiple companies, I've become convinced that our industry is full of imposters. A terrifying number of people who are employed as developers cannot code, beyond making tiny edits to existing code, or slinging StackOverflow answers and AI generated content around. They cannot think for themselves. They rely heavily on their coworkers to make any substantial progress. If anything, modern AIs have made these people more difficult to spot. I've found these imposters within the companies I've worked for too, and if it's not possible to move them to a non-technical role where they can be productive, then it's best to fire them. If that isn't possible, then keep them out of the way of any critical work, but know that they'll continue to be a drag on their team, and will decrease the moral of those on their team who have to keep covering for them. Once I even worked on a team (as a junior) where only 3 members out of ~16 even knew the language that was exclusively used for the job. The remaining team members accomplished nothing every day. Those stand-up meetings were painful. Drawn out to 45+ minutes of standing in a circle while the unproductive team members found new ways to explain why they haven't made progress. Then I'd see them return to their desks, and not even open their IDEs for weeks at a time. They seemed to keep their jobs because their boss didn't want to fire them, and it cost him nothing personally to keep them around, offering them a sort of corporate welfare. Perhaps having such a large team even helped him politically, but it's unclear if he cared about that since he also spent almost all of his time slacking off, playing games in the break room, while the 3 productive team members carried the rest of the team.
- dahart 2y ago
  Why wouldn’t you? How should one evaluate a senior position, in your opinion? Are you suggesting a senior dev shouldn’t need to demonstrate any coding skill? (If not, why not?) Or are you suggesting that there are other faster ways to show coding skills?
  - bhhaskin 2y ago
    I responded to a different comment but it's also relevant here. You're hiring for a senior role, not a junior or mid. Your candidates aren't fresh out of school or just starting their careers. They already have plenty of work experience. They already know how to code and have been doing it for years. A conversation will tell you much more about their approaches to problems solving, team work, and mentoring than a coding challenge will. Looking at their GitHub, their work experience and references will tell you the rest. It's like asking a doctor to go over basic human anatomy.
    - dahart 2y ago
      You’re underestimating the variance in senior devs, and you’re failing to look at this from the interviewer’s perspective. Sometimes doctors should be quizzed on anatomy, btw. They need to be able to compare candidates, and senior devs will be expected to code. Their goal is to pick the best candidate. Asking them to demonstrate their coding skill isn’t just fair game, it’s what half the interviewees actually want. A lot of people prefer being evaluated on code than on soft skills (just read the threads here for tons of evidence.) It’s useful to know whether a candidate is going to be on the principal engineer path or management path. It’s useful to know whether they are actually good at coding, and how good exactly. And it’s also useful to know if they are willing to do what needs to be done once hired. Someone interviewing for a senior coding position who refuses to code during the interview is about as big of a red flag as you can have, and I’ve been part of hiring teams that will politely excuse someone for that, and I agree with the reasoning. This might not make sense until you’re on the hiring side of things, but having a resume does not entitle one to being hired for a higher title and more money.
    - theamk 2y ago
      You'd think so, but during the interviews I did, there are plenty of candidates to senior positions who just cannot code. They talk the great talk about problem solving and team work and mentoring, but once it's time walk the walk, they just can't seem to write any code - and we are not talking advanced algorithms, we are talking Fizz-Buzz level. Maybe it's different for doctors, but as long as such people exist and apply to jobs, we need to ask programming questions. This also means that if _you_ are applying to a job in a larger team for a senior position and they did not ask you for any coding questions, they either: - Really good for firing people fast for low performance - Have people at "senior" position which just give advice and nasty code reviews, but don't actually write any code. Either way, it's a red flag for the company.
- stickfigure 2y ago
  What would you waste your time with? I exclusively give pairing interviews, usually just 1 hr. The variance between good and bad candidates is amazing, and you wouldn't guess at all from resumes or casual conversations. Most candidates have given me very positive feedback. Why wouldn't you want to be interviewed like this?
  - bhhaskin 2y ago
    I'm a senior developer. Of course I know how to code. Do you know how many code challenges I've done over the course of my career? 99% of them where terrible at evaluating my ability to code. They are a waste of time at the senior level. You are much better off with a conversation, looking at past work and experience and references than you are doing code challenges.
    - stickfigure 2y ago
      After interviewing ~50 people using this technique - I'm sorry but there's no "of course". I cannot tell by looking at your past work experience or talking to you whether you know how to code. I know this, because I used to look at resumes and have long conversations before I ran my pairing interview. And I found the results almost uncorrelated. Now I save the fun conversations for people who make it through the screening. But by then I already know if I want to hire them. My pairing interview (it's a pairing session, not a code challenge) covers aspects of engineering that I specifically care about. It's not terribly hard. But at the end I'll have a pretty good idea if I want to turn this person loose in my codebase.
      - kstrauser 2y ago
        I second all of that heartily. Of course the senior eng with 10 years hands-on experience at a household name tech company knows how to code... except that they don't. At all, apparently. That seems so ludicrously unlikely unless you've actually interviewed a lot of people, then it's just the sad fact. To others reading along, I'm not talking about someone not being able to write a multitasking OS on their own. I mean, I'm not sure how these people possibly graduated college not knowing how to write a trivial program in the language of their choosing. Turns out you can get surprisingly far into a software engineering degree without every writing code. I wouldn't have believed it. Evidence proves it though.
- alkonaut 2y ago
  Because for a senior role you need to hire a person with senior role skills, I suppose. Just N years of experience at X company doesn't provide that. I'd rather look at someone's github repo than talk to their reference tbh.
  - John_Cena 2y ago
    I've never worked at a company with a public repo. I never understood this. I have a life outside of work
    - alkonaut 2y ago
      Neither did I. But I'd rather just "smuggle" out some past work and "rewrite it" to make it inconspicuous, if I didn't have my own personal work to show. The company I work for isn't well known so saying "I worked there in a senior role for 10 years" isn't going to get me the next job. If I didn't have anything to show for that I either made for fun or could display from a past job, I'd not apply for another until I had tbh. People say "No I don't have any past hobby work, never contributed to OSS, and all my professional work is verboten to show" that's not unusual, but it's also not an encouraging sign to me when interviewing.
      - bhhaskin 2y ago
        Wouldn't that be a red flag if they don't have any public code at all as a senior developer? They aren't fresh out of school or just starting their careers. They should have something.
        John_Cena 2y ago
        I'm a senior developer and my github is half guitar tabs. I'm not interested in peacocking. Maybe it's because Hackaday refused to put my name on the article with my senior design project years ago and I just don't want to play the game.
        2y ago
- 12_throw_away 2y ago
  > Why would you even do any of that for a senior role? I used to think like that ... then we hired a batch of 3 "senior" devs who could not do simple, everyday coding tasks, even in their ostensible languages of preference. All had come with exceptional resumés and personal recommendations. So now, no one at any level one gets hired without demonstrating that they can at least write a fizzbuzz and commit it to a repository. Now, I doubt that senior engineers in other fields have to do this sort of thing. But, most other engineering fields have licensing/accreditation with accompanying post-graduate academic curricula. We don't (but probably should).
- fishtoaster 2y ago
  Because all evidence[0] I've found has shown that work-sample tests and repeatable, structured indicators are the best indicator of job performance. I understand that a lot of devs think they can get a feel for a candidate by just having a good ol' conversation with them, but the body of study on the subject says that that's just wrong. And so I have people do a task that's as close to the actual job as humanly possible and evaluate them on that. https://www.researchgate.net/publication/283803351_The_Validity_and_Utility_of_Selection_Methods_in_Personnel_Psychology_Practical_and_Theoretical_Implications_of_85_Years_of_Research_Findings https://www.researchgate.net/publication/283803351_The_Valid...
milesvp 2y ago
Interesting. I think I slightly prefer take homes, mostly because they're not as hectic. The problem ends up being, that I don't have any guarantee that the other side will spend any time on it. At least with an in person, I know that the company had to incur a significant cost so I know my time is less likely to be wasted. There's already a growing asymmetry with job searches, and this is one more trend that accelerates that. I still have a non technical friend who's annoyed with me for not completing a take home years ago, because, while the main ask was obviously trivial, there was enough slop in the directions that I could kill hours trying to decide if something was good enough to submit. On top of that it was expecting some fairly narrow set of techs, that, while not difficult to pick up, were not something I was interested in trying to learn enough about for a chance to maybe help some company that "couldn't find any technical talent".
- fishtoaster 2y ago
  > The problem ends up being, that I don't have any guarantee that the other side will spend any time on it. This is a key thing for me. I consider it a moral obligation to give material feedback to anyone who does a takehome test for me - to invest at least some serious time evaluating it. Likewise, I would never give a takehome test until the candidate has at least had a phone screen with someone at the company and there's some level of investment on both sides. On the other hand, I know a few junior devs right now who are submitting resumes and getting sent takehome tests off-the-bat. And, of course, after spending hours on those challenges, they get only form-letter rejections. I understand why companies do that - there's a glut of junior devs right now and any early-career role gets flooded with resumes that you need to somehow pare down - but I still consider it unconscionable. I understand why you might not want to risk dealing with that.
dahart 2y ago
Interesting! I like the idea of choice, but as a hiring manager it makes my problem harder. How do you compare the results from different choices equitably? I find trying to compare candidates fairly to be quite difficult, even when they have the exact same interview. Last time I did a coding interview for real, I had the choice of any programming language, and could choose between 3 different problems to solve. I liked that quite a bit, and was offered the job. Being able to choose Python, instead of, say, C++ in a time-bound interview almost feels like cheating.
- em-bee 2y ago
  in one interview for a python job i was allowed to submit solutions to tests in pike which i had more experience with, but none of the devs at the company had ever seen. for python they had an automated testsuite but i remember my interviewer telling me that the devs were pouring over my test results to verify them manually while we continued other parts of the interview. i got the job.
- sfink 2y ago
  > as a hiring manager it makes my problem harder. How do you compare the results from different choices equitably? That makes sense, and it's the perspective that's being drilled into a lot of us. Implicit bias and all that. But in my experience, the comparison problem has never been that big of an issue in practice. I guess it depends on the hiring climate, but I'm much more familiar with spending a lot of time going through mediocre candidates and looking for someone who's "good enough". And this is when the recruiter is doing their job, and I understand why each candidate made it to the interview stage. Sure, sometimes it's a hot position (or rather, hot group to work for) and we get multiple good candidates, but then the decisionmaking process is more about "do we want A, who has deep and solid experience in this exact area; or B, who blew us away with their breadth of knowledge, flexibility, and creativity?" than something like "do we want A who did great on the take-home test but was unimpressive in person, or B whose solution to the take-home test was so-so but was clearly very knowledgeable and experienced in person?" The latter is in a hypothetical case where everyone did the same stuff, just to make it easier to compare, but even in that setup that's an uncommon and uninteresting choice. You're comparing test results and trying to infer Truth from it. Test results don't give a lot of signal or predictive power in the first place, so if you're trying to be objective by relying only on normalized scores, then you're not working with much of value. Take home tests or whiteboard tests or whatever are ok to use as a filter, but people aren't points along a one-dimensional "quality" line. Use whatever you have to in order to find people that have a decent probability of being good enough, then stop thinking about how they might fail and start thinking about what it would look like if they succeed. They'll have different strengths and advantages. Standardizing your tests isn't going to help you explore those.
commandlinefan 2y ago
> ~90-95% of candidates chose the takehome test I was expecting "3. submit a code sample" to be the overwhelming winner here - did anybody choose that? Seems like a no-brainer, since it's already done...
- SCUSKU 2y ago
  The code I've written in personal projects have been pretty messy -- whereas code I've written professionally tend to be better (not perfect), but of course I can't share that code. On the employer side, I would prefer the take home assignment over existing code, unless the existing code was super high quality or highly relevant to the company.
- deleted 2y ago
  [deleted]
- nemetroid 2y ago
  99.9% of the code I’ve written is proprietary.
- recursivecaveat 2y ago
  Work samples are a trap. Real problems are always way more hairy than contrived ones, which is not appreciated by people who were just introduced to the problem 30s ago. Since you wrote it while trying to actually accomplish something rather than polish up a perfect little nugget to impress someone, you probably spent way less time per line.
- fishtoaster 2y ago
  I only had one code sample. As others have pointed out, there are a lot of problems: - Many engineers don't write code outside of work and so don't have much code they can show off without breaching their employer's trust - Those that do often don't have recent code. - Those that have recent, personal code to show off have often written it to accomplish some goal and the code isn't necessarily a great example they feel like showing off Really, the only people I've seen have good code samples are those who do extensive open-source work. And for those people, I'm probably already aware of that work when I checked their resume + opened their github.
vunderba 2y ago
Yeah, I saw that coming a mile away. Don't you remember uni? Students always breathed a sigh of relief when the final exam was a take home project. Now add this to the fact that a vast majority of your applicants are going to be feeding your assignment directly into an assistive LLM. As an interviewer, you really can't win. - If you have a take-home test, people will either complain that it's too involved, and time consuming. - If you do whiteboarding, people will claim that it's not representative of the actual job. - If you do paired programming, people will claim it's unfair to those who don't test well under pressure.
- skydhash 2y ago
  That's because these three methodologies test different things. Take-home is the best representative, but it's heavily skewed by the prize. Whiteboarding only test for algorithms knowledge and a bit of problem-solving. Paired programming relies more on communication. And the last two add time pressure that rarely exists in real world scenario. The fact is, it all depends on the person you need and the team they will be slotted in. But it seems that the ones who know the criteria is rarely involved in these matters.
- fishtoaster 2y ago
  That's why I offered the choice: different people do well under different circumstances. I figured that it'd allow me to hire a great dev who hates timed challenges or a great dev that hates take-home busywork. In practice, though, everyone like the take-homes, so ¯\_(ツ)_/¯ As for feeding it into an LLM, I go back and forth on that. With their current capabilities, the project is slightly too complex for that to work. You could get a lot of help from an AI, but you still have to put the pieces together yourself. And I'm fine with that! If AI tooling makes you a faster/better dev on a test, I'd expect it to do the same in real work. Longer-term, though, I worry that LLMs still won't be able to "write the whole thing" for real work, but will be able to "write the whole thing" for takehome tests. And as such, I'll have to figure out how to handle that.
mekoka 2y ago
Did you ask why they chose the take home? I'd favor 1 and 2 over 3, because I feel that you understanding the problem I'm solving gives a better context to appreciate my approach. If I take it home and hand it later, you can compare my solution to yours (or others) and better grasp my coding style, even if it's different to yours, since we worked on essentially the same task. If we're working as a pair, you can observe my thought process. I can better express my reasoning for picking certain idioms. With past code, there's almost no context. It's just code that was written for some other project. In my opinion, this can be very useful in preliminary steps, when I'm sending out my resume and you need to validate that I can indeed write code. In later stages, I think it can be impressive if I have a working app and can walk you down the code for a particular feature. But who has working software laying around? We all have code that works with a few hacks, sorta... I'd favor 1 over 2, because it reduces the probability to blunder. Between spending 2 hours on-site with a higher risk to mess up (due to pressure, Murphy's law, etc) and take the assignment home to work on it for an extra 2 hours, with ample time to freely research whatever I don't understand. It's a no-brainer for me.
kuratkull 2y ago
I'm your average skilled introverted engineer. But after more than a dozen years of experience and problem solving, I'd go with #2. I feel i'd be able to explain myself much more easily, have to do much less work, and probably have much more ways to impress the interviewers with face to face. I have also been on the receiving side of take-home tests and I know how hard it is to impress someone with those.
- ozim 2y ago
  For me problem word I see here is „impress”. When I hire I don’t need someone to impress me, he has to present himself as able to do the job. I think a lot of devs think they have to hire someone that impresses them and we end up with insane tests and adversarial interviews. Yeah of course there will be people who will blow through the task much faster than others with better quality than average and they will impress me and they will get the offer in first place - but often times they already get multiple offers and after negotiation they will pick some higher offer and we have to get someone we can afford to do the job.

bhhaskin 2y ago

Why would you even do any of that for a senior role? I wouldn't waste my time with it, and it shows you don't know how to interview/evaluate for a senior position.

0xffff2 2y ago
I've come around to the idea that all people who will be expected to write code as part of their job need to be evaluated for their ability to write code as part of the interview process. I've seen too many people write convincing and impressive resumes without a single ounce of technical ability to their name.
- bhhaskin 2y ago
  You're hiring for a senior role, not a junior or mid. Your candidates aren't fresh out of school or just starting their careers. They already have plenty of work experience. They already know how to code and have been doing it for years. A conversation will tell you much more about their approaches to problems solving, team work, and mentoring than a coding challenge will. Looking at their GitHub, their work experience and references will tell you the rest. It's like asking a doctor to go over basic human anatomy.
  - 0xffff2 2y ago
    >They already know how to code and have been doing it for years. I'm telling you from my own experience that this is observably not true. Twice now I've been burned by internal hires who objectively had years of experience (easy to verify when they have been working within my division) and yet couldn't code their way out of a paper bag. And tangentially, with some of the health care experiences I've had I very well might try to administer a basic anatomy quiz to potential doctors if I thought I could get away with it.
  - WillDaSilva 2y ago
    Have you ever tried to hire developers? I have, many times. It's shocking how incapable most people with a history of working as senior developers for 10+ years are at basic programming questions. I'm talking about stuff at the level of Fizz Buzz, or barely more difficult, being solved in a language of their choice with ample time. Through all of the technical interviews I've run on behalf of multiple companies, I've become convinced that our industry is full of imposters. A terrifying number of people who are employed as developers cannot code, beyond making tiny edits to existing code, or slinging StackOverflow answers and AI generated content around. They cannot think for themselves. They rely heavily on their coworkers to make any substantial progress. If anything, modern AIs have made these people more difficult to spot. I've found these imposters within the companies I've worked for too, and if it's not possible to move them to a non-technical role where they can be productive, then it's best to fire them. If that isn't possible, then keep them out of the way of any critical work, but know that they'll continue to be a drag on their team, and will decrease the moral of those on their team who have to keep covering for them. Once I even worked on a team (as a junior) where only 3 members out of ~16 even knew the language that was exclusively used for the job. The remaining team members accomplished nothing every day. Those stand-up meetings were painful. Drawn out to 45+ minutes of standing in a circle while the unproductive team members found new ways to explain why they haven't made progress. Then I'd see them return to their desks, and not even open their IDEs for weeks at a time. They seemed to keep their jobs because their boss didn't want to fire them, and it cost him nothing personally to keep them around, offering them a sort of corporate welfare. Perhaps having such a large team even helped him politically, but it's unclear if he cared about that since he also spent almost all of his time slacking off, playing games in the break room, while the 3 productive team members carried the rest of the team.
dahart 2y ago
Why wouldn’t you? How should one evaluate a senior position, in your opinion? Are you suggesting a senior dev shouldn’t need to demonstrate any coding skill? (If not, why not?) Or are you suggesting that there are other faster ways to show coding skills?
- bhhaskin 2y ago
  I responded to a different comment but it's also relevant here. You're hiring for a senior role, not a junior or mid. Your candidates aren't fresh out of school or just starting their careers. They already have plenty of work experience. They already know how to code and have been doing it for years. A conversation will tell you much more about their approaches to problems solving, team work, and mentoring than a coding challenge will. Looking at their GitHub, their work experience and references will tell you the rest. It's like asking a doctor to go over basic human anatomy.
  - dahart 2y ago
    You’re underestimating the variance in senior devs, and you’re failing to look at this from the interviewer’s perspective. Sometimes doctors should be quizzed on anatomy, btw. They need to be able to compare candidates, and senior devs will be expected to code. Their goal is to pick the best candidate. Asking them to demonstrate their coding skill isn’t just fair game, it’s what half the interviewees actually want. A lot of people prefer being evaluated on code than on soft skills (just read the threads here for tons of evidence.) It’s useful to know whether a candidate is going to be on the principal engineer path or management path. It’s useful to know whether they are actually good at coding, and how good exactly. And it’s also useful to know if they are willing to do what needs to be done once hired. Someone interviewing for a senior coding position who refuses to code during the interview is about as big of a red flag as you can have, and I’ve been part of hiring teams that will politely excuse someone for that, and I agree with the reasoning. This might not make sense until you’re on the hiring side of things, but having a resume does not entitle one to being hired for a higher title and more money.
  - theamk 2y ago
    You'd think so, but during the interviews I did, there are plenty of candidates to senior positions who just cannot code. They talk the great talk about problem solving and team work and mentoring, but once it's time walk the walk, they just can't seem to write any code - and we are not talking advanced algorithms, we are talking Fizz-Buzz level. Maybe it's different for doctors, but as long as such people exist and apply to jobs, we need to ask programming questions. This also means that if _you_ are applying to a job in a larger team for a senior position and they did not ask you for any coding questions, they either: - Really good for firing people fast for low performance - Have people at "senior" position which just give advice and nasty code reviews, but don't actually write any code. Either way, it's a red flag for the company.
stickfigure 2y ago
What would you waste your time with? I exclusively give pairing interviews, usually just 1 hr. The variance between good and bad candidates is amazing, and you wouldn't guess at all from resumes or casual conversations. Most candidates have given me very positive feedback. Why wouldn't you want to be interviewed like this?
- bhhaskin 2y ago
  I'm a senior developer. Of course I know how to code. Do you know how many code challenges I've done over the course of my career? 99% of them where terrible at evaluating my ability to code. They are a waste of time at the senior level. You are much better off with a conversation, looking at past work and experience and references than you are doing code challenges.
  - stickfigure 2y ago
    After interviewing ~50 people using this technique - I'm sorry but there's no "of course". I cannot tell by looking at your past work experience or talking to you whether you know how to code. I know this, because I used to look at resumes and have long conversations before I ran my pairing interview. And I found the results almost uncorrelated. Now I save the fun conversations for people who make it through the screening. But by then I already know if I want to hire them. My pairing interview (it's a pairing session, not a code challenge) covers aspects of engineering that I specifically care about. It's not terribly hard. But at the end I'll have a pretty good idea if I want to turn this person loose in my codebase.
    - kstrauser 2y ago
      I second all of that heartily. Of course the senior eng with 10 years hands-on experience at a household name tech company knows how to code... except that they don't. At all, apparently. That seems so ludicrously unlikely unless you've actually interviewed a lot of people, then it's just the sad fact. To others reading along, I'm not talking about someone not being able to write a multitasking OS on their own. I mean, I'm not sure how these people possibly graduated college not knowing how to write a trivial program in the language of their choosing. Turns out you can get surprisingly far into a software engineering degree without every writing code. I wouldn't have believed it. Evidence proves it though.
alkonaut 2y ago
Because for a senior role you need to hire a person with senior role skills, I suppose. Just N years of experience at X company doesn't provide that. I'd rather look at someone's github repo than talk to their reference tbh.
- John_Cena 2y ago
  I've never worked at a company with a public repo. I never understood this. I have a life outside of work
  - alkonaut 2y ago
    Neither did I. But I'd rather just "smuggle" out some past work and "rewrite it" to make it inconspicuous, if I didn't have my own personal work to show. The company I work for isn't well known so saying "I worked there in a senior role for 10 years" isn't going to get me the next job. If I didn't have anything to show for that I either made for fun or could display from a past job, I'd not apply for another until I had tbh. People say "No I don't have any past hobby work, never contributed to OSS, and all my professional work is verboten to show" that's not unusual, but it's also not an encouraging sign to me when interviewing.
    - bhhaskin 2y ago
      Wouldn't that be a red flag if they don't have any public code at all as a senior developer? They aren't fresh out of school or just starting their careers. They should have something.
      - John_Cena 2y ago
        I'm a senior developer and my github is half guitar tabs. I'm not interested in peacocking. Maybe it's because Hackaday refused to put my name on the article with my senior design project years ago and I just don't want to play the game.
        2y ago
12_throw_away 2y ago
> Why would you even do any of that for a senior role? I used to think like that ... then we hired a batch of 3 "senior" devs who could not do simple, everyday coding tasks, even in their ostensible languages of preference. All had come with exceptional resumés and personal recommendations. So now, no one at any level one gets hired without demonstrating that they can at least write a fizzbuzz and commit it to a repository. Now, I doubt that senior engineers in other fields have to do this sort of thing. But, most other engineering fields have licensing/accreditation with accompanying post-graduate academic curricula. We don't (but probably should).
fishtoaster 2y ago
Because all evidence[0] I've found has shown that work-sample tests and repeatable, structured indicators are the best indicator of job performance. I understand that a lot of devs think they can get a feel for a candidate by just having a good ol' conversation with them, but the body of study on the subject says that that's just wrong. And so I have people do a task that's as close to the actual job as humanly possible and evaluate them on that. https://www.researchgate.net/publication/283803351_The_Validity_and_Utility_of_Selection_Methods_in_Personnel_Psychology_Practical_and_Theoretical_Implications_of_85_Years_of_Research_Findings https://www.researchgate.net/publication/283803351_The_Valid...

milesvp 2y ago

Interesting. I think I slightly prefer take homes, mostly because they're not as hectic. The problem ends up being, that I don't have any guarantee that the other side will spend any time on it. At least with an in person, I know that the company had to incur a significant cost so I know my time is less likely to be wasted. There's already a growing asymmetry with job searches, and this is one more trend that accelerates that. I still have a non technical friend who's annoyed with me for not completing a take home years ago, because, while the main ask was obviously trivial, there was enough slop in the directions that I could kill hours trying to decide if something was good enough to submit. On top of that it was expecting some fairly narrow set of techs, that, while not difficult to pick up, were not something I was interested in trying to learn enough about for a chance to maybe help some company that "couldn't find any technical talent".

fishtoaster 2y ago
> The problem ends up being, that I don't have any guarantee that the other side will spend any time on it. This is a key thing for me. I consider it a moral obligation to give material feedback to anyone who does a takehome test for me - to invest at least some serious time evaluating it. Likewise, I would never give a takehome test until the candidate has at least had a phone screen with someone at the company and there's some level of investment on both sides. On the other hand, I know a few junior devs right now who are submitting resumes and getting sent takehome tests off-the-bat. And, of course, after spending hours on those challenges, they get only form-letter rejections. I understand why companies do that - there's a glut of junior devs right now and any early-career role gets flooded with resumes that you need to somehow pare down - but I still consider it unconscionable. I understand why you might not want to risk dealing with that.

dahart 2y ago

Interesting! I like the idea of choice, but as a hiring manager it makes my problem harder. How do you compare the results from different choices equitably? I find trying to compare candidates fairly to be quite difficult, even when they have the exact same interview. Last time I did a coding interview for real, I had the choice of any programming language, and could choose between 3 different problems to solve. I liked that quite a bit, and was offered the job. Being able to choose Python, instead of, say, C++ in a time-bound interview almost feels like cheating.

em-bee 2y ago
in one interview for a python job i was allowed to submit solutions to tests in pike which i had more experience with, but none of the devs at the company had ever seen. for python they had an automated testsuite but i remember my interviewer telling me that the devs were pouring over my test results to verify them manually while we continued other parts of the interview. i got the job.
sfink 2y ago
> as a hiring manager it makes my problem harder. How do you compare the results from different choices equitably? That makes sense, and it's the perspective that's being drilled into a lot of us. Implicit bias and all that. But in my experience, the comparison problem has never been that big of an issue in practice. I guess it depends on the hiring climate, but I'm much more familiar with spending a lot of time going through mediocre candidates and looking for someone who's "good enough". And this is when the recruiter is doing their job, and I understand why each candidate made it to the interview stage. Sure, sometimes it's a hot position (or rather, hot group to work for) and we get multiple good candidates, but then the decisionmaking process is more about "do we want A, who has deep and solid experience in this exact area; or B, who blew us away with their breadth of knowledge, flexibility, and creativity?" than something like "do we want A who did great on the take-home test but was unimpressive in person, or B whose solution to the take-home test was so-so but was clearly very knowledgeable and experienced in person?" The latter is in a hypothetical case where everyone did the same stuff, just to make it easier to compare, but even in that setup that's an uncommon and uninteresting choice. You're comparing test results and trying to infer Truth from it. Test results don't give a lot of signal or predictive power in the first place, so if you're trying to be objective by relying only on normalized scores, then you're not working with much of value. Take home tests or whiteboard tests or whatever are ok to use as a filter, but people aren't points along a one-dimensional "quality" line. Use whatever you have to in order to find people that have a decent probability of being good enough, then stop thinking about how they might fail and start thinking about what it would look like if they succeed. They'll have different strengths and advantages. Standardizing your tests isn't going to help you explore those.

commandlinefan 2y ago

> ~90-95% of candidates chose the takehome test I was expecting "3. submit a code sample" to be the overwhelming winner here - did anybody choose that? Seems like a no-brainer, since it's already done...

SCUSKU 2y ago
The code I've written in personal projects have been pretty messy -- whereas code I've written professionally tend to be better (not perfect), but of course I can't share that code. On the employer side, I would prefer the take home assignment over existing code, unless the existing code was super high quality or highly relevant to the company.
deleted 2y ago
[deleted]
nemetroid 2y ago
99.9% of the code I’ve written is proprietary.
recursivecaveat 2y ago
Work samples are a trap. Real problems are always way more hairy than contrived ones, which is not appreciated by people who were just introduced to the problem 30s ago. Since you wrote it while trying to actually accomplish something rather than polish up a perfect little nugget to impress someone, you probably spent way less time per line.
fishtoaster 2y ago
I only had one code sample. As others have pointed out, there are a lot of problems: - Many engineers don't write code outside of work and so don't have much code they can show off without breaching their employer's trust - Those that do often don't have recent code. - Those that have recent, personal code to show off have often written it to accomplish some goal and the code isn't necessarily a great example they feel like showing off Really, the only people I've seen have good code samples are those who do extensive open-source work. And for those people, I'm probably already aware of that work when I checked their resume + opened their github.

vunderba 2y ago

Yeah, I saw that coming a mile away. Don't you remember uni? Students always breathed a sigh of relief when the final exam was a take home project. Now add this to the fact that a vast majority of your applicants are going to be feeding your assignment directly into an assistive LLM. As an interviewer, you really can't win. - If you have a take-home test, people will either complain that it's too involved, and time consuming. - If you do whiteboarding, people will claim that it's not representative of the actual job. - If you do paired programming, people will claim it's unfair to those who don't test well under pressure.

skydhash 2y ago
That's because these three methodologies test different things. Take-home is the best representative, but it's heavily skewed by the prize. Whiteboarding only test for algorithms knowledge and a bit of problem-solving. Paired programming relies more on communication. And the last two add time pressure that rarely exists in real world scenario. The fact is, it all depends on the person you need and the team they will be slotted in. But it seems that the ones who know the criteria is rarely involved in these matters.
fishtoaster 2y ago
That's why I offered the choice: different people do well under different circumstances. I figured that it'd allow me to hire a great dev who hates timed challenges or a great dev that hates take-home busywork. In practice, though, everyone like the take-homes, so ¯\_(ツ)_/¯ As for feeding it into an LLM, I go back and forth on that. With their current capabilities, the project is slightly too complex for that to work. You could get a lot of help from an AI, but you still have to put the pieces together yourself. And I'm fine with that! If AI tooling makes you a faster/better dev on a test, I'd expect it to do the same in real work. Longer-term, though, I worry that LLMs still won't be able to "write the whole thing" for real work, but will be able to "write the whole thing" for takehome tests. And as such, I'll have to figure out how to handle that.

mekoka 2y ago

Did you ask why they chose the take home? I'd favor 1 and 2 over 3, because I feel that you understanding the problem I'm solving gives a better context to appreciate my approach. If I take it home and hand it later, you can compare my solution to yours (or others) and better grasp my coding style, even if it's different to yours, since we worked on essentially the same task. If we're working as a pair, you can observe my thought process. I can better express my reasoning for picking certain idioms. With past code, there's almost no context. It's just code that was written for some other project. In my opinion, this can be very useful in preliminary steps, when I'm sending out my resume and you need to validate that I can indeed write code. In later stages, I think it can be impressive if I have a working app and can walk you down the code for a particular feature. But who has working software laying around? We all have code that works with a few hacks, sorta... I'd favor 1 over 2, because it reduces the probability to blunder. Between spending 2 hours on-site with a higher risk to mess up (due to pressure, Murphy's law, etc) and take the assignment home to work on it for an extra 2 hours, with ample time to freely research whatever I don't understand. It's a no-brainer for me.

kuratkull 2y ago

I'm your average skilled introverted engineer. But after more than a dozen years of experience and problem solving, I'd go with #2. I feel i'd be able to explain myself much more easily, have to do much less work, and probably have much more ways to impress the interviewers with face to face. I have also been on the receiving side of take-home tests and I know how hard it is to impress someone with those.

ozim 2y ago
For me problem word I see here is „impress”. When I hire I don’t need someone to impress me, he has to present himself as able to do the job. I think a lot of devs think they have to hire someone that impresses them and we end up with insane tests and adversarial interviews. Yeah of course there will be people who will blow through the task much faster than others with better quality than average and they will impress me and they will get the offer in first place - but often times they already get multiple offers and after negotiation they will pick some higher offer and we have to get someone we can afford to do the job.