7 ms·

In the end, it's a crap shoot... Even their highest marker of validity is only about 50%. In the end, if you make it too hard in an employee market, you will h

In the end, it's a crap shoot... Even their highest marker of validity is only about 50%. In the end, if you make it too hard in an employee market, you will have good to great candidates drop off... if you make it too easy, you will wind up at the bottom of the barrel. Market pay will also vary wildly and dramatically, and perception is equally varied. Where I work now, we have a small pre-interview code challenge... it's not particularly difficult and in general should be resolvable in under an hour. I've seen questions on the other side, that I've literally spent 3+ hours on something that said an hour. I suggest those coming up with these things actually implement their challenge on a clean computer (only tools, no reference code) and time themselves. I now cut myself off at an hour and submit what I have stating as much. As to structured interviews, it will depend. It bugs me both ways when I see more generic, low level pattern questions in interviews only because the practical implications are usually very limited. On the interviewer side, it does help to ask some of the more quiz-like questions as it tends to give you a decent baseline of what the person being interviewed understands. In the end, personal questions along the lines of what a person works on (developers) outside of educational or work settings gives a ton of information on how they work as well as what they know. Culture, for as much of a B.S. thing that it is, is actually very important as well. A functional team or set of teams needs a few different types of personalities and worker types in order to do things well as a team. Being able to get along together is also imperitive. In the end.. it's a crap-shoot and understanding the needs, fill and fit are all varied a bit... trying to normalize doesn't help much. Also, business needs and talent pools will vary greatly.

MaxBarraclough 7y ago
> if you make it too hard in an employee market, you will have good to great candidates drop off... if you make it too easy, you will wind up at the bottom of the barrel And where you set the bar will depend on, roughly speaking, your prestige. Google can afford to be extremely selective when hiring technologists. Similar thing goes for Harvard and the Air Force. > what a person works on (developers) outside of educational or work settings gives a ton of information on how they work as well as what they know The usual counterpoint: nuclear engineers and surgeons tend to have hobbies with no bearing on nuclear engineering or surgery. Nothing wrong with a competent technology professional keeping it to work hours.
- tschiller 7y ago
  All teams should be selective. But if you don't have a Google-sized brand and budget, you need to develop a clearer understanding of your teams strengths/weaknesses and needs. This understanding will enable you to figure out which hiring criteria do you need to maximize vs. which do you need to satisfy and where the bar is. Also, by interviewing more effectively, you punch above your weight by spending more face-to-face time selling your company/team/role.
  - geoffroberts 7y ago
    I love this point—the more time you can spend selling the candidate the better. And I think the more effective your interview process, candidates take notice and probably need to be "sold" that much less.
tschiller 7y ago
Great points - there are things we're trying to address over at Interview GPS. We're building a Wikipedia/Stack Overflow-style knowledge base of the best interviewer guides/rubrics: come join the cause at https://www.interviewgps.com/community/ https://www.interviewgps.com/community/ We tag questions in 3 areas: Personal Values: instead of "culture" we help you assess shared values. For example: do you value fun vs. achievement? Competencies: does your software developer need attention to detail and critical thinking? Or creativity and collaboration? Skills: what skills are necessary vs. what can be learned on the job? E.g., should you assess for React skills? Or assess for programming/debugging skills in general? Also, P.S. you have to be careful/thoughtful when asking about what developers work on outside of work. Worst case, you end up breaking the law by discriminating against protected groups. I'd suggest only looking to that if their work history is relatively lacking and you're trying to develop a positive case for the candidate
- geoffroberts 7y ago
  Love your site—thanks for sharing! And +1 with regards to the potential legal issues.
- tracker1 7y ago
  By what they do outside work/school... I specifically mean if they work on software projects, using skills related to the skills they use at work. I've seen both sides of this and tend not to discriminate when someone has more experience vs. less... it's an offset, more important for interviewing someone out of a bootcamp, or with no formal education. It really takes all types, so to speak. At this point, I've interviewed and met a pretty broad spectrum. The same goes for on the interviewee side as well. Again... it's a balance... work history, personal projects (github activity), the pre-interview code challenge, as well as the technical interview are a matter of finding what's needed... someone can be less in one area and make up for it in another. To me at least, independent drive makes up for experience in a lot of ways, especially for more junior-mid roles.
derefr 7y ago
> In the end, if you make it too hard in an employee market, you will have good to great candidates drop off... if you make it too easy, you will wind up at the bottom of the barrel. Market pay will also vary wildly and dramatically, and perception is equally varied. I mean, you're talking about a different thing than "job performance" here. Job performance is a satisficing measure: you only need to get someone who's good enough to perform the duties you need of them. (Yes, even in software engineering.) You're talking about ranking measures. Does it matter if someone is "the best" if they're still not good enough to do your job? Does it matter if someone is "the bottom of the barrel" if they are good enough to do your job? I would say no, in both cases. IMHO, the hiring bar is way too high in most bigcorps; they care about vanity metrics about having "the best", when 100% of the work assigned to these hires could be easily done by people who are not, in fact, "the best." The point of a job interview, in the abstract, is to filter out the candidates who aren't qualified to perform the job duties even after a few weeks of on-the-job training. This leaves you with a pool of people who are all qualified. At that point, you should either just hire one of them at random, or try to optimize for a second criterion, such as "is willing to take the lowest compensation to do the job" (which usually translates to "has the fewest formal qualifications to do the job, while still actually being qualified in practice.")
- wpietri 7y ago
  > Job performance is a satisficing measure: you only need to get someone who's good enough to perform the duties you need of them. (Yes, even in software engineering.) Is that really the case for software? I mean, I'm very skeptical of the "only the best and brightest" nonsense, as I think it selects for the wrong people. But I'm also skeptical of the "eh, good enough" heuristic. Right now I'm helping a friend deal with a code base that was built by an outside dev shop. It undeniably works; the business has been running on it for a number of years. And honestly, like most software, it's not a complicated app; mostly CRUD stuff. But overall it's a spaghetti snarl. Any given concept might be in the code, or in a config file, or driven by SQL data, or even driven by JSON blobs stuffed into the database, mutated by the main code base, and passed along to the JS front end. Now that I can see the code, I understand why the dev shop needed so many people working on it, and why all features were hard to do. It's because when you use a lot of mediocre programmers, they all create work for themselves and one another. Not intentionally, but just because they aren't able to see how the differing approaches and sloppy work slowly diminish productivity. So although I agree with you that a lot of companies hire on vanity metrics, I'm really skeptical that "whoever, as long as they barely qualify" is a better approach.
geoffroberts 7y ago
I completely agree that if you're designing an sort of assessment process, you should go through it yourself or ask other engineers to go through it so it's reasonably scoped in terms of expected time commitment. You can also use assessment tools that specifically limit how much time an engineer can invest in working on the challenge, which is also useful in showing how much progress each candidate made in a set amount of time. The problem with this approach can often be that the candidate feels like they're working against the clock though, which isn't "real world."
- tracker1 7y ago
  That's fair... a few days ago, got a solution (the problem is generally a cli program to transform a csv) that included a full API, data access layer and React ui. It was frankly impressive, and total overkill... turns out the request was unclear from management on our end. I'm still surprised that almost nobody has used an actual library for csv parsing (naive solutions reading a line and splitting mostly), which to me results in so many headaches in practice.
pmiller2 7y ago
> In the end, it's a crap shoot... Even their highest marker of validity is only about 50%. Here's the thing though: all these different tests aren't completely independent. You can combine a couple of them and get a criterion validity value of up to .65. See https://books.google.com/books?id=IvKYhrXAfRUC&pg=PA198&lpg=PA198&dq=structured+interview+plus+work+sample+test&source=bl&ots=7oLClho8-K&sig=ACfU3U2Az2oDif8VhFMJUNo1l7sKnXaNog&hl=en&sa=X&ved=2ahUKEwjYm9OCmLzoAhVCnp4KHfuBCDsQ6AEwD3oECAsQAQ#v=onepage&q=structured%20interview%20plus%20work%20sample%20test&f=false https://books.google.com/books?id=IvKYhrXAfRUC&pg=PA198&lpg=...

MaxBarraclough 7y ago

> if you make it too hard in an employee market, you will have good to great candidates drop off... if you make it too easy, you will wind up at the bottom of the barrel And where you set the bar will depend on, roughly speaking, your prestige. Google can afford to be extremely selective when hiring technologists. Similar thing goes for Harvard and the Air Force. > what a person works on (developers) outside of educational or work settings gives a ton of information on how they work as well as what they know The usual counterpoint: nuclear engineers and surgeons tend to have hobbies with no bearing on nuclear engineering or surgery. Nothing wrong with a competent technology professional keeping it to work hours.

tschiller 7y ago
All teams should be selective. But if you don't have a Google-sized brand and budget, you need to develop a clearer understanding of your teams strengths/weaknesses and needs. This understanding will enable you to figure out which hiring criteria do you need to maximize vs. which do you need to satisfy and where the bar is. Also, by interviewing more effectively, you punch above your weight by spending more face-to-face time selling your company/team/role.
- geoffroberts 7y ago
  I love this point—the more time you can spend selling the candidate the better. And I think the more effective your interview process, candidates take notice and probably need to be "sold" that much less.

tschiller 7y ago

Great points - there are things we're trying to address over at Interview GPS. We're building a Wikipedia/Stack Overflow-style knowledge base of the best interviewer guides/rubrics: come join the cause at https://www.interviewgps.com/community/ https://www.interviewgps.com/community/ We tag questions in 3 areas: Personal Values: instead of "culture" we help you assess shared values. For example: do you value fun vs. achievement? Competencies: does your software developer need attention to detail and critical thinking? Or creativity and collaboration? Skills: what skills are necessary vs. what can be learned on the job? E.g., should you assess for React skills? Or assess for programming/debugging skills in general? Also, P.S. you have to be careful/thoughtful when asking about what developers work on outside of work. Worst case, you end up breaking the law by discriminating against protected groups. I'd suggest only looking to that if their work history is relatively lacking and you're trying to develop a positive case for the candidate

geoffroberts 7y ago
Love your site—thanks for sharing! And +1 with regards to the potential legal issues.
tracker1 7y ago
By what they do outside work/school... I specifically mean if they work on software projects, using skills related to the skills they use at work. I've seen both sides of this and tend not to discriminate when someone has more experience vs. less... it's an offset, more important for interviewing someone out of a bootcamp, or with no formal education. It really takes all types, so to speak. At this point, I've interviewed and met a pretty broad spectrum. The same goes for on the interviewee side as well. Again... it's a balance... work history, personal projects (github activity), the pre-interview code challenge, as well as the technical interview are a matter of finding what's needed... someone can be less in one area and make up for it in another. To me at least, independent drive makes up for experience in a lot of ways, especially for more junior-mid roles.

derefr 7y ago

> In the end, if you make it too hard in an employee market, you will have good to great candidates drop off... if you make it too easy, you will wind up at the bottom of the barrel. Market pay will also vary wildly and dramatically, and perception is equally varied. I mean, you're talking about a different thing than "job performance" here. Job performance is a satisficing measure: you only need to get someone who's good enough to perform the duties you need of them. (Yes, even in software engineering.) You're talking about ranking measures. Does it matter if someone is "the best" if they're still not good enough to do your job? Does it matter if someone is "the bottom of the barrel" if they are good enough to do your job? I would say no, in both cases. IMHO, the hiring bar is way too high in most bigcorps; they care about vanity metrics about having "the best", when 100% of the work assigned to these hires could be easily done by people who are not, in fact, "the best." The point of a job interview, in the abstract, is to filter out the candidates who aren't qualified to perform the job duties even after a few weeks of on-the-job training. This leaves you with a pool of people who are all qualified. At that point, you should either just hire one of them at random, or try to optimize for a second criterion, such as "is willing to take the lowest compensation to do the job" (which usually translates to "has the fewest formal qualifications to do the job, while still actually being qualified in practice.")

wpietri 7y ago
> Job performance is a satisficing measure: you only need to get someone who's good enough to perform the duties you need of them. (Yes, even in software engineering.) Is that really the case for software? I mean, I'm very skeptical of the "only the best and brightest" nonsense, as I think it selects for the wrong people. But I'm also skeptical of the "eh, good enough" heuristic. Right now I'm helping a friend deal with a code base that was built by an outside dev shop. It undeniably works; the business has been running on it for a number of years. And honestly, like most software, it's not a complicated app; mostly CRUD stuff. But overall it's a spaghetti snarl. Any given concept might be in the code, or in a config file, or driven by SQL data, or even driven by JSON blobs stuffed into the database, mutated by the main code base, and passed along to the JS front end. Now that I can see the code, I understand why the dev shop needed so many people working on it, and why all features were hard to do. It's because when you use a lot of mediocre programmers, they all create work for themselves and one another. Not intentionally, but just because they aren't able to see how the differing approaches and sloppy work slowly diminish productivity. So although I agree with you that a lot of companies hire on vanity metrics, I'm really skeptical that "whoever, as long as they barely qualify" is a better approach.

geoffroberts 7y ago

I completely agree that if you're designing an sort of assessment process, you should go through it yourself or ask other engineers to go through it so it's reasonably scoped in terms of expected time commitment. You can also use assessment tools that specifically limit how much time an engineer can invest in working on the challenge, which is also useful in showing how much progress each candidate made in a set amount of time. The problem with this approach can often be that the candidate feels like they're working against the clock though, which isn't "real world."

tracker1 7y ago
That's fair... a few days ago, got a solution (the problem is generally a cli program to transform a csv) that included a full API, data access layer and React ui. It was frankly impressive, and total overkill... turns out the request was unclear from management on our end. I'm still surprised that almost nobody has used an actual library for csv parsing (naive solutions reading a line and splitting mostly), which to me results in so many headaches in practice.

pmiller2 7y ago

> In the end, it's a crap shoot... Even their highest marker of validity is only about 50%. Here's the thing though: all these different tests aren't completely independent. You can combine a couple of them and get a criterion validity value of up to .65. See https://books.google.com/books?id=IvKYhrXAfRUC&pg=PA198&lpg=PA198&dq=structured+interview+plus+work+sample+test&source=bl&ots=7oLClho8-K&sig=ACfU3U2Az2oDif8VhFMJUNo1l7sKnXaNog&hl=en&sa=X&ved=2ahUKEwjYm9OCmLzoAhVCnp4KHfuBCDsQ6AEwD3oECAsQAQ#v=onepage&q=structured%20interview%20plus%20work%20sample%20test&f=false https://books.google.com/books?id=IvKYhrXAfRUC&pg=PA198&lpg=...