5 ms·

Coding Interviews Are BS (largely)

35 points by coscreen 5y ago

coscreen 5y ago
Jason Thomas here, CTO at CoScreen. Deep inside I had always suspected coding interviews were in large part bullshit. Originally I based my ambitions on the collective wisdom of the Internet (what could go wrong?), and of course looked towards the FANG interview process, and especially Google. I was the first full-time engineer hired at the video interviewing pioneer HireVue, and have a decade of experience in the HR and interviewing technology space. This experience made me a firm believer in the structured interview. Line up a set of candidates, give them a set of standard questions in a structured fashion and look for genius in the answer of routine questions. What could be fairer and more organized than lining up candidates one by one, question by question, rating each, and just hiring the best rated out of all? Read my post for what I learned and how to set up a truly fair and objective interviewing framework.
- mint2 5y ago
  No offense, but what could be fairer than offering college admissions to people based on lining them up and having them take a standardized test? Honesty, there is no perfect solution. There’s a better solution than the status quo but there is no “fairest solution because there are 68,000 ways to define fair Edit: uggg I fell for the “bait” Anyway, could you point to the year when the collective wisdom of hacker news or even the “internet” considered coding interviews like googles or leet code to be good and not the subject of universal ire? I don’t ever remember that time so I’m confused by the tone of the comment with its points like: >“ Originally I based my ambitions on the collective wisdom of the Internet (what could go wrong?), and of course looked towards the FANG interview process, and especially Google.”
  - thedracle 5y ago
    Jason, CTO of CoScreen here on my normal account. So, this is about coding interviews for hiring developers, and it's more about the goal of hiring a good engineer. Obviously evaluating them dynamically will be more heuristic, and less fair, but it has worked far better for me than using standard uniform async interviews.
m0llusk 5y ago
Possibly the best analysis of this I have yet seen is the video Why our generals were most successful in World War II that in Korea, Vietnam or Iraq/Afghanistan: https://www.youtube.com/watch?v=AxZWxxZ2JGE https://www.youtube.com/watch?v=AxZWxxZ2JGE Essentially, the answer is the hiring methodology. If someone seemed like a best fit for a job then they were hired. They then had two weeks to demonstrate basic functional capacity and six weeks to make significant progress. If at any point things did not work out then they were immediately dismissed and replaced with someone else. The high rate of this process meant that there were always open opportunities even for those who had been dismissed from other positions, so things worked out for both employers and employees.
- paulryanrogers 5y ago
  Great, if not for the healthcare and retirement savings being tied to employment. Having left a role after only 4 months my SO is not happy with so many changes.
  - tonyedgecombe 5y ago
    That's not really the employers problem. I'm surprised more companies don't do this, at least in America where there seems little legislation in the way.
  - HWR_14 5y ago
    Getting rid of healthcare and retirement savings being tied to employment is good for a variety of reasons - this is just one of them. In fact, I'm not sure why anyone would want to keep them tied to employment.
    - doktorhladnjak 5y ago
      Employers like it for retention
      - m0zg 5y ago
        I don't think even the employers like it. It costs a lot and it's a pain in the ass to even sign up for if you're a small company. To add an insult to an injury, costs go up every year like clockwork in a largely out-of-control fashion. I worked for several companies now which started from scratch. "Retention" was never a part of the conversation, but "pain in the ass" always was. I think employers provide health insurance simply because employees expect it, as well as to pay you less than they would have to pay you otherwise. It's tax advantaged on their end. If your accountant is any good, it can even be tax advantaged for single person LLCs. Most people don't know this, though. Why this is not tax deductible for "regular" people I have no idea. IMO it should either be tax deductible for everyone, or no one. To have this in-between situation is idiotic.
VirusNewbie 5y ago
Orthogonal to the app being a cool way to do interviews, am I the only one perplexed by the example this person chose to highlight problems with coding interviews? If I understand correctly, they hired someone who seemed to have a good grasp of algorithms but when it came time to code, took forever to write code and took forever to type something out? Isn't that the exact thing that code interviews are good for? Finding out if someone can write legible, working code quickly? Code interviews are terrible for finding out if someone can do deep thinking on a very difficult problem over a long period of time, but the exact problem scenario this hire had should have been caught. I'm not sure how having them write/debug some slightly more complex code in IDEA vs an online Vim approximation is going to suddenly keep someone similar to the 'candidate' referenced from slipping through. I'd also argue that bootstrapping a project is very different than managing a great code base. I'd be very skeptical of the idea that evaluating someone's toy project tells you how well they'll do in a massive code base.
- thedracle 5y ago
  Jason (article writer) here on my normal ycombinator account. > but the exact problem scenario this hire had should have been caught. I think if people weren't taking courses specifically training for engineering interviews, and common interview challenges, maybe this wouldn't be a problem. The truth is gamified programming challenges (commonly used for interviews) do not have a strong relationship to actual day to day programming skills, like reading code, understanding documentation, and doing research. I suppose I'm just reiterating your point, but I'm not sure why you come to a different conclusion regarding the person I hired being slow reading our code, doing research, and setting up basic tooling being something such an interview process would weed out. It didn't, and hasn't like I said, any more than a coin flip, in my personal (anecdotal of course) experience. > I'm not sure how having them write/debug some slightly more complex code in IDEA vs an online Vim approximation is going to suddenly catch people like this? It won't, if that's all I was suggesting. I was suggesting good engineers may be being filtered due to duress in the interview being caused by poor tooling. The real focus was having them do a small semi-real project of some kind, and then having a solid engineer pair-program with them. This has proven to be a successful strategy for us.
thedracle 5y ago
Someone called this to my attention in terms of making engineering interviews more fair, and there are some overlapping points, particularly with not biasing your interviews based on tooling: https://faircoding.com/interviews/ https://faircoding.com/interviews/
- Richard_Kayala 5y ago
  Website and person look fake AF to me. Let me prove it: "Requiring engineers to dress in any specific way." "I've been writing software for over 20 years. " Old habits die hard. In 1995 you would still be wearing a collar shirt to work. So either he's been amazingly stubborn about the habit or has never worked corporate for a significant amount of time in the old days. "A huge part of software development is seeing how things are done, and then continuing the pattern. " A good senior engineer would have added "and breaking that pattern too." "Success criteria is hidden" Bullshit. Companies standardize their loops and ask their interview group to grade along dimensions and make people justify their decisions to other engineers. "Suppose you have a developer who has written code in 7 different languages spread over 20 years. ( me ) Average length of time coding any specific language then is 3 years. Now suppose you have a developer who has written code in only one language for 5 years. Which developer do you think will impress more at a coding interview? See the problem?" No I really don't. Because if someone says they know Java, they better know the damn in and out of it. I would love to see them talk about how JVM garbage collects as well and other beneficial APIs. You can either know 7 things poorly or 1 thing very well. Any competent engineer would know that persistence and master of 1 thing would mean an insane depth of knowledge: a depth that can't be replicated easily and would make a engineer highly valuable. "The skills required to create effective software that makes the company money are very different from those skills that are used to pass a coding interview. " Again, bullshit. What are you and your 7 languages over 20 years going to do make the company money? How is that Lisp, Haskell, and PHP knowledge working out for you? "Senior engineers who have been writing code for 10+ years may either hate coding interviews or simply be terrible at them because they have focused on delivering business value" No, because there is something called the System Design interview that they would have a better affinity for. In fact, at the Senior Level, its 50/50 (or 30/70 depending where you go) weight between DS&A and SD. This guy is just plain uninformed. "They are a form of random chance" If its truly random, then you should see an equal mix of competent and incompetent engineers hired at all engineering levels. If this was the case, why aren't people being fired en mass all the time? If it was random, then its equally likely that shit engineers would get hired at the same rates as good engineers and you would be forced to believe that Google is full of incompetent morons that magically ship amazing products. I would be stunned to learn that a turd flinging monkey could keep Google Search Engine online. I could keep going but bottom line: this guy is a fucking fake who would suck 2 dicks to get into FAANG in an instant and has no idea what coding interviews really entail.
vbg 5y ago
That’s a very long and wordy advertisement for using coscreen.
- thedracle 5y ago
  Fair, you could use a lot of other platforms to do the same thing. But you can't fault me for plugging CoScreen ;)
Richard_Kayala 5y ago
This is bullshit. Let me explain why. The author posits that the best way to interview is to simulate an actual work environment for the candidate as if he was working in an actual code environment. In theory, this would be good. After all, the best way to understand a candidate is to see how he actually works. The reality is that the Leetcode style coding interview, as we know it today, is best approached the same way you would approach the engineering process in general: prototype, refine the idea, test the idea, revise based on inefficiencies, code it. Any semi-competent moron can do this if he gives a damn about his career because this is actually what people should be doing day to day: thinking about how and why the approach to code would be effective. This is logical thinking that can take years to perfect and adjust, especially when your career progression means tackling increasingly difficult problem areas (from deterministic products to nondeterministic infrastructure issues). Which is why most of the internet thinks the interview is BS: because they think like code monkies, not engineers. Most people believe software engineering is copy-pasting stuff from Github and then moving on when its using software and code to solve a problem. In the author's eyes, he thinks that more weight should be given to the soft skills. Soft skills that can easily be learned on the job. I mean, how long does it take for someone to dive into the code? Just follow the history of changes of that line. A day or 2 of practice. How long does it take for someone to learn new tools like ripgrep? 2 days. These are things that can easily be picked up with less than a year of industrial programming. Which is effectively what onboarding is: getting a person used to the tools, nuances, and implicit culture of a company. And different companies have different needs. When you're a startup, you need to code fast and quick, ship the product, and itterate. In a larger corporate environment, you need to code intelligently, be careful of the leverage you are given, and truly understand what you're writing. In each of the cases, the dimensions you're judged against are going to be different. You artificially restrict your own candidate pool to culture fits instead of people who can think like actual engineers. For big tech companies, it doesn't matter: they have so many people coming through the door that they couldn't care less. If anything, this hurts startups (ironically) because now they have a smaller group of people to work with. Or a larger one if you're dismissing all the people who can actually thinking like a proper engineer. I mean, if a startup wants to hire someone who says "I don't know why the code works, I copied it from stack overflow and moved on with my day" then by all means, have him write the Robinhood options platform. Ceter paribus, I would take the former candidate: someone who's clear in logical thinking, then ram him into reality and the company until he learns. At least he would learn quicker. On the other hand, it would take someone who's the opposite, it would take me at least 5 years for him to even begin to understand to not be a code monkey. Secondly, the Leetcode style interview is actually scalable. Any engineer can give it and can evaluate the problem solving strategies that the candidate uses. That's what allows companies to be efficient with their hiring: people are interchangeable. On the other hand, doing something as nebulous as judging the soft skills of engineering is going to be way too subjective and give more noise-to-signal. Example: if an infrastructure engineer is judging a product engineer, there's going to be a big disconnect considering that the problem space of infra engineers is more nebulous and the solution is more unknown. A product engineer would shit his pants if he was asked how to migrate a codebase at scale. Of course, you could get around this by standardizing the test and pairing engineers with engineers of similar backgrounds. But along what dimensions would you standardize? What if the candidate can solve the issue without tracing the code history? How can you create a test where a candidate has to go through the code history without the answer being blatantly telegraphed (ie. when did this bug occur?)? And how could you do this effectively at scale without screwing over your own company's resources (again, think startups. Only so many engineers to go around.) A more honest way to achieve the same thing (if the author truly wanted to see if the candidate was well suited for the company) without going into Leetcode would be the company to put a probationary period on new-hires. It may cost more but you might as well filter hard and fast instead of using a easy/wrong filter simply because a bunch of newgrad Reddit idiots who can't pass a coding interview are crying their collective heads off about it being unfair. But if a bunch of smart and successful engineers can consistently land offers, then why can't other smart and successful engineers? Even Max Howell, creator of homebrew, who flamed the coding interview for being shit (his tweet about Homebrew and inverting a binary tree), still got an Senior Apple offer. So the fact of the matter is that it works. It might not be efficient, yes. But you cannot deny it works. TL;DR: Author is full of shit about Leetcode interviews and his thesis would actually screw over smaller companies.
throwntoday 5y ago
Coding assessments are largely used to filter unqualified candidates and save time. You can't reasonably interview every applicant, so you set a small barrier in place and those who can cross it get the time. Coding in interviews is more about understanding your logical decisions and brain flow, whether you ask the right questions or approach problems in a reasonable way.
- thedracle 5y ago
  Full disclosure, we still use hackerrank for just this internally. I just think if a candidate scores a 70%, I don't expect them to be better or worse than a candidate who scores 100%. Having criteria to perform an interview is one thing, but if your interview is just to go in with another toy challenge on a whiteboard, I think it would be much better to do a demo project, and immediately pair-program before making the decision to hire.

35 points by coscreen 5y ago

Jason Thomas here, CTO at CoScreen. Deep inside I had always suspected coding interviews were in large part bullshit. Originally I based my ambitions on the collective wisdom of the Internet (what could go wrong?), and of course looked towards the FANG interview process, and especially Google. I was the first full-time engineer hired at the video interviewing pioneer HireVue, and have a decade of experience in the HR and interviewing technology space. This experience made me a firm believer in the structured interview. Line up a set of candidates, give them a set of standard questions in a structured fashion and look for genius in the answer of routine questions. What could be fairer and more organized than lining up candidates one by one, question by question, rating each, and just hiring the best rated out of all? Read my post for what I learned and how to set up a truly fair and objective interviewing framework.

mint2 5y ago
No offense, but what could be fairer than offering college admissions to people based on lining them up and having them take a standardized test? Honesty, there is no perfect solution. There’s a better solution than the status quo but there is no “fairest solution because there are 68,000 ways to define fair Edit: uggg I fell for the “bait” Anyway, could you point to the year when the collective wisdom of hacker news or even the “internet” considered coding interviews like googles or leet code to be good and not the subject of universal ire? I don’t ever remember that time so I’m confused by the tone of the comment with its points like: >“ Originally I based my ambitions on the collective wisdom of the Internet (what could go wrong?), and of course looked towards the FANG interview process, and especially Google.”
- thedracle 5y ago
  Jason, CTO of CoScreen here on my normal account. So, this is about coding interviews for hiring developers, and it's more about the goal of hiring a good engineer. Obviously evaluating them dynamically will be more heuristic, and less fair, but it has worked far better for me than using standard uniform async interviews.

m0llusk 5y ago

Possibly the best analysis of this I have yet seen is the video Why our generals were most successful in World War II that in Korea, Vietnam or Iraq/Afghanistan: https://www.youtube.com/watch?v=AxZWxxZ2JGE https://www.youtube.com/watch?v=AxZWxxZ2JGE Essentially, the answer is the hiring methodology. If someone seemed like a best fit for a job then they were hired. They then had two weeks to demonstrate basic functional capacity and six weeks to make significant progress. If at any point things did not work out then they were immediately dismissed and replaced with someone else. The high rate of this process meant that there were always open opportunities even for those who had been dismissed from other positions, so things worked out for both employers and employees.

paulryanrogers 5y ago
Great, if not for the healthcare and retirement savings being tied to employment. Having left a role after only 4 months my SO is not happy with so many changes.
- tonyedgecombe 5y ago
  That's not really the employers problem. I'm surprised more companies don't do this, at least in America where there seems little legislation in the way.
- HWR_14 5y ago
  Getting rid of healthcare and retirement savings being tied to employment is good for a variety of reasons - this is just one of them. In fact, I'm not sure why anyone would want to keep them tied to employment.
  - doktorhladnjak 5y ago
    Employers like it for retention
    - m0zg 5y ago
      I don't think even the employers like it. It costs a lot and it's a pain in the ass to even sign up for if you're a small company. To add an insult to an injury, costs go up every year like clockwork in a largely out-of-control fashion. I worked for several companies now which started from scratch. "Retention" was never a part of the conversation, but "pain in the ass" always was. I think employers provide health insurance simply because employees expect it, as well as to pay you less than they would have to pay you otherwise. It's tax advantaged on their end. If your accountant is any good, it can even be tax advantaged for single person LLCs. Most people don't know this, though. Why this is not tax deductible for "regular" people I have no idea. IMO it should either be tax deductible for everyone, or no one. To have this in-between situation is idiotic.

VirusNewbie 5y ago

Orthogonal to the app being a cool way to do interviews, am I the only one perplexed by the example this person chose to highlight problems with coding interviews? If I understand correctly, they hired someone who seemed to have a good grasp of algorithms but when it came time to code, took forever to write code and took forever to type something out? Isn't that the exact thing that code interviews are good for? Finding out if someone can write legible, working code quickly? Code interviews are terrible for finding out if someone can do deep thinking on a very difficult problem over a long period of time, but the exact problem scenario this hire had should have been caught. I'm not sure how having them write/debug some slightly more complex code in IDEA vs an online Vim approximation is going to suddenly keep someone similar to the 'candidate' referenced from slipping through. I'd also argue that bootstrapping a project is very different than managing a great code base. I'd be very skeptical of the idea that evaluating someone's toy project tells you how well they'll do in a massive code base.

thedracle 5y ago
Jason (article writer) here on my normal ycombinator account. > but the exact problem scenario this hire had should have been caught. I think if people weren't taking courses specifically training for engineering interviews, and common interview challenges, maybe this wouldn't be a problem. The truth is gamified programming challenges (commonly used for interviews) do not have a strong relationship to actual day to day programming skills, like reading code, understanding documentation, and doing research. I suppose I'm just reiterating your point, but I'm not sure why you come to a different conclusion regarding the person I hired being slow reading our code, doing research, and setting up basic tooling being something such an interview process would weed out. It didn't, and hasn't like I said, any more than a coin flip, in my personal (anecdotal of course) experience. > I'm not sure how having them write/debug some slightly more complex code in IDEA vs an online Vim approximation is going to suddenly catch people like this? It won't, if that's all I was suggesting. I was suggesting good engineers may be being filtered due to duress in the interview being caused by poor tooling. The real focus was having them do a small semi-real project of some kind, and then having a solid engineer pair-program with them. This has proven to be a successful strategy for us.

thedracle 5y ago

Someone called this to my attention in terms of making engineering interviews more fair, and there are some overlapping points, particularly with not biasing your interviews based on tooling: https://faircoding.com/interviews/ https://faircoding.com/interviews/

Richard_Kayala 5y ago
Website and person look fake AF to me. Let me prove it: "Requiring engineers to dress in any specific way." "I've been writing software for over 20 years. " Old habits die hard. In 1995 you would still be wearing a collar shirt to work. So either he's been amazingly stubborn about the habit or has never worked corporate for a significant amount of time in the old days. "A huge part of software development is seeing how things are done, and then continuing the pattern. " A good senior engineer would have added "and breaking that pattern too." "Success criteria is hidden" Bullshit. Companies standardize their loops and ask their interview group to grade along dimensions and make people justify their decisions to other engineers. "Suppose you have a developer who has written code in 7 different languages spread over 20 years. ( me ) Average length of time coding any specific language then is 3 years. Now suppose you have a developer who has written code in only one language for 5 years. Which developer do you think will impress more at a coding interview? See the problem?" No I really don't. Because if someone says they know Java, they better know the damn in and out of it. I would love to see them talk about how JVM garbage collects as well and other beneficial APIs. You can either know 7 things poorly or 1 thing very well. Any competent engineer would know that persistence and master of 1 thing would mean an insane depth of knowledge: a depth that can't be replicated easily and would make a engineer highly valuable. "The skills required to create effective software that makes the company money are very different from those skills that are used to pass a coding interview. " Again, bullshit. What are you and your 7 languages over 20 years going to do make the company money? How is that Lisp, Haskell, and PHP knowledge working out for you? "Senior engineers who have been writing code for 10+ years may either hate coding interviews or simply be terrible at them because they have focused on delivering business value" No, because there is something called the System Design interview that they would have a better affinity for. In fact, at the Senior Level, its 50/50 (or 30/70 depending where you go) weight between DS&A and SD. This guy is just plain uninformed. "They are a form of random chance" If its truly random, then you should see an equal mix of competent and incompetent engineers hired at all engineering levels. If this was the case, why aren't people being fired en mass all the time? If it was random, then its equally likely that shit engineers would get hired at the same rates as good engineers and you would be forced to believe that Google is full of incompetent morons that magically ship amazing products. I would be stunned to learn that a turd flinging monkey could keep Google Search Engine online. I could keep going but bottom line: this guy is a fucking fake who would suck 2 dicks to get into FAANG in an instant and has no idea what coding interviews really entail.

vbg 5y ago

That’s a very long and wordy advertisement for using coscreen.

thedracle 5y ago
Fair, you could use a lot of other platforms to do the same thing. But you can't fault me for plugging CoScreen ;)

Richard_Kayala 5y ago

This is bullshit. Let me explain why. The author posits that the best way to interview is to simulate an actual work environment for the candidate as if he was working in an actual code environment. In theory, this would be good. After all, the best way to understand a candidate is to see how he actually works. The reality is that the Leetcode style coding interview, as we know it today, is best approached the same way you would approach the engineering process in general: prototype, refine the idea, test the idea, revise based on inefficiencies, code it. Any semi-competent moron can do this if he gives a damn about his career because this is actually what people should be doing day to day: thinking about how and why the approach to code would be effective. This is logical thinking that can take years to perfect and adjust, especially when your career progression means tackling increasingly difficult problem areas (from deterministic products to nondeterministic infrastructure issues). Which is why most of the internet thinks the interview is BS: because they think like code monkies, not engineers. Most people believe software engineering is copy-pasting stuff from Github and then moving on when its using software and code to solve a problem. In the author's eyes, he thinks that more weight should be given to the soft skills. Soft skills that can easily be learned on the job. I mean, how long does it take for someone to dive into the code? Just follow the history of changes of that line. A day or 2 of practice. How long does it take for someone to learn new tools like ripgrep? 2 days. These are things that can easily be picked up with less than a year of industrial programming. Which is effectively what onboarding is: getting a person used to the tools, nuances, and implicit culture of a company. And different companies have different needs. When you're a startup, you need to code fast and quick, ship the product, and itterate. In a larger corporate environment, you need to code intelligently, be careful of the leverage you are given, and truly understand what you're writing. In each of the cases, the dimensions you're judged against are going to be different. You artificially restrict your own candidate pool to culture fits instead of people who can think like actual engineers. For big tech companies, it doesn't matter: they have so many people coming through the door that they couldn't care less. If anything, this hurts startups (ironically) because now they have a smaller group of people to work with. Or a larger one if you're dismissing all the people who can actually thinking like a proper engineer. I mean, if a startup wants to hire someone who says "I don't know why the code works, I copied it from stack overflow and moved on with my day" then by all means, have him write the Robinhood options platform. Ceter paribus, I would take the former candidate: someone who's clear in logical thinking, then ram him into reality and the company until he learns. At least he would learn quicker. On the other hand, it would take someone who's the opposite, it would take me at least 5 years for him to even begin to understand to not be a code monkey. Secondly, the Leetcode style interview is actually scalable. Any engineer can give it and can evaluate the problem solving strategies that the candidate uses. That's what allows companies to be efficient with their hiring: people are interchangeable. On the other hand, doing something as nebulous as judging the soft skills of engineering is going to be way too subjective and give more noise-to-signal. Example: if an infrastructure engineer is judging a product engineer, there's going to be a big disconnect considering that the problem space of infra engineers is more nebulous and the solution is more unknown. A product engineer would shit his pants if he was asked how to migrate a codebase at scale. Of course, you could get around this by standardizing the test and pairing engineers with engineers of similar backgrounds. But along what dimensions would you standardize? What if the candidate can solve the issue without tracing the code history? How can you create a test where a candidate has to go through the code history without the answer being blatantly telegraphed (ie. when did this bug occur?)? And how could you do this effectively at scale without screwing over your own company's resources (again, think startups. Only so many engineers to go around.) A more honest way to achieve the same thing (if the author truly wanted to see if the candidate was well suited for the company) without going into Leetcode would be the company to put a probationary period on new-hires. It may cost more but you might as well filter hard and fast instead of using a easy/wrong filter simply because a bunch of newgrad Reddit idiots who can't pass a coding interview are crying their collective heads off about it being unfair. But if a bunch of smart and successful engineers can consistently land offers, then why can't other smart and successful engineers? Even Max Howell, creator of homebrew, who flamed the coding interview for being shit (his tweet about Homebrew and inverting a binary tree), still got an Senior Apple offer. So the fact of the matter is that it works. It might not be efficient, yes. But you cannot deny it works. TL;DR: Author is full of shit about Leetcode interviews and his thesis would actually screw over smaller companies.

throwntoday 5y ago

Coding assessments are largely used to filter unqualified candidates and save time. You can't reasonably interview every applicant, so you set a small barrier in place and those who can cross it get the time. Coding in interviews is more about understanding your logical decisions and brain flow, whether you ask the right questions or approach problems in a reasonable way.

thedracle 5y ago
Full disclosure, we still use hackerrank for just this internally. I just think if a candidate scores a 70%, I don't expect them to be better or worse than a candidate who scores 100%. Having criteria to perform an interview is one thing, but if your interview is just to go in with another toy challenge on a whiteboard, I think it would be much better to do a demo project, and immediately pair-program before making the decision to hire.