4 ms·

I really like the idea of "chopping the onion". I've personally had a handful of devs who passed our interview process with flying colors, but couldn't actually

I really like the idea of "chopping the onion". I've personally had a handful of devs who passed our interview process with flying colors, but couldn't actually deliver anything once they got in the door. They could recite any leetcode problem from memory, but didn't know how to contribute value to their team or company. > The grading is automatic. I think is ok as _one_ metric, but I wouldn't want it to the be the _only_ metric that is being looked at. ICs aren't really ICs. You can be a JIRA champion, knocking tickets into the done column at the speed of light, but Great Engineers do more than just that. Great Engineers build up their team, and help others grow. They know how to push back on unrealistic asks. They can incite help when they get stuck. They can translate complex issues into digestible bites. Writing good code is the bare minimum quality of being a Great Engineer. The main barrier I see to hiring Great Engineers is that you really need to hire Great People. But a lot of hiring practices are taking all the human aspects out of hiring (except for the "behavioral" questions, but even these can be gamed like system design or leetcode). Great People worthy of hiring are driven, passionate (not "spent all my waking time coding", but "truly believe in the mission/goals of the company/project"), empathetic, intelligent, and trust worthy. None of those traits are easily discernible from random logic puzzles. I don't know what the solution is. At some point there has to be a "can actually write code" litmus test, but I don't believe any of the current methods work well to prove that. I think take home tests are closer to the "chopping onions" idea, but only require commitment from the applicant, not the employer, which leads to a lot of wasted time. Personally, I think temp-to-hire or work-for-a-day type models could work, if there was some standards to prevent abuse. I would rather sacrifice a few days to a week to see if I actually like working somewhere, and to prove to them that I can do the job, then look at CTCI ever again.

Fiveplus 6y ago
> handful of devs who passed our interview process with flying colors, but couldn't actually deliver anything once they got in the door. They could recite any leetcode problem from memory, but didn't know how to contribute value to their team or company. Interesting, why do you think was this the case?
- alpha_squared 6y ago
  Not the person you're responding to, but my perspective is that there is often a false equivalency between academic understanding (solving hard/clever problems) and productive development (meeting deadlines, building/fixing projects). Many of the stereotypical developer interviews assume "good developers" who are good at one are good at the other. My anecdotal experience is that most are not, they excel in one or the other.
  - deleted 6y ago
    [deleted]
  - Olreich 6y ago
    I don’t think they are exclusive, you can be a great engineer that knows all about the algorithms in CS books and keeps up on the latest research in parts of the academic field. They aren’t even different skill sets, just different places to find the knowledge. Game development and cryptography bear this out well. The top engineers are the ones that read all the papers and that can build solid systems based on them.
    - alpha_squared 6y ago
      I don't think I framed it as exclusive, but that the intersection of both sets of skills is uncommon (I attest they are two different sets). Sure, you can aim to hire only people who excel in both, but the typical software interview certainly doesn't assess the latter.
- gedy 6y ago
  I see this time and again from observation over 20 years. Also many great engineers don't have cs backgrounds and don't do well on whiteboard/leetcode
- PopeDotNinja 6y ago
  It's like memorizing the answers to an IQ test. A relative of mind knows all the answers, as he conducted the test on many occasions, but he doesn't have the max IQ.
bonoboTP 6y ago
> "truly believe in the mission/goals of the company/project" Well, first make sure there actually is some "mission" there to "believe in". In 99.9% of the cases people work on perhaps technically exciting but impactwise boring industry projects, whose main mission is to make money. Requiring passion is a euphemism for being over-workable. That you'll work outside working hours, you can be called during vacation etc, because of course this job is the mission of your life, right? Otherwise, you're not passionate... In practice, these "passionate" people are often young and naive or just really ambitious and work their ass off to build their resume and will be on their way soon enough. A good engineer contributes high value and knows this. And therefore will make the company pay up, and will not take any bullshit. And being that good, they will be picking where to work and you as the company compete to get them. In these cases the interview is about the engineer judging the company and looking for exactly things like the requirement to "believe in the mission" or whether the conversation is on a professional, adult level.
- protonimitate 6y ago
  > In 99.9% of the cases people work on perhaps technically exciting but impactwise boring industry projects, whose main mission is to make money. Passion, imo, is an attitude - and is reflected in quality of work and teamwork. I do think it is often used to trick people into getting less than they deserve, but doesn't have to be the case. When I say I want to work with passionate people, I don't mean people who are willing to work 80 hours a week for 10 dollars an hour because they love coding or want to solve world hunger. I mean I want to work with people who give their best, put the user/experience first, and are genuinely motivated to do good work. This doesn't mean you have to fake smile, or get taken advantage of, it just means you have bring some positivity to the table.
  - vsareto 6y ago
    Some people give their best, but sometimes it isn't good enough. In those cases, the company will just look for someone else. Then that passion counts for little. This is the issue with signalling passion - it causes people to drive their expectations of you higher. You have a higher chance of not living up to someone's standard.
carlosdp 6y ago
> Personally, I think temp-to-hire or work-for-a-day type models could work I've seen the contract-to-hire stuff come back up recently. There are a couple big flaws with that model, beyond possible abuse, though. 1) You basically are excluding yourself from any candidate that still has a full time job. It would be crazy for them to leave their current gig for a place that might not actually bring them on full time. 2) It doesn't scale, bringing on new team members properly takes time and effort. Not only are you potentially wasting their time, you are also wasting your team's time, and you can only do this kind of evaluation so many times. I do agree though that the standard interview model doesn't make sense. Memorizing leetcode problems doesn't map to good engineering, and take home problems filter out a lot of good engineers that won't bother spending the extra time.
ilyaa 6y ago
Hey, thanks for insightful feedback (blog post author here)! I fully agree with you about temp-to-hire or work-for-a-day models. And I think if the company has such a process in place, it is the sign of a well-developed culture. It is exactly the point of the blogpost -- evaluate using the right metric for the job. However, this is rare in practice, and I would say for the same reason the well-established culture is rare: it requires a lot of work to be done by the very same great people in advance. It is a chicken and egg problem. I saw countless times in multiple companies the signs of immaturity that prevent them from giving people access to their production: secrets in git, shared accounts, lack of established onboarding and offboarding processes, no processes for credential rotation, absence of audit logs, legal issues, you name it. The majority of companies can not afford a work-for-a-day option. Maintaining the dedicated challenge is another option for them, but it has its downsides, too. In our experience, the main one is that you either invest in creating the grading infrastructure upfront or spend a lot of time manually grading the submissions. Checking the correctness of the code is a hard task unless one can actually run it and see how it performs under a barrage of tests. I'm not saying there should not be a code review, of course. I am saying that the automated assessment of the code should be done before you start spending human time. Machines are cheap, your engineers are not. And at this point, one doesn't need AutoIterative, but has effectively reinvented it :) We walked this path ourselves over the years. We did fizzbuzz style interviews, that was suboptimal. We graded challenges manually for a long time, which was tedious and error-prone. We made all the mistakes and tried to fix them. We were using the wrong metric. Qualitative improvement was when we started using the right metric, and automating its assessment was the next logical step.
collyw 6y ago
> They could recite any leetcode problem from memory, There is your problem. You are testing for something unrelated to the job.
kamaal 6y ago
>>They could recite any leetcode problem from memory, but didn't know how to contribute value to their team or company. That's understandable any one with that much proficiency on leetcode would be wasting their whole day on that site than building real things.

Fiveplus 6y ago

> handful of devs who passed our interview process with flying colors, but couldn't actually deliver anything once they got in the door. They could recite any leetcode problem from memory, but didn't know how to contribute value to their team or company. Interesting, why do you think was this the case?

alpha_squared 6y ago
Not the person you're responding to, but my perspective is that there is often a false equivalency between academic understanding (solving hard/clever problems) and productive development (meeting deadlines, building/fixing projects). Many of the stereotypical developer interviews assume "good developers" who are good at one are good at the other. My anecdotal experience is that most are not, they excel in one or the other.
- deleted 6y ago
  [deleted]
- Olreich 6y ago
  I don’t think they are exclusive, you can be a great engineer that knows all about the algorithms in CS books and keeps up on the latest research in parts of the academic field. They aren’t even different skill sets, just different places to find the knowledge. Game development and cryptography bear this out well. The top engineers are the ones that read all the papers and that can build solid systems based on them.
  - alpha_squared 6y ago
    I don't think I framed it as exclusive, but that the intersection of both sets of skills is uncommon (I attest they are two different sets). Sure, you can aim to hire only people who excel in both, but the typical software interview certainly doesn't assess the latter.
gedy 6y ago
I see this time and again from observation over 20 years. Also many great engineers don't have cs backgrounds and don't do well on whiteboard/leetcode
PopeDotNinja 6y ago
It's like memorizing the answers to an IQ test. A relative of mind knows all the answers, as he conducted the test on many occasions, but he doesn't have the max IQ.

bonoboTP 6y ago

> "truly believe in the mission/goals of the company/project" Well, first make sure there actually is some "mission" there to "believe in". In 99.9% of the cases people work on perhaps technically exciting but impactwise boring industry projects, whose main mission is to make money. Requiring passion is a euphemism for being over-workable. That you'll work outside working hours, you can be called during vacation etc, because of course this job is the mission of your life, right? Otherwise, you're not passionate... In practice, these "passionate" people are often young and naive or just really ambitious and work their ass off to build their resume and will be on their way soon enough. A good engineer contributes high value and knows this. And therefore will make the company pay up, and will not take any bullshit. And being that good, they will be picking where to work and you as the company compete to get them. In these cases the interview is about the engineer judging the company and looking for exactly things like the requirement to "believe in the mission" or whether the conversation is on a professional, adult level.

protonimitate 6y ago
> In 99.9% of the cases people work on perhaps technically exciting but impactwise boring industry projects, whose main mission is to make money. Passion, imo, is an attitude - and is reflected in quality of work and teamwork. I do think it is often used to trick people into getting less than they deserve, but doesn't have to be the case. When I say I want to work with passionate people, I don't mean people who are willing to work 80 hours a week for 10 dollars an hour because they love coding or want to solve world hunger. I mean I want to work with people who give their best, put the user/experience first, and are genuinely motivated to do good work. This doesn't mean you have to fake smile, or get taken advantage of, it just means you have bring some positivity to the table.
- vsareto 6y ago
  Some people give their best, but sometimes it isn't good enough. In those cases, the company will just look for someone else. Then that passion counts for little. This is the issue with signalling passion - it causes people to drive their expectations of you higher. You have a higher chance of not living up to someone's standard.

carlosdp 6y ago

> Personally, I think temp-to-hire or work-for-a-day type models could work I've seen the contract-to-hire stuff come back up recently. There are a couple big flaws with that model, beyond possible abuse, though. 1) You basically are excluding yourself from any candidate that still has a full time job. It would be crazy for them to leave their current gig for a place that might not actually bring them on full time. 2) It doesn't scale, bringing on new team members properly takes time and effort. Not only are you potentially wasting their time, you are also wasting your team's time, and you can only do this kind of evaluation so many times. I do agree though that the standard interview model doesn't make sense. Memorizing leetcode problems doesn't map to good engineering, and take home problems filter out a lot of good engineers that won't bother spending the extra time.

ilyaa 6y ago

Hey, thanks for insightful feedback (blog post author here)! I fully agree with you about temp-to-hire or work-for-a-day models. And I think if the company has such a process in place, it is the sign of a well-developed culture. It is exactly the point of the blogpost -- evaluate using the right metric for the job. However, this is rare in practice, and I would say for the same reason the well-established culture is rare: it requires a lot of work to be done by the very same great people in advance. It is a chicken and egg problem. I saw countless times in multiple companies the signs of immaturity that prevent them from giving people access to their production: secrets in git, shared accounts, lack of established onboarding and offboarding processes, no processes for credential rotation, absence of audit logs, legal issues, you name it. The majority of companies can not afford a work-for-a-day option. Maintaining the dedicated challenge is another option for them, but it has its downsides, too. In our experience, the main one is that you either invest in creating the grading infrastructure upfront or spend a lot of time manually grading the submissions. Checking the correctness of the code is a hard task unless one can actually run it and see how it performs under a barrage of tests. I'm not saying there should not be a code review, of course. I am saying that the automated assessment of the code should be done before you start spending human time. Machines are cheap, your engineers are not. And at this point, one doesn't need AutoIterative, but has effectively reinvented it :) We walked this path ourselves over the years. We did fizzbuzz style interviews, that was suboptimal. We graded challenges manually for a long time, which was tedious and error-prone. We made all the mistakes and tried to fix them. We were using the wrong metric. Qualitative improvement was when we started using the right metric, and automating its assessment was the next logical step.

collyw 6y ago

> They could recite any leetcode problem from memory, There is your problem. You are testing for something unrelated to the job.

kamaal 6y ago

>>They could recite any leetcode problem from memory, but didn't know how to contribute value to their team or company. That's understandable any one with that much proficiency on leetcode would be wasting their whole day on that site than building real things.