9 ms·

As with everything, it depends. Live coding interviews work. They’re not the best candidate experience, but they work at Meta, Google scale, minimizing false po

by domrdy 1y ago

As with everything, it depends. Live coding interviews work. They’re not the best candidate experience, but they work at Meta, Google scale, minimizing false positives better than most other formats. What makes them stressful is the lack of interviewer training and the abstract, puzzle-like nature of the problems, which you can really only solve if you’ve spent time studying (e.g., LeetCode) or you’re fresh out of college or academia. I’ve worked in the assessment space for 6 years and have seen many hiring processes, from Fortune 10 companies to startups hiring their first engineers. The range of signal required, "how much time can my engineering team spend with a candidate", and how much candidate experience you can get away with, is huge. I’ve also been a candidate myself and failed many live coding interviews. It made me feel terrible about myself. The last time for a role at Ycombinator (the interviewer was super nice). When I work on my product, I try to view it through the lens of empowering candidates to show their skills and potential. I encourage our customers to use assessments that somewhat resemble on-the-job skills. I don’t like the phrasing “real work” anymore. An assessment shouldn’t be unpaid labor, it should be a way for candidates to demonstrate that they can do the job and handle future work thrown at them, and for hiring managers to feel confident extending what are often very high salaries in tech. With AI, unfortunately, short take-homes (what I prefer as a candidate, using my own tools and editor) are becoming harder to maintain as a fair signal due to AI assistance. I’ve seen companies move back to onsite, and competitors deploy all kinds of proctoring and invasive monitoring. The perfect solution, in my view, would be an assessment where the candidate feels relaxed and able to perform at their best, with their own editor and configuration, knowing that every other candidate in the pool has the same constraints in terms of time and tooling. It’s a tough problem to solve. I think about it daily and have not come up with a solution.

zwnow 1y ago
A brick layer isn't asked to build a wall prior to getting a job. A certificate confirming he did learn the job is enough for companies to employ them. Same goes for a lot of other jobs. Just hire devs and kick them out if they dont fit ur needs. Why go through these humiliating corpo processes?
- appease7727 1y ago
  So you think only people wealthy enough for a college degree should be programmers?
  - zwnow 1y ago
    How did u get to this conclusion?
    - darkwater 1y ago
      You wrote > A certificate confirming he did learn the job What would that certificate be?
      - pjmlp 1y ago
        In Germany that would be a recomendation letter, pretty much valuable by most HR departments, and not showing up with one isn't really recommended, a shitty one is better than none at all.
        zwnow 1y ago
        Yea apprenticeship certification is what I have (Germany). It's a lot different from other countries apprenticeships and shows ppl "Hey that dude worked this job for 3 years". College degree should also work as you usually gather some job experience during university. What i not meant were specific certificates like the Cisco ones. Even a letter of recommendation from a previous employer would count for me.
        darkwater 1y ago
        Aren't those the dreaded recommendation letters where "good" means "bad", "very good" means "average" and "extraordinarily good" means "good" ?
  - ozgung 1y ago
    That's the real problem with the current process. It assumes all the candidates are just random people without any credentials. So candidates need to prove that they can code each and every time. Even a Phd degree doesn't count. Years of work experience doesn't count. And they test you with challenging competitive programming questions. Not even just normal programming. This process used to be only for Google etc, where demand is too high. It's just an elimination process. But today every company in every country blindly copies this insanity. Solving a hard dynamic programming question in 20 minutes after 2 months of studying means nothing from a real engineering perspective.
belter 1y ago
Sounds you are looking to hire competitive coders not software engineers,
- nyantaro1 1y ago
  I would love to see data about correlation between competitive programming competence and actual software engineering competence. I would naively assume there is a big positive signal, but it is pure speculation.
  - belter 1y ago
    Very low correlation...A bit like pro soccer player vs soccer acrobatics street performer...Drone-racing pilot vs aerospace engineer certifying a commercial airframe...
throwzasdf 1y ago
- What makes them stressful is the lack of interviewer training and the abstract, puzzle-like nature of the problems, which you can really only solve if you’ve spent time studying (e.g., LeetCode) or you’re fresh out of college or academia. This allows ageism without repercussions.
gabrieledarrigo 1y ago
> As with everything, it depends. Live coding interviews work You cannot start your reasoning with "it depends" and then continue with an absolute. I could do the same: As with everything, it depends. Live coding interviews don't work. What's the difference?
- domrdy 1y ago
  Thank you. I'm not a native speaker, what I was trying to say was, that it depends on the "use-case".
pelcg 1y ago
> As with everything, it depends. Live coding interviews work. They’re not the best candidate experience, but they work at Meta, Google scale, minimizing false positives better than most other formats. What makes them stressful is the lack of interviewer training and the abstract, puzzle-like nature of the problems, which you can really only solve if you’ve spent time studying (e.g., LeetCode) or you’re fresh out of college or academia. They work (for some) and leet-code weeds out the frauds that really cannot problem solve and assesses those who have not built anything to show to justify not doing it and can be applied to companies that are joining from an acquisition. > The perfect solution, in my view, would be an assessment where the candidate feels relaxed and able to perform at their best, knowing that every other candidate in the pool has the same constraints in terms of time and tooling. It’s a tough problem to solve. And that would be the fairest one which is to do the leetcode interview in person on a projected whiteboard and pair programming with the interviewer. Very relaxing.
exabrial 1y ago
>Live coding interviews work do they?
lubesGordi 1y ago
Just make the take home assume that AI is going to be used.
deleted 1y ago
[deleted]
lesuorac 1y ago
> As with everything, it depends. That's the issue though. If you're paying top 1% wages then you should get top 1% performers. If you want to pay bottom 20% wages then how do you select them using a whiteboard test?
nailer 1y ago
> I encourage our customers to use assessments that somewhat resemble on-the-job skills. On the job will I constantly have to craft a narrative for another person while I code? Do you know how many autistic people who are great programmers you're throwing away by asking them to multitask rather than letting them deep focus?
- domrdy 1y ago
  I agree with you. I’ve failed many live-coding interviews because I start focusing on how the interviewer perceives me instead of the task at hand haha. Hiring managers still want to hear the candidate talk through a technical problem. I built a way for candidates to screen-record their take-home solution using only their microphone and screen share, with as many takes as they want. This lets hiring managers hear how candidates communicate and explain their technical implementations, a skill that matters on teams. But it means managers have to watch those videos. Many of our customers use it and candidates like it, but it still takes extra effort on the candidate’s and hiring managers part, with no guarantee that it will "pay off".
  - nailer 1y ago
    > This lets hiring managers hear how candidates communicate and explain their technical implementations, a skill that matters on teams. Yes, communicating and explaining technical implementations matters, but not while I am coding. Allow people to present separately from coding.
    - domrdy 1y ago
      Agreed, the way it is implemented is that candidates do the recording after they submit their work.
timeinput 1y ago
> They’re not the best candidate experience, but they work at Meta, Google scale, minimizing false positives better than most other formats. Do they? That is an assertion with out any data. I've worked with F tier developers who worked at Facebook, and Google. They're trying to minimize their false positive rate, but they don't realistically publish it other than laying off large portions of their work force. Presumably because they weren't great, but passed the interview process. If you lay off 3-5% of your staff in a year you had a 3-5% false positive rate. Maybe it's minimized by these in person interviews, but that seems like an unacceptably high false positive rate given how much time the interviewers spend on the process. 3% is probably about as accurate as 'solve fizz buzz in python' prior to AI, and that's where they were in 2022.
- lubujackson 1y ago
  It sure is easy to cover up bad hires when you print money. And "false positives" come in lots of different flavors. Are they an asshole? Do they add unnecessary complexity in everything they touch? Do they interrupt at meetings and interject their own pet ideas? If you hire specifically on ability to invert btrees you will get precisely that. The question is how relevant that is to the various jobs being done there.
- Apocryphon 1y ago
  How'd they get hired, in your opinion?
  - timeinput 1y ago
    At a FAANG company? A 5% or higher false positive rate in the FAANG hiring process. I feel like I specifically said that. At the company I worked in at the time? An ineffectual hiring process for different reasons that was more on "feels", these folks probably would have been hired whether or not they worked for FAANG, but "they worked for FAANG" was always in the out brief. They seemed talented (I'm sure that's why they were hired at FAANG companes). I'm not proposing a solution, or saying that the company I worked for was perfect, but <worked for FAANG> means they made it through a 95%cutoff that is unreliable rather than they're a good SWE. Additionally emulating FAANG hiring processes is probably not a great idea unless you can hire 1000 engineers and fifty of them as that is what FAANG does. If you can only hire 10 engineers you can not realistically mimic FAANG hiring processes.
crystal_revenge 1y ago
> minimizing false positives This has not been my experience at all. Years ago being ex-FAANG was a pretty good signal, but as the filter has increasingly become "regurgitate a bunch of leetcode" the quality of FAANG engineers has plummeted. The teams I've worked that have used live coding filters all had far worse devs than all the startups I've worked with that didn't require live coding. In the old "algorithm whiteboarding" days, I think it was a decent signal, but now it's just a sad parody of this and the results show.
sotix 1y ago
Some of the best engineers I've ever met are always false positives. They get nervous in live interviews. I don't think one can say live coding interviews work so matter of factly when it's eliminating some top, top computer scientists.
quails8mydog 1y ago
Yeah, in my admittedly limited experience the grade I assigned in a live code test (slightly more than fizzbuzz, but no tricks or algorithms required) matched up very closely with real world performance. I even have knowledge of some of the fails as people higher up the chain decided to hire them anyway. They didn't do well from the feedback I received.

by domrdy 1y ago

zwnow 1y ago

A brick layer isn't asked to build a wall prior to getting a job. A certificate confirming he did learn the job is enough for companies to employ them. Same goes for a lot of other jobs. Just hire devs and kick them out if they dont fit ur needs. Why go through these humiliating corpo processes?

appease7727 1y ago
So you think only people wealthy enough for a college degree should be programmers?
- zwnow 1y ago
  How did u get to this conclusion?
  - darkwater 1y ago
    You wrote > A certificate confirming he did learn the job What would that certificate be?
    - pjmlp 1y ago
      In Germany that would be a recomendation letter, pretty much valuable by most HR departments, and not showing up with one isn't really recommended, a shitty one is better than none at all.
      - zwnow 1y ago
        Yea apprenticeship certification is what I have (Germany). It's a lot different from other countries apprenticeships and shows ppl "Hey that dude worked this job for 3 years". College degree should also work as you usually gather some job experience during university. What i not meant were specific certificates like the Cisco ones. Even a letter of recommendation from a previous employer would count for me.
      - darkwater 1y ago
        Aren't those the dreaded recommendation letters where "good" means "bad", "very good" means "average" and "extraordinarily good" means "good" ?
- ozgung 1y ago
  That's the real problem with the current process. It assumes all the candidates are just random people without any credentials. So candidates need to prove that they can code each and every time. Even a Phd degree doesn't count. Years of work experience doesn't count. And they test you with challenging competitive programming questions. Not even just normal programming. This process used to be only for Google etc, where demand is too high. It's just an elimination process. But today every company in every country blindly copies this insanity. Solving a hard dynamic programming question in 20 minutes after 2 months of studying means nothing from a real engineering perspective.

belter 1y ago

Sounds you are looking to hire competitive coders not software engineers,

nyantaro1 1y ago
I would love to see data about correlation between competitive programming competence and actual software engineering competence. I would naively assume there is a big positive signal, but it is pure speculation.
- belter 1y ago
  Very low correlation...A bit like pro soccer player vs soccer acrobatics street performer...Drone-racing pilot vs aerospace engineer certifying a commercial airframe...

throwzasdf 1y ago

- What makes them stressful is the lack of interviewer training and the abstract, puzzle-like nature of the problems, which you can really only solve if you’ve spent time studying (e.g., LeetCode) or you’re fresh out of college or academia. This allows ageism without repercussions.

gabrieledarrigo 1y ago

> As with everything, it depends. Live coding interviews work You cannot start your reasoning with "it depends" and then continue with an absolute. I could do the same: As with everything, it depends. Live coding interviews don't work. What's the difference?

domrdy 1y ago
Thank you. I'm not a native speaker, what I was trying to say was, that it depends on the "use-case".

pelcg 1y ago

> As with everything, it depends. Live coding interviews work. They’re not the best candidate experience, but they work at Meta, Google scale, minimizing false positives better than most other formats. What makes them stressful is the lack of interviewer training and the abstract, puzzle-like nature of the problems, which you can really only solve if you’ve spent time studying (e.g., LeetCode) or you’re fresh out of college or academia. They work (for some) and leet-code weeds out the frauds that really cannot problem solve and assesses those who have not built anything to show to justify not doing it and can be applied to companies that are joining from an acquisition. > The perfect solution, in my view, would be an assessment where the candidate feels relaxed and able to perform at their best, knowing that every other candidate in the pool has the same constraints in terms of time and tooling. It’s a tough problem to solve. And that would be the fairest one which is to do the leetcode interview in person on a projected whiteboard and pair programming with the interviewer. Very relaxing.

exabrial 1y ago

>Live coding interviews work do they?

lubesGordi 1y ago

Just make the take home assume that AI is going to be used.

deleted 1y ago

[deleted]

lesuorac 1y ago

> As with everything, it depends. That's the issue though. If you're paying top 1% wages then you should get top 1% performers. If you want to pay bottom 20% wages then how do you select them using a whiteboard test?

nailer 1y ago

> I encourage our customers to use assessments that somewhat resemble on-the-job skills. On the job will I constantly have to craft a narrative for another person while I code? Do you know how many autistic people who are great programmers you're throwing away by asking them to multitask rather than letting them deep focus?

domrdy 1y ago
I agree with you. I’ve failed many live-coding interviews because I start focusing on how the interviewer perceives me instead of the task at hand haha. Hiring managers still want to hear the candidate talk through a technical problem. I built a way for candidates to screen-record their take-home solution using only their microphone and screen share, with as many takes as they want. This lets hiring managers hear how candidates communicate and explain their technical implementations, a skill that matters on teams. But it means managers have to watch those videos. Many of our customers use it and candidates like it, but it still takes extra effort on the candidate’s and hiring managers part, with no guarantee that it will "pay off".
- nailer 1y ago
  > This lets hiring managers hear how candidates communicate and explain their technical implementations, a skill that matters on teams. Yes, communicating and explaining technical implementations matters, but not while I am coding. Allow people to present separately from coding.
  - domrdy 1y ago
    Agreed, the way it is implemented is that candidates do the recording after they submit their work.

timeinput 1y ago

> They’re not the best candidate experience, but they work at Meta, Google scale, minimizing false positives better than most other formats. Do they? That is an assertion with out any data. I've worked with F tier developers who worked at Facebook, and Google. They're trying to minimize their false positive rate, but they don't realistically publish it other than laying off large portions of their work force. Presumably because they weren't great, but passed the interview process. If you lay off 3-5% of your staff in a year you had a 3-5% false positive rate. Maybe it's minimized by these in person interviews, but that seems like an unacceptably high false positive rate given how much time the interviewers spend on the process. 3% is probably about as accurate as 'solve fizz buzz in python' prior to AI, and that's where they were in 2022.

lubujackson 1y ago
It sure is easy to cover up bad hires when you print money. And "false positives" come in lots of different flavors. Are they an asshole? Do they add unnecessary complexity in everything they touch? Do they interrupt at meetings and interject their own pet ideas? If you hire specifically on ability to invert btrees you will get precisely that. The question is how relevant that is to the various jobs being done there.
Apocryphon 1y ago
How'd they get hired, in your opinion?
- timeinput 1y ago
  At a FAANG company? A 5% or higher false positive rate in the FAANG hiring process. I feel like I specifically said that. At the company I worked in at the time? An ineffectual hiring process for different reasons that was more on "feels", these folks probably would have been hired whether or not they worked for FAANG, but "they worked for FAANG" was always in the out brief. They seemed talented (I'm sure that's why they were hired at FAANG companes). I'm not proposing a solution, or saying that the company I worked for was perfect, but <worked for FAANG> means they made it through a 95%cutoff that is unreliable rather than they're a good SWE. Additionally emulating FAANG hiring processes is probably not a great idea unless you can hire 1000 engineers and fifty of them as that is what FAANG does. If you can only hire 10 engineers you can not realistically mimic FAANG hiring processes.

crystal_revenge 1y ago

> minimizing false positives This has not been my experience at all. Years ago being ex-FAANG was a pretty good signal, but as the filter has increasingly become "regurgitate a bunch of leetcode" the quality of FAANG engineers has plummeted. The teams I've worked that have used live coding filters all had far worse devs than all the startups I've worked with that didn't require live coding. In the old "algorithm whiteboarding" days, I think it was a decent signal, but now it's just a sad parody of this and the results show.

sotix 1y ago

Some of the best engineers I've ever met are always false positives. They get nervous in live interviews. I don't think one can say live coding interviews work so matter of factly when it's eliminating some top, top computer scientists.

quails8mydog 1y ago

Yeah, in my admittedly limited experience the grade I assigned in a live code test (slightly more than fizzbuzz, but no tricks or algorithms required) matched up very closely with real world performance. I even have knowledge of some of the fails as people higher up the chain decided to hire them anyway. They didn't do well from the feedback I received.