7 ms·

We can already automate leetcode interviews much more easily then why don't we do it? Because people cheat. U can ask your competitive coder friend or search go

We can already automate leetcode interviews much more easily then why don't we do it? Because people cheat. U can ask your competitive coder friend or search google to get the answers. Similarly, here also friends can give the complete solutions even more easily. Heck there are sites dedicated to these kinds of services. This leads to following consequences: 1. In both automated leetcode and automated "coding" interviews, most people get full score. This even turns into a negative bias as an actual good candidate who did not cheat has greater chance of getting rejected here than the cheaters. 2. Interviewers put both automated leetcode and in-person leetcode to first filter out candidates, then put them through actual interview hoops. If your product succeeds, this means another layer of "interviewing" and more wastage of engineer's time. Sorry to say this isn't solving any problem but has the potential to create more harm for both sides.

imglorp 5y ago
We counter this by giving applicants a small homework problem and ask them to solve it in the language of their choice. Then we have tech interview centered around their solution. We'd know in about a minute if someone else did the did the thinking for them. It's also a good jumping off point for all kinds of deeper questions, systems knowledge, engineering practices, wherever we need to go.
austincheney 5y ago
I suspect cheating is not the primary problem. I suspect, like so many other decisions in software, the primary problem is unrealistic and unfounded expectations. This behavior typically comes from unmeasured assumptions that may work well in one narrow circumstance applied into an unrelated circumstance and then blaming the result instead of the decision. This is bias. Unless there is a solution that directly addresses hiring bias this company will fail like all the others before it for exactly the same reason: their clientele will apply bias (the actual cause of the problem), be unhappy with the result, and then blame the service provider. In sales the client is never wrong, which means the service provider is always wrong. This is why software overspends on recruiters and then invents fantasies like React developers with a year of experience are rare.
indymike 5y ago
I routinely see 75% fail.
keerthiko 5y ago
I think you missed the entire premise of Litebulb — they are avoiding Leetcode-style testing in favor of development tasks on the real business codebase, and quickly ramping up environments that are based on your business' proprietary dev stack. Many software companies opt to avoid leetcode-like interviewing entirely. If leetcode and prepackaged, recycled, self-contained code tests are your workplace's primary process for hiring, then I don't think this service is for you at all.
justshowpost 5y ago
I don’t think cheating prevention is in scope for a tool like this and neither should be. Any decent employer puts the new hire on a trial period first. During the trial period, the employer should task them with real work. When the candidate doesn’t deliver, the employer can let them go. When they do deliver, the employer can keep them. In both cases, does it really matter if they cheated their way in or not? No. Preventing cheating is the wrong place to optimize. While we’re here, dumb memorization of LeetCode style Q&As instead of deep understanding of underlying principles can be seen as cheating just as well. Doing actual work is better in every way. And many employers already do exactly this coding work already, just manually. They make up a problem themselves, and give you a time limit to commit a solution without checking if you’re cheating or not. Litebulb just automates this, by taking over the creation of the problem and the analysis of the solution. There’s lots of advantages of the centralization of this, like creating more problems, fixing and updating existing ones, etc. Provided the problems and the analysis are as good as advertised, I think Litebulb has a great value proposition. I wish them success and hope they can have a lasting impact on the hiring process for software developers.
- garyjlin 5y ago
  Really appreciate the support here and you make some excellent points! We're trying to make something that already exists just a bit better. :)
- visarga 5y ago
  > During the trial period, the employer should task them with real work. When the candidate doesn’t deliver, the employer can let them go. That would mean you waste a few months for every bad trial hire, especially if you're looking to fill just one or two positions.
  - justshowpost 5y ago
    If your company takes months to figure out if someone does good work or not then it has some serious problems. If you have expertise in a field you can normally guess pretty quickly when someone bullshits their way through or actually knows what they’re talking about. But your hiring staff usually don’t have any or deep enough expertise for the field they’re hiring for, which only gets worse the bigger and more advanced your company gets. But your employees with which the new hire will be working with should have this expertise (or again your company has a problem). So they will find out pretty quickly if the hire is a fraud or not. Cheating surely creates costs but not as much as you suggest. And as others have mentioned above it is surely not as widespread as you suggest, limiting the cost further.
garyjlin 5y ago
1. That point is actually one of the reasons behind Litebulb! For most LeetCode-style tools, getting 100% is easy if you just google the question, copy the top answer, then change up a few variables. For Litebulb, a legitimate submission has multiple commits, so when we do git history matching, no two legitimate submissions will have the exact same commit history, which means submissions either have a <5% or >95% match. The >95% matches are plagiarism and will be flagged. 2. Litebulb is meant to replace the initial LeetCode question + the mid-funnel technical screen. We still recommend doing a hiring manager screen at the beginning and team onsite at the end, but using Litebulb correctly should lessen an engineer's interviewing burden, not increase it.
- 2c2c2c 5y ago
  so you're gunning to replace the 30 minute leetcode easy over the phone??
  - garyjlin 5y ago
    It's idealistic and naive to say that we'll replace the entire onsite interview, and I wouldn't even recommend we go in that direction. What we're trying to do is clear out the technical evaluation part of an onsite so that there's more time for engineers to have insightful conversations with the candidate, and to get to know them more as humans and potential colleagues. If you're spending too much time with a candidate staring at a whiteboard going over code, you don't have enough time to gauge culture fit. However, because we're still early, it's not fair to ask clients to chuck out the tech eval part of their onsites, since we haven't proven effectiveness to them yet. So instead, we'll replace whatever they use mid-funnel, and also early funnel auto-interviews. We still recommend someone talk to the candidate both before and after the Litebulb interview.

imglorp 5y ago

We counter this by giving applicants a small homework problem and ask them to solve it in the language of their choice. Then we have tech interview centered around their solution. We'd know in about a minute if someone else did the did the thinking for them. It's also a good jumping off point for all kinds of deeper questions, systems knowledge, engineering practices, wherever we need to go.

austincheney 5y ago

I suspect cheating is not the primary problem. I suspect, like so many other decisions in software, the primary problem is unrealistic and unfounded expectations. This behavior typically comes from unmeasured assumptions that may work well in one narrow circumstance applied into an unrelated circumstance and then blaming the result instead of the decision. This is bias. Unless there is a solution that directly addresses hiring bias this company will fail like all the others before it for exactly the same reason: their clientele will apply bias (the actual cause of the problem), be unhappy with the result, and then blame the service provider. In sales the client is never wrong, which means the service provider is always wrong. This is why software overspends on recruiters and then invents fantasies like React developers with a year of experience are rare.

indymike 5y ago

I routinely see 75% fail.

keerthiko 5y ago

I think you missed the entire premise of Litebulb — they are avoiding Leetcode-style testing in favor of development tasks on the real business codebase, and quickly ramping up environments that are based on your business' proprietary dev stack. Many software companies opt to avoid leetcode-like interviewing entirely. If leetcode and prepackaged, recycled, self-contained code tests are your workplace's primary process for hiring, then I don't think this service is for you at all.

justshowpost 5y ago

I don’t think cheating prevention is in scope for a tool like this and neither should be. Any decent employer puts the new hire on a trial period first. During the trial period, the employer should task them with real work. When the candidate doesn’t deliver, the employer can let them go. When they do deliver, the employer can keep them. In both cases, does it really matter if they cheated their way in or not? No. Preventing cheating is the wrong place to optimize. While we’re here, dumb memorization of LeetCode style Q&As instead of deep understanding of underlying principles can be seen as cheating just as well. Doing actual work is better in every way. And many employers already do exactly this coding work already, just manually. They make up a problem themselves, and give you a time limit to commit a solution without checking if you’re cheating or not. Litebulb just automates this, by taking over the creation of the problem and the analysis of the solution. There’s lots of advantages of the centralization of this, like creating more problems, fixing and updating existing ones, etc. Provided the problems and the analysis are as good as advertised, I think Litebulb has a great value proposition. I wish them success and hope they can have a lasting impact on the hiring process for software developers.

garyjlin 5y ago
Really appreciate the support here and you make some excellent points! We're trying to make something that already exists just a bit better. :)
visarga 5y ago
> During the trial period, the employer should task them with real work. When the candidate doesn’t deliver, the employer can let them go. That would mean you waste a few months for every bad trial hire, especially if you're looking to fill just one or two positions.
- justshowpost 5y ago
  If your company takes months to figure out if someone does good work or not then it has some serious problems. If you have expertise in a field you can normally guess pretty quickly when someone bullshits their way through or actually knows what they’re talking about. But your hiring staff usually don’t have any or deep enough expertise for the field they’re hiring for, which only gets worse the bigger and more advanced your company gets. But your employees with which the new hire will be working with should have this expertise (or again your company has a problem). So they will find out pretty quickly if the hire is a fraud or not. Cheating surely creates costs but not as much as you suggest. And as others have mentioned above it is surely not as widespread as you suggest, limiting the cost further.

garyjlin 5y ago

1. That point is actually one of the reasons behind Litebulb! For most LeetCode-style tools, getting 100% is easy if you just google the question, copy the top answer, then change up a few variables. For Litebulb, a legitimate submission has multiple commits, so when we do git history matching, no two legitimate submissions will have the exact same commit history, which means submissions either have a <5% or >95% match. The >95% matches are plagiarism and will be flagged. 2. Litebulb is meant to replace the initial LeetCode question + the mid-funnel technical screen. We still recommend doing a hiring manager screen at the beginning and team onsite at the end, but using Litebulb correctly should lessen an engineer's interviewing burden, not increase it.

2c2c2c 5y ago
so you're gunning to replace the 30 minute leetcode easy over the phone??
- garyjlin 5y ago
  It's idealistic and naive to say that we'll replace the entire onsite interview, and I wouldn't even recommend we go in that direction. What we're trying to do is clear out the technical evaluation part of an onsite so that there's more time for engineers to have insightful conversations with the candidate, and to get to know them more as humans and potential colleagues. If you're spending too much time with a candidate staring at a whiteboard going over code, you don't have enough time to gauge culture fit. However, because we're still early, it's not fair to ask clients to chuck out the tech eval part of their onsites, since we haven't proven effectiveness to them yet. So instead, we'll replace whatever they use mid-funnel, and also early funnel auto-interviews. We still recommend someone talk to the candidate both before and after the Litebulb interview.