6 ms·

Services that automate the interviewing or vetting process benefit the hiring manager, but feel impersonal and disrespectful to candidates. (Isn't my time valua

Services that automate the interviewing or vetting process benefit the hiring manager, but feel impersonal and disrespectful to candidates. (Isn't my time valuable too?) In today's market you will likely see a large percentage of candidates opt out of the interview process when you send them an automated technical interview.

__turbobrew__ 5y ago
Yes. If I’m going to put a large time and effort investment into interviewing I want an equal investment by the hiring team. We both need “skin in the game” so I know that they are semi-serious about hiring me.
- garyjlin 5y ago
  Yup, our clients have brought this up with us as well, which is why we recommend they offer 3 choices for the candidate to do the interview: 1. Hop on a call, do the interview in one sitting together and share thoughts/questions 2. Hop on a call, make sure set up is smooth, prompt is clear, questions are answered, then candidate goes off on their own. Hop on another call later in the day to demo the solution and talk over design decisions. 3. Do it completely async, submit the solution whenever you have time From what we've seen, this flexibility has simultaneously reduced dropoff rate and increased candidate experience, since we're catering to the candidate's needs. Some would prefer to do it face-to-face, others prefer to work on their own.
bobbylarrybobby 5y ago
Sounds like the perfect tool to select for candidates willing to do lots of grunt work.
- garyjlin 5y ago
  I wouldn't necessarily put it that way, primarily because some of our interviews require less than 20 lines of code! It's a bit more about selecting candidates with the right fundamentals. You don't need to write too much code, but the code that you do write, is it clean? Are you following best practices? Is it efficient? Have you thought about how your new service affects the rest of the system, and what the implications are? I do agree though that some of our frontend interviews are a bit heavier on code + css and less on systems design thinking. Btw just to clarify, we have a policy where we won't host any interviews that are just open dev tickets to build a feature that will be used in production. Like, we DO NOT tolerate companies using interviews to get free labor.
  - Nextgrid 5y ago
    > we DO NOT tolerate companies using interviews to get free labor. What does that change from a candidate perspective? In both cases the candidate wastes their time. I'd offer a better solution: the company pays market rate for the time needed for the test, and can then ask for anything they want - in which case an actual ticket is the best solution as it will evaluate the candidate's performance on the kinds of problems they'd actually be working day-to-day.
davidweatherall 5y ago
I am eagerly awaiting a YC startup that automates the coding interview from the candidate's side.
- Melatonic 5y ago
  Not a bad idea - you could get "vetted" by the company doing the interview and then companies that want to hire you could just search by type. So kind of LinkedIn badges but with actual meaning :-D
  - jahewson 5y ago
    That’s how Triplebyte use to operate. It didn’t work out.
    - Nextgrid 5y ago
      Do you know why it didn't work out? I'm curious.
    - treis 5y ago
      I think it did work out for them. Just not to the extent to justify the $100 million of VC money they took.
- rhizome 5y ago
  I believe this already exists. However, my impression is that while they may ask for your github, no potential employer ever actually checks it.
  - postalrat 5y ago
    Next level: a tool that rates quality and innovation in your github.
    - garyjlin 5y ago
      I've thought about this as well, the biggest problem here is that I have Github repos from 7 years ago with very, very, poor code quality, and also just a bunch of incomplete test repos. I wouldn't want that to lower the perception of my current skillset. If I could select which repos to include, that could be very interesting. Also interesting additions: open source contribution analyzer, Stack Overflow Q&A quality analyzer.
Melatonic 5y ago
If I was a hiring manager with the actual time to do this I think what I would likely do is come up with a series of thought experiments and some actual technical tests for the candidate. Probably I would try to pick things that I did not (off the top of my head) know the answer to and exactly how to do it. And then I might try to go through with the candidate and see their thought process through the whole thing while also volunteering some of my thought process to try to solve it together. The point would not even be to try to necessarily solve the problem itself - more just to see how I got along with them. Of course some people might freeze up when asked to do this, but I think that happens in any higher pressure interview environment. And if I do not know the solution off the top of my head either and am obviously even struggling in some parts myself I think that takes a ton of the pressure off of them to be a wizard that can solve it in the moment.
- garyjlin 5y ago
  This makes a ton of sense to me. Seeing how you solve a difficult problem with someone collaboratively is a great insight on how you'll continue working together, because that's exactly what you'll be doing. It's also very valuable to see how someone receives suggestions or criticism and how they respond. We'd actually like to build a lot of these concepts into Litebulb, ie: responding to code review requests? As an add-on, an interesting question one of our clients asks as a conversation starter is "what's the most technically complex project you've ever worked on?". This question sets up the space such that the candidate is the expert (since they've already worked on it), and it almost becomes a session where the candidate teaches the interviewer.
  - phibz 5y ago
    What matters most to me as a hiring manager is your ability to think critically. There's so much to know, and so many different permutations with how those things can present themselves that I'm not hiring you to know the answer every time. I'm hiring you to figure it out.
baby_wipe 5y ago
Making it cheaper for companies to interview increases the potential opportunities for me as a candidate. I do not take personal offense to being tested in an automated fashion.
- phibz 5y ago
  Have you taken a bunch of these and still gotten the same ghosting, empty promises, non communication and put in multiple hours of your time for free? I certainly have. The best interviews for me are where I can talk in depth about a topic and to a lesser extent, about a hypothetical problem and how I might handle it.
  - datavirtue 5y ago
    I tried a few of these with different companies. Each one has failed due to technical difficulties of the platform (in my case prerecorded video interviews) and in every case I tried to file a ticket and never heard anything back. Black hole. In one case I did a prerecorded interview and they later contacted everyone because something went wrong and they wanted everyone to do it again. Done feeling like a number.
jahewson 5y ago
I would have no problem with this as a 45min screen after having talking with a hiring manager. I’m going to invest that time anyway and it’s nicer to not have to “do it live”. There does need to be humans for the later rounds though - because I’m assessing the company as much as they’re assessing me!
- garyjlin 5y ago
  Yup! That's why we recommend using Litebulb as a mid-funnel tool. Initial screening should be with a hiring manager (or a founder if it's an early stage startup), final meetings should be with team members/managers to get cultural fit. We just try to make the technical coding evaluation part easier.
garyjlin 5y ago
I completely get what you're saying. We've seen many candidates that have expressed the concern of "if I'm going to put X units of time into this, you should also put X units of time into this". A good approach of running a Litebulb interview is to offer to do it in a sit-down session over a call with the interviewer, or to do it asynchronously. When we did sit-down sessions, we offered candidates a choice to just go off and do it throughout the day, whenever they can find the time. 100% of candidates chose to do it async. As it turns out, trying to code something with an interviewer breathing down your neck wasn't a good experience and made them very nervous. Note that it's an option, not mandatory, to do it async. If the candidate wanted to do it in a sit down session, then we prioritize making time for it to be there the entire session. The reason we offer this option is because devs that have busy lives often can't easily find a single consecutive multi-hour session to do an interview. They have full time jobs, and might have family obligations at home and over the weekends. Basically, the only way to actually do a full onsite interview is to take a day off work to go do the interview. We're giving the flexibility for the candidate to find a chunk of time here, a chunk of time there to complete the interview. We tested this above approach, because just as you said, we did see a huge dropoff when we just emailed candidates the interview link. After we offered to hop on a call and to be available for support, we saw dropoff reduced to less than 15%. Also, a good chunk of candidates that drop out of interview processes are senior engineers that can't be bothered with lengthy processes, which is fair. I wouldn't give a Litebulb interview to a senior engineer to begin with. Debatably I wouldn't give any kind of technical interview, actually. If they have decades of experience leading teams at big or high-growth tech companies, those achievements speak for itself. The interview process for seniors should be more geared towards finding a product-team-eng fit, rather than an evaluation.
- bryans 5y ago
  > As it turns out, trying to code something with an interviewer breathing down your neck wasn't a good experience and made them very nervous. That is the fundamental issue with technical interviews, because there is very little about that manufactured situation which is applicable to the candidate's ability to do the work. There are extremely few real-world coding situations that come anywhere close to the pressures felt while having someone hyper-analyze everything you do in real-time, particularly when that someone is responsible for deciding whether you're able to pay bills next month or feed your family next week. It's a needless, pointless and ultimately self-harming methodology that filters out exceptional candidates in favor of the most extroverted or arrogantly confident, which is a bizarre twist considering the most influential and game-changing coders are also famously some of the most introverted and "unusual" people. On top of that, because we're all in dependency hell at this point, far more time is spent googling for framework errors that don't make any sense until you find that one undocumented command line argument that a single person posted deep inside a GitHub issue thread. If you want to test someone's ability to do the real work, then give them an obscure error code from a little-used utility and see how long it takes to find the fix.
brimble 5y ago
Can confirm, I bail if you ask any amount of time at all be put into a "weed out" exercise before even talking to a real person. If my resume's not enough to get you interested in a conversation, I'll look elsewhere. If you have that many candidates and you find enough of them equally interesting that doing this is necessary, I reckon my odds are terrible anyway, so it's just a waste of time doing work and not just not getting paid, but very likely not seeing any kind of reward whatsoever. Moreover, if this is necessary, I kinda assume the pay is mediocre at best. If you knew how to get enough value out of me to be able to pay me really, really well, and if I look like a good fit for the job, then again, my resume ought to justify our talking. If any of that's not the case, then it seems like the pay's gonna suck or I'm just not a good fit, so I'm out either way. I want someone who's fishing for me with a lure, not by dragging a huge net through the water. Seems like a better sign all around.
- martinald 5y ago
  I do understand that, but from my experience short coding tests give by far the best signal. We ran an experiment and found that actually not doing a pre interview and straight to code test gave us better results than interview then coding test. Why? Because people tend to hire people with qualities like themselves, and overlook many things. It also then caused people to overrate/underrate the analysis of coding tests based on the previous interview (as they (dis)liked the person). This also massively increased the diversity of our team.
- rsyring 5y ago
  As a developer, I understand this perspective. But, from an employer's perspective, resumes tell you very little about a candidate and are essentially worthless. The sole usable metric we get from them is years of relevant experience. We respond with a questionnaire that gets us more uniform answers about experience and technologies used. Assuming the above doesn't filter the candidate out, we move on to a 90 min max "take home" exercise. In our case, it's not a coding exercise, but that's not that important for this discussion. Only if all of the above hasn't filtered out the candidate, do we move to a Zoom meeting. In the Zoom meeting, after an initial introduction, we do a few easy coding exercises. Like, reverse a string or sum the integers in a file easy. A few notable things from this experience: 1. The number of candidates who struggle with the Zoom interview coding exercise is surprising. It's a lot higher than expected and almost all of our candidates at that stage have 2+ years of professional development experience. 2. The number of candidates filtered out by the "take home" exercise is very high. 85+% We literally don't have enough time to talk to every candidate that applies with the right amount of experience. And, even if we did, many/most candidates on the market aren't good developers. Our signal to noise ratio is already bad. In that case, it would be terrible. As a developer, I sympathize with your perspective. But you may be passing on some good opportunities by not being willing to "play the game" a bit. Some employers are indeed bad actors and/or terrible at hiring. But given the current market, even those making a valiant effort to do it well, due to sheer volume, are usually going to have an exercise of some kind. We post our salary ranges up-front, don't want to waste anyone's time that way. We also put a ton of effort into a very detailed job description and application process. The hope is that the candidate can see the effort we put in and will be willing to reciprocate. If that doesn't end up being the case, we assume, like you, it probably wouldn't have been a good for anyway. FWIW.
oscargrouch 5y ago
With COVID and the rise of remote working there are a whole lot of companies where the developers are the asset, the commodity, where those companies are just the middle-man receiving for the work the developers do and getting the big part of the cut in the process. Those are mostly the "sweat-shops" that hire developers that can work for less for several reasons. I see that in this particular case this solution might be very welcome to assess new hires. But for more high-profile jobs and companies, the company that eventually follow this path will probably end with low quality hires, unless of course it just create crud's anyway. Because this is a self-fulfilling prophecy in the end. Once the candidates know what they will face, they will train until they get good in that game, which most of the time doesn't mirror the qualities required for the job.
hkt 5y ago
I second this. What's worse is that being a devops guy, most coding interviews I've ever done are totally irrelevant to the roles I've had. In a good market I can avoid companies using services like this.
ivanmaeder 5y ago
I built a very similar product 10 years ago and even back then we heard feedback like this. And I believe the situation may be more pronounced today. There seem to be markets (or large enough organizations?) where coding job ads get a tonne of applicants, many of them inexperienced, and so automation is a good fit. But not when things are tipped the other way. Our bet was for real-world tests too but it wasn't enough. A few things we missed that might help… - Candidates don't want to be treated like cattle - For many companies a good interview platform will be more beneficial than automation - Companies say they care about the experience candidates receive, they'll say they're rigorous and try to be as objective as possible with the way they collate information and make decisions, they'll talk about how high their standards are, etc etc. Be weary - There may be more gains to be made outside of the tech space - The problem most companies complained about and is probably still the hardest: going out and finding people I like the way you've solved the "real-world challenge" problem. Good luck and all the best!

__turbobrew__ 5y ago

Yes. If I’m going to put a large time and effort investment into interviewing I want an equal investment by the hiring team. We both need “skin in the game” so I know that they are semi-serious about hiring me.

garyjlin 5y ago
Yup, our clients have brought this up with us as well, which is why we recommend they offer 3 choices for the candidate to do the interview: 1. Hop on a call, do the interview in one sitting together and share thoughts/questions 2. Hop on a call, make sure set up is smooth, prompt is clear, questions are answered, then candidate goes off on their own. Hop on another call later in the day to demo the solution and talk over design decisions. 3. Do it completely async, submit the solution whenever you have time From what we've seen, this flexibility has simultaneously reduced dropoff rate and increased candidate experience, since we're catering to the candidate's needs. Some would prefer to do it face-to-face, others prefer to work on their own.

bobbylarrybobby 5y ago

Sounds like the perfect tool to select for candidates willing to do lots of grunt work.

garyjlin 5y ago
I wouldn't necessarily put it that way, primarily because some of our interviews require less than 20 lines of code! It's a bit more about selecting candidates with the right fundamentals. You don't need to write too much code, but the code that you do write, is it clean? Are you following best practices? Is it efficient? Have you thought about how your new service affects the rest of the system, and what the implications are? I do agree though that some of our frontend interviews are a bit heavier on code + css and less on systems design thinking. Btw just to clarify, we have a policy where we won't host any interviews that are just open dev tickets to build a feature that will be used in production. Like, we DO NOT tolerate companies using interviews to get free labor.
- Nextgrid 5y ago
  > we DO NOT tolerate companies using interviews to get free labor. What does that change from a candidate perspective? In both cases the candidate wastes their time. I'd offer a better solution: the company pays market rate for the time needed for the test, and can then ask for anything they want - in which case an actual ticket is the best solution as it will evaluate the candidate's performance on the kinds of problems they'd actually be working day-to-day.

davidweatherall 5y ago

I am eagerly awaiting a YC startup that automates the coding interview from the candidate's side.

Melatonic 5y ago
Not a bad idea - you could get "vetted" by the company doing the interview and then companies that want to hire you could just search by type. So kind of LinkedIn badges but with actual meaning :-D
- jahewson 5y ago
  That’s how Triplebyte use to operate. It didn’t work out.
  - Nextgrid 5y ago
    Do you know why it didn't work out? I'm curious.
  - treis 5y ago
    I think it did work out for them. Just not to the extent to justify the $100 million of VC money they took.
rhizome 5y ago
I believe this already exists. However, my impression is that while they may ask for your github, no potential employer ever actually checks it.
- postalrat 5y ago
  Next level: a tool that rates quality and innovation in your github.
  - garyjlin 5y ago
    I've thought about this as well, the biggest problem here is that I have Github repos from 7 years ago with very, very, poor code quality, and also just a bunch of incomplete test repos. I wouldn't want that to lower the perception of my current skillset. If I could select which repos to include, that could be very interesting. Also interesting additions: open source contribution analyzer, Stack Overflow Q&A quality analyzer.

Melatonic 5y ago

If I was a hiring manager with the actual time to do this I think what I would likely do is come up with a series of thought experiments and some actual technical tests for the candidate. Probably I would try to pick things that I did not (off the top of my head) know the answer to and exactly how to do it. And then I might try to go through with the candidate and see their thought process through the whole thing while also volunteering some of my thought process to try to solve it together. The point would not even be to try to necessarily solve the problem itself - more just to see how I got along with them. Of course some people might freeze up when asked to do this, but I think that happens in any higher pressure interview environment. And if I do not know the solution off the top of my head either and am obviously even struggling in some parts myself I think that takes a ton of the pressure off of them to be a wizard that can solve it in the moment.

garyjlin 5y ago
This makes a ton of sense to me. Seeing how you solve a difficult problem with someone collaboratively is a great insight on how you'll continue working together, because that's exactly what you'll be doing. It's also very valuable to see how someone receives suggestions or criticism and how they respond. We'd actually like to build a lot of these concepts into Litebulb, ie: responding to code review requests? As an add-on, an interesting question one of our clients asks as a conversation starter is "what's the most technically complex project you've ever worked on?". This question sets up the space such that the candidate is the expert (since they've already worked on it), and it almost becomes a session where the candidate teaches the interviewer.
- phibz 5y ago
  What matters most to me as a hiring manager is your ability to think critically. There's so much to know, and so many different permutations with how those things can present themselves that I'm not hiring you to know the answer every time. I'm hiring you to figure it out.

baby_wipe 5y ago

Making it cheaper for companies to interview increases the potential opportunities for me as a candidate. I do not take personal offense to being tested in an automated fashion.

phibz 5y ago
Have you taken a bunch of these and still gotten the same ghosting, empty promises, non communication and put in multiple hours of your time for free? I certainly have. The best interviews for me are where I can talk in depth about a topic and to a lesser extent, about a hypothetical problem and how I might handle it.
- datavirtue 5y ago
  I tried a few of these with different companies. Each one has failed due to technical difficulties of the platform (in my case prerecorded video interviews) and in every case I tried to file a ticket and never heard anything back. Black hole. In one case I did a prerecorded interview and they later contacted everyone because something went wrong and they wanted everyone to do it again. Done feeling like a number.

jahewson 5y ago

I would have no problem with this as a 45min screen after having talking with a hiring manager. I’m going to invest that time anyway and it’s nicer to not have to “do it live”. There does need to be humans for the later rounds though - because I’m assessing the company as much as they’re assessing me!

garyjlin 5y ago
Yup! That's why we recommend using Litebulb as a mid-funnel tool. Initial screening should be with a hiring manager (or a founder if it's an early stage startup), final meetings should be with team members/managers to get cultural fit. We just try to make the technical coding evaluation part easier.

garyjlin 5y ago

I completely get what you're saying. We've seen many candidates that have expressed the concern of "if I'm going to put X units of time into this, you should also put X units of time into this". A good approach of running a Litebulb interview is to offer to do it in a sit-down session over a call with the interviewer, or to do it asynchronously. When we did sit-down sessions, we offered candidates a choice to just go off and do it throughout the day, whenever they can find the time. 100% of candidates chose to do it async. As it turns out, trying to code something with an interviewer breathing down your neck wasn't a good experience and made them very nervous. Note that it's an option, not mandatory, to do it async. If the candidate wanted to do it in a sit down session, then we prioritize making time for it to be there the entire session. The reason we offer this option is because devs that have busy lives often can't easily find a single consecutive multi-hour session to do an interview. They have full time jobs, and might have family obligations at home and over the weekends. Basically, the only way to actually do a full onsite interview is to take a day off work to go do the interview. We're giving the flexibility for the candidate to find a chunk of time here, a chunk of time there to complete the interview. We tested this above approach, because just as you said, we did see a huge dropoff when we just emailed candidates the interview link. After we offered to hop on a call and to be available for support, we saw dropoff reduced to less than 15%. Also, a good chunk of candidates that drop out of interview processes are senior engineers that can't be bothered with lengthy processes, which is fair. I wouldn't give a Litebulb interview to a senior engineer to begin with. Debatably I wouldn't give any kind of technical interview, actually. If they have decades of experience leading teams at big or high-growth tech companies, those achievements speak for itself. The interview process for seniors should be more geared towards finding a product-team-eng fit, rather than an evaluation.

bryans 5y ago
> As it turns out, trying to code something with an interviewer breathing down your neck wasn't a good experience and made them very nervous. That is the fundamental issue with technical interviews, because there is very little about that manufactured situation which is applicable to the candidate's ability to do the work. There are extremely few real-world coding situations that come anywhere close to the pressures felt while having someone hyper-analyze everything you do in real-time, particularly when that someone is responsible for deciding whether you're able to pay bills next month or feed your family next week. It's a needless, pointless and ultimately self-harming methodology that filters out exceptional candidates in favor of the most extroverted or arrogantly confident, which is a bizarre twist considering the most influential and game-changing coders are also famously some of the most introverted and "unusual" people. On top of that, because we're all in dependency hell at this point, far more time is spent googling for framework errors that don't make any sense until you find that one undocumented command line argument that a single person posted deep inside a GitHub issue thread. If you want to test someone's ability to do the real work, then give them an obscure error code from a little-used utility and see how long it takes to find the fix.

brimble 5y ago

Can confirm, I bail if you ask any amount of time at all be put into a "weed out" exercise before even talking to a real person. If my resume's not enough to get you interested in a conversation, I'll look elsewhere. If you have that many candidates and you find enough of them equally interesting that doing this is necessary, I reckon my odds are terrible anyway, so it's just a waste of time doing work and not just not getting paid, but very likely not seeing any kind of reward whatsoever. Moreover, if this is necessary, I kinda assume the pay is mediocre at best. If you knew how to get enough value out of me to be able to pay me really, really well, and if I look like a good fit for the job, then again, my resume ought to justify our talking. If any of that's not the case, then it seems like the pay's gonna suck or I'm just not a good fit, so I'm out either way. I want someone who's fishing for me with a lure, not by dragging a huge net through the water. Seems like a better sign all around.

martinald 5y ago
I do understand that, but from my experience short coding tests give by far the best signal. We ran an experiment and found that actually not doing a pre interview and straight to code test gave us better results than interview then coding test. Why? Because people tend to hire people with qualities like themselves, and overlook many things. It also then caused people to overrate/underrate the analysis of coding tests based on the previous interview (as they (dis)liked the person). This also massively increased the diversity of our team.
rsyring 5y ago
As a developer, I understand this perspective. But, from an employer's perspective, resumes tell you very little about a candidate and are essentially worthless. The sole usable metric we get from them is years of relevant experience. We respond with a questionnaire that gets us more uniform answers about experience and technologies used. Assuming the above doesn't filter the candidate out, we move on to a 90 min max "take home" exercise. In our case, it's not a coding exercise, but that's not that important for this discussion. Only if all of the above hasn't filtered out the candidate, do we move to a Zoom meeting. In the Zoom meeting, after an initial introduction, we do a few easy coding exercises. Like, reverse a string or sum the integers in a file easy. A few notable things from this experience: 1. The number of candidates who struggle with the Zoom interview coding exercise is surprising. It's a lot higher than expected and almost all of our candidates at that stage have 2+ years of professional development experience. 2. The number of candidates filtered out by the "take home" exercise is very high. 85+% We literally don't have enough time to talk to every candidate that applies with the right amount of experience. And, even if we did, many/most candidates on the market aren't good developers. Our signal to noise ratio is already bad. In that case, it would be terrible. As a developer, I sympathize with your perspective. But you may be passing on some good opportunities by not being willing to "play the game" a bit. Some employers are indeed bad actors and/or terrible at hiring. But given the current market, even those making a valiant effort to do it well, due to sheer volume, are usually going to have an exercise of some kind. We post our salary ranges up-front, don't want to waste anyone's time that way. We also put a ton of effort into a very detailed job description and application process. The hope is that the candidate can see the effort we put in and will be willing to reciprocate. If that doesn't end up being the case, we assume, like you, it probably wouldn't have been a good for anyway. FWIW.

oscargrouch 5y ago

With COVID and the rise of remote working there are a whole lot of companies where the developers are the asset, the commodity, where those companies are just the middle-man receiving for the work the developers do and getting the big part of the cut in the process. Those are mostly the "sweat-shops" that hire developers that can work for less for several reasons. I see that in this particular case this solution might be very welcome to assess new hires. But for more high-profile jobs and companies, the company that eventually follow this path will probably end with low quality hires, unless of course it just create crud's anyway. Because this is a self-fulfilling prophecy in the end. Once the candidates know what they will face, they will train until they get good in that game, which most of the time doesn't mirror the qualities required for the job.

hkt 5y ago

I second this. What's worse is that being a devops guy, most coding interviews I've ever done are totally irrelevant to the roles I've had. In a good market I can avoid companies using services like this.

ivanmaeder 5y ago

I built a very similar product 10 years ago and even back then we heard feedback like this. And I believe the situation may be more pronounced today. There seem to be markets (or large enough organizations?) where coding job ads get a tonne of applicants, many of them inexperienced, and so automation is a good fit. But not when things are tipped the other way. Our bet was for real-world tests too but it wasn't enough. A few things we missed that might help… - Candidates don't want to be treated like cattle - For many companies a good interview platform will be more beneficial than automation - Companies say they care about the experience candidates receive, they'll say they're rigorous and try to be as objective as possible with the way they collate information and make decisions, they'll talk about how high their standards are, etc etc. Be weary - There may be more gains to be made outside of the tech space - The problem most companies complained about and is probably still the hardest: going out and finding people I like the way you've solved the "real-world challenge" problem. Good luck and all the best!