8 ms·

Company A wants to hire an engineer, an AI could solve all their tech interview questions, so why not hire that AI instead? There's very likely a real answer t

Company A wants to hire an engineer, an AI could solve all their tech interview questions, so why not hire that AI instead? There's very likely a real answer to that question, and that answer should shape the way that engineer should be assessed and hired. For example, it could be that the company wants the engineer to do some kind of assessment whether a feature should be implemented at all, and if yes, in what way. Then you could, in an interview, give a bit of context and then ask the candidate to think out loud about an example feature request. It seems to me the heart of the problem is that companies aren't very clear about what value the engineers add, and so they have trouble deciding whether a candidate could provide that value.

bitwizeshift 2y ago
Tech interviews in general need to be overhauled, and if they were it’d be less likely that AI would be as helpful in the process to begin with (at least for LLMs in their current state). Current LLMs can do some basic coding and stitch it together to form cool programs, but it struggles at good design work that scales. Design-focused interviews paired with soft-skill-focus is a better measure of how a dev will be in the workplace in general. Yet, most interviews are just “if you can solve this esoteric problem we don’t use at all at work, you are hired”. I’d take a bad solution with a good design over a good solution with a bad design any day, because the former is always easier to refactor and iterate on. AI is not really good at that yet; it’s trained on a lot of public data that skews towards worse designs. It’s also not all that great at behaving like a human during code reviews; it agrees too much, is overly verbose, it hallucinates, etc.
nottorp 2y ago
Problem is, company A doesn't need an engineer to solve those interview questions but real problems.
- okdood64 2y ago
  This is the answer. Let's not pretend 95% of companies are asking asinine interview questions (though I understand the reasons why) that LLMs can easily solve.
  - nottorp 2y ago
    Let's go one step further: LLMs can't solve anything, but most interview questions are covered so much online that they'll parrot a passable answer.
    - Jensson 2y ago
      Yeah, what you want is a General Intelligence that has learned the topic you care about. Google search returning an algorithm when you ask it doesn't mean that you shouldn't test candidates on that algorithm, since you still need a General Intelligence that knows it and not just the algorithm itself.
- placardloop 2y ago
  “Real problems” aren’t something that can be effectively discussed in the time span of an interview, so companies concoct unreal problems that are meant to be good indicators.
  - 0_____0 2y ago
    Really? How short are your interviews, and how big are these Real Problems such that you can't get a sense of how your candidate would start to tackle them?
    - placardloop 2y ago
      The “real problems” most companies want people to help solve involve the evolution of products that last for years, involve repeated design discussions, in depth research, and applying retrospective learning. I don’t need someone that can just glue a Rails API together. If I did, I can literally just download that from the internet for free. If my problems could be solved in the time span of an interview, why would I waste my time doing that interview instead of just solving it?
      - 0_____0 2y ago
        I don't see the issue here. Nobody expects candidates to build actual product during the interview. Having a (targeted, scope and time-limited) design discussion or giving your candidate some made-up context around an engineering cycle and then doing a retrospective with them are practical and useful ways to interview a candidate. I'm also not sure what the alternative is? Just not hiring?
        placardloop 2y ago
        > Having a (targeted, scope and time-limited) design discussion or giving your candidate some made-up context around an engineering cycle and then doing a retrospective with them You just described a contrived, “unreal” problem. > I'm also not sure what the alternative is? Just not hiring? The alternative is to come up with questions that are representative of skills related to “real problems”, as you just did, and use those instead. Unfortunately candidates consistently complain that such questions aren’t realistic.
juujian 2y ago
The even bigger challenge is that hiring experts in any domain requires domain knowledge, but hiring has been shifted to HR. They aren't experts in anything, and for some years they made do with formulaic approaches, but that doesn't cut it anymore. So now if your group wants to get it done, and done well, you have to get involved yourself, and it's a lot of work on top of your regular tasks. Maybe more work because HR is deeply involved.
- ctkhn 2y ago
  I saw this at the big corporate (not faang/tech) place I work at. Engineers run and score interviews, but we don't make the final decision. That goes to HR and the hiring manager who usually has no technically background.
  - moodyredtimes 2y ago
    Yup, I have seen some really poor decisions as a result of this. I'm also curious - what will be the effect of AI assistance during behavior interviews, etc.
- ghaff 2y ago
  >hiring has been shifted to HR Well, unless you know sufficiently senior people. But I suspect that is a deeply unsatisfactory answer to many people in this forum. My long term last, only technically-adjacent, job came through a combination of knowing execs, having gone to the same school as my ultimate manager, and knowing various other people involved. (And having a portfolio of public work.)
  - tempodox 2y ago
    Personal networks only disadvantage those who have none.
    - ghaff 2y ago
      I suspect many people who don't have strong networks for whatever reason resent that. To which you could probably tack on not having gone to the "right" schools or having a public portfolio.
johnrealtoast 2y ago
[dead]
dahart 2y ago
This is a great point. Though what if the answer is that the company can hire that AI to solve a significant fraction of its actual problems? People who do the assessments and decide what features should look like are often called managers (product, engineering, etc.). For a while I’ve been skeptical that the rate of hiring of engineers would change significantly because of LLMs, but I’m starting to feel like maybe I’m wrong and it’s already changing and companies are looking toward AI to lower costs and require fewer humans. In that case they are probably still going to want people who are technically exceptional - maybe even more so - but are able and willing to create, integrate, and babysit AI generated code, and also do PM and EM style feature management. If companies are slowing hiring due to AI, I would expect interviews to get worse before they get better.
siva7 2y ago
> For example, it could be that the company wants the engineer to do some kind of assessment whether a feature should be implemented at all, and if yes, in what way. Then you could, in an interview, give a bit of context and then ask the candidate to think out loud about an example feature request. So a Product Manager?
- perlgeek 2y ago
  Maybe. Maybe now, or maybe in a year or two, AI coding tools will be good enough that a single semi-technical person can be Product Manager for a small product, and implement all the feature through AI/LLM tools. Probably not for something of the complexity of Google Maps, but for a simpler website with some interactive elements, that could work. But then, this was just an example. There can be lots of reasons that companies still need engineers, my point was that they need to think about these reasons, and then use these reasons to decide how to select their engineers.
- camdenreslink 2y ago
  In most companies every engineer above a junior level is expected to pass features and bugfixes through their common sense filter and provide feedback. Product managers and designers aren't infallible and sometimes lack knowledge about the system or product that an engineer might have. You can't just take requirements and churn out code without a critical eye at what you're doing.
michaelt 2y ago
> Company A wants to hire an engineer, an AI could solve all their tech interview questions, so why not hire that AI instead? Interview coding questions aren't like the day-to-day job, because of the nature of an interview. In an hour-long interview, I have to be able to state the problem in a way the candidate can understand, within 10 minutes or so. We don't have time for a lecture on the intricacies of voucher calculation and global sales tax law. It also has to be a problem that's solvable within about 40 minutes. The problem needs to test the candidate meets the company's hiring bar - while also having enough nuance that there's an opportunity for absolutely great candidates to impress me. And the problem has to be possible to state unambiguously. Can't have a candidate solving the problem, but failing the interview because there was a secret requirement and they failed to read my mind. And of course, if we're doing it in person on a whiteboard (do people do that these days?) it has to be solvable without any reference to documentation.
- janoc 2y ago
  >Interview coding questions aren't like the day-to-day job, because of the nature of an interview. You have missed his point. If the interview questions are such that AI can solve them, they are the wrong questions being asked, by definition. Unless that company is trying to hire a robot, of course.
  - deleted 2y ago
    [deleted]
  - zeroonetwothree 2y ago
    That’s silly. It’s like saying that a car can do a 100m dash faster than a human so it’s not a useful test for selecting players for an NFL team.
- gopher_space 2y ago
  > In an hour-long interview, I have to be able to state the problem in a way the candidate can understand, within 10 minutes or so. We don't have time for a lecture on the intricacies of voucher calculation and global sales tax law. If you send me a rubric I can pre-load whatever you want to talk about. If you tell me what you're trying to build and what you need help with, I can show up with a game plan. You need to make time for a conversation on the intricacies of voucher calculation and global sales tax law if you want to find people jazzed about the problem space.
diob 2y ago
It's because coding interview questions aren't so much assessing job skills as much as they are thinly veiled IQ tests. I think if it was socially acceptable they'd just do the latter.
- vasco 2y ago
  A lot of companies have IQ like tests, in particular big consulting companies like McKinsey and so on.
  - frankfrank13 2y ago
    McK's case interview is just as game-able as HackerRank style interviews. There are entire consulting clubs at many colleges that teach this exact interview style. It's true that it's harder (but not impossible) to use AI to help, but calling it an IQ-like test is true only as much as any other technical interview. That being said, McK did create an entire game that they claim can't be studied for ahead of time. If the intention is to test true problem solving skills, then maybe that's roughly equivalent to a systems interview, which is hard(er) to cheat .
    - SJC_Hacker 2y ago
      > That being said, McK did create an entire game that they claim can't be studied for ahead of time. If the intention is to test true problem solving skills, then maybe that's roughly equivalent to a systems interview, which is hard(er) to cheat . Sure, right up until someone leaks it
      - frankfrank13 2y ago
        Maybe, but they have hundreds! And new ones every quarter! Having done both systems and case interviews I can say that leaking is not as big a problem as with coding interviews. Sure you can memorize how to build Netflix/Zoom, or how to analyze P&L for a sandwich shop, but those interviews depend on interviewers throwing in complications or asking clarifying questions.
    - 2y ago
lanstin 2y ago
I want to hire people who can be given some problem and will go off and work on it and come to me with questions when specs are unclear or there's some weird thing that cropped up. AI is 100% not that. You have to watch it like a 15 year old driver.
Imnimo 2y ago
A company wants to hire someone to perform tasks X, Y and Z. It's difficult to cleanly evaluate someone's ability to do these tasks in a short amount of time, so they do their best to construct a task A which is easy to test, and such that most people who can do A can also do X, Y and Z. Now someone comes along and builds a machine that can do A. It turns out that while for humans, A was a good indicator of X, Y and Z, for the machine it is not. A is easy for the machine, but X, Y and Z are still difficult. This isn't a sign that the company was wrong to ask A, nor is it a sign that they could just hire the machine.
rurp 2y ago
One of the best interviews I've encountered as a candidate wasn't exactly a pair programming session but it was similar. The interviewer pulled up a webpage of theirs and showed me a problem with it, and then asked how I would approach fixing it. We worked our way through many parts of their stack and while it was me driving most of the way we ended up having a number of interesting conversations that cropped up organically at various points. It was scheduled for an hour and the time actually flew by. I felt like I got a good sense of what he would be like to work with and he got to see how I approached various problems. It avoided the live coding problems of needing to remember a bunch of syntax trivia on the spot and having to focus on a quick small solution, rather than a large scalable one that you need more often for actual work problems.

bitwizeshift 2y ago

Tech interviews in general need to be overhauled, and if they were it’d be less likely that AI would be as helpful in the process to begin with (at least for LLMs in their current state). Current LLMs can do some basic coding and stitch it together to form cool programs, but it struggles at good design work that scales. Design-focused interviews paired with soft-skill-focus is a better measure of how a dev will be in the workplace in general. Yet, most interviews are just “if you can solve this esoteric problem we don’t use at all at work, you are hired”. I’d take a bad solution with a good design over a good solution with a bad design any day, because the former is always easier to refactor and iterate on. AI is not really good at that yet; it’s trained on a lot of public data that skews towards worse designs. It’s also not all that great at behaving like a human during code reviews; it agrees too much, is overly verbose, it hallucinates, etc.

nottorp 2y ago

Problem is, company A doesn't need an engineer to solve those interview questions but real problems.

okdood64 2y ago
This is the answer. Let's not pretend 95% of companies are asking asinine interview questions (though I understand the reasons why) that LLMs can easily solve.
- nottorp 2y ago
  Let's go one step further: LLMs can't solve anything, but most interview questions are covered so much online that they'll parrot a passable answer.
  - Jensson 2y ago
    Yeah, what you want is a General Intelligence that has learned the topic you care about. Google search returning an algorithm when you ask it doesn't mean that you shouldn't test candidates on that algorithm, since you still need a General Intelligence that knows it and not just the algorithm itself.
placardloop 2y ago
“Real problems” aren’t something that can be effectively discussed in the time span of an interview, so companies concoct unreal problems that are meant to be good indicators.
- 0_____0 2y ago
  Really? How short are your interviews, and how big are these Real Problems such that you can't get a sense of how your candidate would start to tackle them?
  - placardloop 2y ago
    The “real problems” most companies want people to help solve involve the evolution of products that last for years, involve repeated design discussions, in depth research, and applying retrospective learning. I don’t need someone that can just glue a Rails API together. If I did, I can literally just download that from the internet for free. If my problems could be solved in the time span of an interview, why would I waste my time doing that interview instead of just solving it?
    - 0_____0 2y ago
      I don't see the issue here. Nobody expects candidates to build actual product during the interview. Having a (targeted, scope and time-limited) design discussion or giving your candidate some made-up context around an engineering cycle and then doing a retrospective with them are practical and useful ways to interview a candidate. I'm also not sure what the alternative is? Just not hiring?
      - placardloop 2y ago
        > Having a (targeted, scope and time-limited) design discussion or giving your candidate some made-up context around an engineering cycle and then doing a retrospective with them You just described a contrived, “unreal” problem. > I'm also not sure what the alternative is? Just not hiring? The alternative is to come up with questions that are representative of skills related to “real problems”, as you just did, and use those instead. Unfortunately candidates consistently complain that such questions aren’t realistic.

juujian 2y ago

The even bigger challenge is that hiring experts in any domain requires domain knowledge, but hiring has been shifted to HR. They aren't experts in anything, and for some years they made do with formulaic approaches, but that doesn't cut it anymore. So now if your group wants to get it done, and done well, you have to get involved yourself, and it's a lot of work on top of your regular tasks. Maybe more work because HR is deeply involved.

ctkhn 2y ago
I saw this at the big corporate (not faang/tech) place I work at. Engineers run and score interviews, but we don't make the final decision. That goes to HR and the hiring manager who usually has no technically background.
- moodyredtimes 2y ago
  Yup, I have seen some really poor decisions as a result of this. I'm also curious - what will be the effect of AI assistance during behavior interviews, etc.
ghaff 2y ago
>hiring has been shifted to HR Well, unless you know sufficiently senior people. But I suspect that is a deeply unsatisfactory answer to many people in this forum. My long term last, only technically-adjacent, job came through a combination of knowing execs, having gone to the same school as my ultimate manager, and knowing various other people involved. (And having a portfolio of public work.)
- tempodox 2y ago
  Personal networks only disadvantage those who have none.
  - ghaff 2y ago
    I suspect many people who don't have strong networks for whatever reason resent that. To which you could probably tack on not having gone to the "right" schools or having a public portfolio.

johnrealtoast 2y ago

[dead]

dahart 2y ago

This is a great point. Though what if the answer is that the company can hire that AI to solve a significant fraction of its actual problems? People who do the assessments and decide what features should look like are often called managers (product, engineering, etc.). For a while I’ve been skeptical that the rate of hiring of engineers would change significantly because of LLMs, but I’m starting to feel like maybe I’m wrong and it’s already changing and companies are looking toward AI to lower costs and require fewer humans. In that case they are probably still going to want people who are technically exceptional - maybe even more so - but are able and willing to create, integrate, and babysit AI generated code, and also do PM and EM style feature management. If companies are slowing hiring due to AI, I would expect interviews to get worse before they get better.

siva7 2y ago

> For example, it could be that the company wants the engineer to do some kind of assessment whether a feature should be implemented at all, and if yes, in what way. Then you could, in an interview, give a bit of context and then ask the candidate to think out loud about an example feature request. So a Product Manager?

perlgeek 2y ago
Maybe. Maybe now, or maybe in a year or two, AI coding tools will be good enough that a single semi-technical person can be Product Manager for a small product, and implement all the feature through AI/LLM tools. Probably not for something of the complexity of Google Maps, but for a simpler website with some interactive elements, that could work. But then, this was just an example. There can be lots of reasons that companies still need engineers, my point was that they need to think about these reasons, and then use these reasons to decide how to select their engineers.
camdenreslink 2y ago
In most companies every engineer above a junior level is expected to pass features and bugfixes through their common sense filter and provide feedback. Product managers and designers aren't infallible and sometimes lack knowledge about the system or product that an engineer might have. You can't just take requirements and churn out code without a critical eye at what you're doing.

michaelt 2y ago

> Company A wants to hire an engineer, an AI could solve all their tech interview questions, so why not hire that AI instead? Interview coding questions aren't like the day-to-day job, because of the nature of an interview. In an hour-long interview, I have to be able to state the problem in a way the candidate can understand, within 10 minutes or so. We don't have time for a lecture on the intricacies of voucher calculation and global sales tax law. It also has to be a problem that's solvable within about 40 minutes. The problem needs to test the candidate meets the company's hiring bar - while also having enough nuance that there's an opportunity for absolutely great candidates to impress me. And the problem has to be possible to state unambiguously. Can't have a candidate solving the problem, but failing the interview because there was a secret requirement and they failed to read my mind. And of course, if we're doing it in person on a whiteboard (do people do that these days?) it has to be solvable without any reference to documentation.

janoc 2y ago
>Interview coding questions aren't like the day-to-day job, because of the nature of an interview. You have missed his point. If the interview questions are such that AI can solve them, they are the wrong questions being asked, by definition. Unless that company is trying to hire a robot, of course.
- deleted 2y ago
  [deleted]
- zeroonetwothree 2y ago
  That’s silly. It’s like saying that a car can do a 100m dash faster than a human so it’s not a useful test for selecting players for an NFL team.
gopher_space 2y ago
> In an hour-long interview, I have to be able to state the problem in a way the candidate can understand, within 10 minutes or so. We don't have time for a lecture on the intricacies of voucher calculation and global sales tax law. If you send me a rubric I can pre-load whatever you want to talk about. If you tell me what you're trying to build and what you need help with, I can show up with a game plan. You need to make time for a conversation on the intricacies of voucher calculation and global sales tax law if you want to find people jazzed about the problem space.

diob 2y ago

It's because coding interview questions aren't so much assessing job skills as much as they are thinly veiled IQ tests. I think if it was socially acceptable they'd just do the latter.

vasco 2y ago
A lot of companies have IQ like tests, in particular big consulting companies like McKinsey and so on.
- frankfrank13 2y ago
  McK's case interview is just as game-able as HackerRank style interviews. There are entire consulting clubs at many colleges that teach this exact interview style. It's true that it's harder (but not impossible) to use AI to help, but calling it an IQ-like test is true only as much as any other technical interview. That being said, McK did create an entire game that they claim can't be studied for ahead of time. If the intention is to test true problem solving skills, then maybe that's roughly equivalent to a systems interview, which is hard(er) to cheat .
  - SJC_Hacker 2y ago
    > That being said, McK did create an entire game that they claim can't be studied for ahead of time. If the intention is to test true problem solving skills, then maybe that's roughly equivalent to a systems interview, which is hard(er) to cheat . Sure, right up until someone leaks it
    - frankfrank13 2y ago
      Maybe, but they have hundreds! And new ones every quarter! Having done both systems and case interviews I can say that leaking is not as big a problem as with coding interviews. Sure you can memorize how to build Netflix/Zoom, or how to analyze P&L for a sandwich shop, but those interviews depend on interviewers throwing in complications or asking clarifying questions.
  - 2y ago

lanstin 2y ago

I want to hire people who can be given some problem and will go off and work on it and come to me with questions when specs are unclear or there's some weird thing that cropped up. AI is 100% not that. You have to watch it like a 15 year old driver.

Imnimo 2y ago

A company wants to hire someone to perform tasks X, Y and Z. It's difficult to cleanly evaluate someone's ability to do these tasks in a short amount of time, so they do their best to construct a task A which is easy to test, and such that most people who can do A can also do X, Y and Z. Now someone comes along and builds a machine that can do A. It turns out that while for humans, A was a good indicator of X, Y and Z, for the machine it is not. A is easy for the machine, but X, Y and Z are still difficult. This isn't a sign that the company was wrong to ask A, nor is it a sign that they could just hire the machine.

rurp 2y ago

One of the best interviews I've encountered as a candidate wasn't exactly a pair programming session but it was similar. The interviewer pulled up a webpage of theirs and showed me a problem with it, and then asked how I would approach fixing it. We worked our way through many parts of their stack and while it was me driving most of the way we ended up having a number of interesting conversations that cropped up organically at various points. It was scheduled for an hour and the time actually flew by. I felt like I got a good sense of what he would be like to work with and he got to see how I approached various problems. It avoided the live coding problems of needing to remember a bunch of syntax trivia on the spot and having to focus on a quick small solution, rather than a large scalable one that you need more often for actual work problems.