4 ms·

Great job on further eroding the trust from a prospective employer. Require a formal degree in CS? That's gatekeeping. Need to pass a whiteboard exam? Not rep

Great job on further eroding the trust from a prospective employer. Require a formal degree in CS? That's gatekeeping. Need to pass a whiteboard exam? Not representative of the actual work. Live coding session? Biased against people who don't perform well under pressure. Take home project? It's too much work to do for free. Showcase a personal portfolio? Not fair to people with families or other obligations. Either you enforce a minimal level of competency upfront in the form of an academic degree, industry-standard exams such as a PE Exam, etc. OR you push the entire responsibility of vetting prospective applicants downstream to the employer—which is exactly why interviews are multiple week long gauntlets. The tech world likes to complain about all of this but other occupations 100% DO have high standards - it's just that it's paid up-front. Want to become a lawyer? - You've got to pass the LSAT, get into law school, and pass the bar. Want to become a doctor? - You've got to pass the MCAT, get into medical school, and do residency. Want to become a pilot? - You've got to get your PPL, pass your check ride, do your IFR, multi-engine, commercial rating, ATP God there are some days that I ABSOLUTELY HATE THIS INDUSTRY.

thot_experiment 2y ago
If your hiring process is broken by people using AI then the process is either not evaluating the right things, or maybe it's fine because realistically speaking you're hiring a dev who's going to be using AI anyway.
- sgarland 2y ago
  I work in infra. I don’t want an incident to be the time that I find out my coworkers don’t actually know how anything works, and are frantically posting questions to ChatGPT. That said, I also don’t know how to adequately test, other than to carefully watch people’s eyes during questions; even that isn’t foolproof.
  - viraptor 2y ago
    "tell me about potential issues while doing X", then dig into answers? We may get there one day with LLMs, but currently it would be extremely hard to keep the conversation context and process the longer answers while talking to someone. The voice recognition quality and generating speed would be an issue too if you wanted to monitor the whole conversation. Currently the cheating mostly aims for coding tasks because they're pretty simple to do and self-contained. Much less for helping with a good dialog.
  - pockmarked19 2y ago
    Some of the best tests come from taking something mundane and twisting it just a little off the beaten path in a way that LLMs cannot handle. The usual example is the river crossing puzzle which has a standard format and solution but can be modified to make that solution incorrect. For code, an example would be parsing log lines to compile insights but switching it up so that the timestamp means something other than the time the log was collected, etc. You can completely change the complexity of a bog standard leetcode question with the slightest of modifications, and I’m certain no extant LLM will catch such a nuance if your problem is a very common one. A fun exercise is to ask chatGPT what the river crossing goat puzzle is and then say “do the same puzzle but with two goats and a cabbage please”. It can’t get it right even with multiple corrections. Might stop working at some point now that I posted this though!
- rich_sasha 2y ago
  Interview performance used to be correlated to job aptitude, but no longer.
- dannyeei 2y ago
  Unfortunately real life problems involve a lot of context and introducing that to interviews is hard. It either requires a long conversation where someone might lose track, or involve writing a large ish code base and getting them to work on that (which is a lot of work, and once again they don’t have context). In my experience if a problem simple enough to be completed in an hour, it’s simple enough for AI.
  - thot_experiment 2y ago
    IMO if your interview question has a right answer that's like, a basic early screening tool to see if the person is paying the slightest bit of attention. The sorts of questions that really give you useful insight into who someone is as an engineer are the ones that don't have a right answer, and can't be reasonably solved in an interview. If your question has an achievable correct answer you're clipping the gamut of info you can glean from it. I recently had a question where I was just given a bunch of tuples and told "let me know anything you can glean about this data in two hours". I think you learn much more by seeing how someone fails at a complex nebulous task than whether someone can implement a doubly linked list from memory or whatever. Real life programming is like 80% failing at complex nebulous tasks, picking yourself up and trying again (by volume). Interviews should simulate that. If AI helps you, so be it.
- rahimnathwani 2y ago
  Interviews seek to estimate long-term job performance. Since they cannot measure this directly, they measure attributes that correlate with performance. If you interfere with these measurements (e.g. through fraud), you break those correlations. This doesn't mean the interviews are measuring the wrong thing, or that people who cheat on the interviews will meet the performance bar in the actual job. If someone uses AI to defeat a CAPTCHA, does that mean the CAPTCHA was poorly designed because "the bot is just as valid a user as any human"? Of course not - the CAPTCHA's entire purpose was to establish that correlation between "can solve this" and "is human," and breaking that correlation through automated tools defeats its purpose entirely.
  - thot_experiment 2y ago
    First off, yes, actually that's exactly what it means. If bots can solve CAPTCHAS then the CAPTCHAS are poorly designed, that's the whole point. I don't think this sort of CAPTCHA type step should have a significant place in a good interview process in the first place, if it is important to you to put your applicants through this sort of gauntlet you're not treating them with the dignity they deserve. If you're just looking to fill cogs in your machine, sure, but in that case you're just as likely to get kids who studied a bunch of interview questions but also don't know shit. I'd just as soon have someone who's good at ChatGPT. Let me put it a different, perhaps simpler way: Let's agree on the premise that good human engineers are much better hires than good AI users. If your hiring process fails to distinguish between good human engineers and good AI users that's evidence that your process is not producing good data. In the exact same way that if your CAPTCHA is getting solved by bots it's a bad CAPTCHA.
    - rahimnathwani 2y ago
      Right, so the problem with the interview process isn't that it tests the wrong skills, but that it fails to prevent the secret use of AI such that a human can appear to possess those skills when they do not, in fact, possess them. It seems there are at least two different perspectives: A) If a human can use AI to pass a remote interview that they would otherwise fail, then the questions are unsuitable for use in interviews, whether remote or in person. B) If a human can use AI to pass a remote interview that they would fail if it were face to face, then the questions may be reasonable, but the safeguards for remote interviews are insufficient. I disagree with A.
- hilbert42 2y ago
  I find this all rather intriguing because to me it says more about the employers/interviewers than the applicants. Over the years I've interviewed many people for technical positions and there are many ways of getting to the core of a prospective employee's competence. Of course there are questions about training, past experience, qualifications and those about the work you expect them to perform but other matters also enter the equation and they're almost as important (the person's demeanor, confidence, etc.). By letting the person do most of the talking—explaining things about themselves and their work—you can learn a great deal. Stating the obvious, interviewers ought to be on top of the job they're hiring for but from this story and from other anecdotal info I know that's not always so. In my case as head of the department I was pretty much up to speed on most aspects of the work, so it was clear to me when an applicant was spinning me BS. If they knew more than I did then they almost certainly got the job (life's easier when you've smart employees, for starters one can ask them questions instead of vice versa). What's that leading up to? Well, when interviewing it was obvious to me (and to the other interviewers) within just the first few minutes whether the person was worth considering. It's uncanny how good one can become at doing this (and my record speaks for itself, I was pretty pleased with those employees I hired). Let me illustrate with an example: some decades ago I had an applicant apply for an electronics job. He spoke only Chinese and a few broken words in English and none of us on the interview panel spoke Chinese. BTW, he turned up with a Chinese/English dictionary in hand. I dispelled with many of the questions I usually asked and showed him a somewhat complex circuit diagram with many subsections that performed very different functions (if I showed this at all then it was usually towards the end of an interview). Within seconds of looking at the diagram he progressed logically through the processes from beginning to the end (note the processes/stages were not drawn in linear progression and consisted of many branches). He did this mainly by pointing. It was immediately obvious to both my independent technical assessor and me that this guy knew his subject very well. The only point was a minor objection from the nontechnical employment officer who questioned if we'd manage to communicate adequately with someone who didn't speak English. I didn't see that as a significant issue. He turned out to be an excellent employee who knew his work very well. And within six months his English was good enough not to be an issue. In summary, if AI is a problem now then it will be more so in the future and that needs to be tackled. That said, I find it hard to accept that an experienced interviewer can't quickly cut to the core even when an interviewee puts up smoke screens and distracting noise. Like, Popper's falsifiability it doesn't take much to knock a BS or inconsistent argument off its pedestal when one has enough data, refutability usually works well. That said, one must always keep in mind that just because an applicant can get some things wrong it doesn't mean that he/she isn't the best person for the job. All that tells me is that I have to probe a little deeper. Edit: asking awkward or trick questions usually ought to be avoided. An interviewee is usually already under considerable stress and can easily be flummoxed when asked them (even if they know the answers they can bind up and not spell them out). Usually, it's immediately obvious when an interviewee doesn't know an answer to a question or is struggling with it because of stress—body language alone usually makes that abundantly clear. It's then best to gently segue to another question. If the question is really relevant to the job then one can return to it at the end of the interview when the person is more at ease. By then, it's pretty clear to the panel whether the applicant is in the running or not. If not, then do not return to the question.
pockmarked19 2y ago
There’s an argument to be made that by the time your business runs out of vetted candidates to hire you’re already staffed at some of the highest levels by corporate parasites who got there by gaming the system, and you can tolerate adding more elsewhere.
BeFlatXIII 2y ago
The pushback is precisely because paying the high standards up front pulls up the ladder that made tech such an attractive option to people without traditional academic skills who managed to be computer whizzes. Take home projects and personal portfolios are the options that are the best. Those are the ones where candidates are given the time to show their best work and don't require the debt of a college degree in a specific major (whose knowledge is probably largely irrelevant to your day-to-day coding).
cleanerbob 2y ago
[dead]
jaredklewis 2y ago
I prefer our current system over the industry standards exam system you seem to be advocating for. It’s more flexible. Different companies care about different things. Different employees have different strengths. It all works out.
lhamil64 2y ago
I'm a little confused about your reasoning that take home projects are too much work to do for free. Aren't lawyers and doctors expected to spend a ton of time studying and taking the licensing exams without being paid (I assume even having to pay for it)? My gut feeling is that's more work than a take home project.
- fragmede 2y ago
  Licensing exam scores are usable for applying generally to all places. A take home I do for company A isn't useful in applying at company B.
deleted 2y ago
[deleted]
gopher_space 2y ago
You're complaining about a process you've implemented yourself. Have you tried hiring in a completely different manner? At what point do you question an approach so easily gamed, or any derived metric?
RadiozRadioz 2y ago
> Biased against people who don't perform well under pressure. I think performance under pressure is a virtue. An important job is never going to be entirely free from stress - you want somebody reliable who will take that in their stride, you don't want someone who will buckle when things get difficult. I'm all for in-person live collaborative problem solving, be it on a whiteboard or just a discussion. You see first-hand exactly what the person has in their memory, their problem solving techniques, how quickly they get things, their communication skills, and yes, their ability to work under pressure. If done correctly and collaboratively, you get a dry-run of a real design meeting and have some feeling of what it would be like to work with the person.
- rootsofallevil 2y ago
  > I think performance under pressure is a virtue. An important job is never going to be entirely free from stress - you want somebody reliable who will take that in their stride, you don't want someone who will buckle when things get difficult. Some people experience extraordinary high levels of stress in an interview setting. From your comment, I suspect that you are not one of them and might have trouble imagining what these levels of stress feel like.
  - RadiozRadioz 2y ago
    Fair enough. I don't want to be mean, but I wonder, if they are the type of person to experience this in interviews, what other situations would set them off in a similar way? It won't be everything, but I'll bet there's some comorbidity; there is a combination of stressors in an interview setting that triggers them, where else can these occur? From my experience, there is more overlap with people who generally let stress get to them, than people who are solely stressed out by interviews specifically and nothing else. I have a relative like this. They struggle in interviews due to stress, they also struggle in exams due to stress. That's two things, I don't want to be surprised by the third at a critical and stressful time in my team.
kazinator 2y ago
> Require a formal degree in CS? That's gatekeeping. Selecting people for a job is literally gatekeeping. The company has the equivalent of a gate (or perhaps an actual one), and most people are kept out of it by the fact that they are not employees or authorized visitors. No part of would-be employee screening can be faulted for being gatekeeping; i.e. being that which it is.
medicore_duck 2y ago
Shut the fuck up. We'll stop cheating when these corps stop making us solve binary search algorithms to interview for front end React roles. Probably 80% of all technical interviews I've had were completely irrelevant to the actual daily work. They engineered this shitty game, we're just playing the game. No one's making you play this game.
aguaviva 2y ago
Live coding session? Biased against people who don't perform well under pressure. No, it's biased against people who perform perfectly well (sometimes exceptionally well) under real, actual pressure. But who do not take kindly to the bullshit / theatrical pressure typical of the standard interview context, these days. And to companies that just can't tell the difference (or are under the delusion that performance in the latter category is in some way predictive of performance in the former). Take home project? It's too much work to do for free. It's more that the companies simply abuse the process in various callous and careless ways -- either by assigning projects with inadequate specification and/or unknown goalposts (or plainly unrealistic expectations of the actual time required, given these ambiguities); or simply assigning scores of these assignments (often automatically to every candidate who applies) when in fact they have no intention of even looking at the vast majority of the submissions. Easily ameliorated by simply paying people for their time.

thot_experiment 2y ago

If your hiring process is broken by people using AI then the process is either not evaluating the right things, or maybe it's fine because realistically speaking you're hiring a dev who's going to be using AI anyway.

sgarland 2y ago
I work in infra. I don’t want an incident to be the time that I find out my coworkers don’t actually know how anything works, and are frantically posting questions to ChatGPT. That said, I also don’t know how to adequately test, other than to carefully watch people’s eyes during questions; even that isn’t foolproof.
- viraptor 2y ago
  "tell me about potential issues while doing X", then dig into answers? We may get there one day with LLMs, but currently it would be extremely hard to keep the conversation context and process the longer answers while talking to someone. The voice recognition quality and generating speed would be an issue too if you wanted to monitor the whole conversation. Currently the cheating mostly aims for coding tasks because they're pretty simple to do and self-contained. Much less for helping with a good dialog.
- pockmarked19 2y ago
  Some of the best tests come from taking something mundane and twisting it just a little off the beaten path in a way that LLMs cannot handle. The usual example is the river crossing puzzle which has a standard format and solution but can be modified to make that solution incorrect. For code, an example would be parsing log lines to compile insights but switching it up so that the timestamp means something other than the time the log was collected, etc. You can completely change the complexity of a bog standard leetcode question with the slightest of modifications, and I’m certain no extant LLM will catch such a nuance if your problem is a very common one. A fun exercise is to ask chatGPT what the river crossing goat puzzle is and then say “do the same puzzle but with two goats and a cabbage please”. It can’t get it right even with multiple corrections. Might stop working at some point now that I posted this though!
rich_sasha 2y ago
Interview performance used to be correlated to job aptitude, but no longer.
dannyeei 2y ago
Unfortunately real life problems involve a lot of context and introducing that to interviews is hard. It either requires a long conversation where someone might lose track, or involve writing a large ish code base and getting them to work on that (which is a lot of work, and once again they don’t have context). In my experience if a problem simple enough to be completed in an hour, it’s simple enough for AI.
- thot_experiment 2y ago
  IMO if your interview question has a right answer that's like, a basic early screening tool to see if the person is paying the slightest bit of attention. The sorts of questions that really give you useful insight into who someone is as an engineer are the ones that don't have a right answer, and can't be reasonably solved in an interview. If your question has an achievable correct answer you're clipping the gamut of info you can glean from it. I recently had a question where I was just given a bunch of tuples and told "let me know anything you can glean about this data in two hours". I think you learn much more by seeing how someone fails at a complex nebulous task than whether someone can implement a doubly linked list from memory or whatever. Real life programming is like 80% failing at complex nebulous tasks, picking yourself up and trying again (by volume). Interviews should simulate that. If AI helps you, so be it.
rahimnathwani 2y ago
Interviews seek to estimate long-term job performance. Since they cannot measure this directly, they measure attributes that correlate with performance. If you interfere with these measurements (e.g. through fraud), you break those correlations. This doesn't mean the interviews are measuring the wrong thing, or that people who cheat on the interviews will meet the performance bar in the actual job. If someone uses AI to defeat a CAPTCHA, does that mean the CAPTCHA was poorly designed because "the bot is just as valid a user as any human"? Of course not - the CAPTCHA's entire purpose was to establish that correlation between "can solve this" and "is human," and breaking that correlation through automated tools defeats its purpose entirely.
- thot_experiment 2y ago
  First off, yes, actually that's exactly what it means. If bots can solve CAPTCHAS then the CAPTCHAS are poorly designed, that's the whole point. I don't think this sort of CAPTCHA type step should have a significant place in a good interview process in the first place, if it is important to you to put your applicants through this sort of gauntlet you're not treating them with the dignity they deserve. If you're just looking to fill cogs in your machine, sure, but in that case you're just as likely to get kids who studied a bunch of interview questions but also don't know shit. I'd just as soon have someone who's good at ChatGPT. Let me put it a different, perhaps simpler way: Let's agree on the premise that good human engineers are much better hires than good AI users. If your hiring process fails to distinguish between good human engineers and good AI users that's evidence that your process is not producing good data. In the exact same way that if your CAPTCHA is getting solved by bots it's a bad CAPTCHA.
  - rahimnathwani 2y ago
    Right, so the problem with the interview process isn't that it tests the wrong skills, but that it fails to prevent the secret use of AI such that a human can appear to possess those skills when they do not, in fact, possess them. It seems there are at least two different perspectives: A) If a human can use AI to pass a remote interview that they would otherwise fail, then the questions are unsuitable for use in interviews, whether remote or in person. B) If a human can use AI to pass a remote interview that they would fail if it were face to face, then the questions may be reasonable, but the safeguards for remote interviews are insufficient. I disagree with A.
hilbert42 2y ago
I find this all rather intriguing because to me it says more about the employers/interviewers than the applicants. Over the years I've interviewed many people for technical positions and there are many ways of getting to the core of a prospective employee's competence. Of course there are questions about training, past experience, qualifications and those about the work you expect them to perform but other matters also enter the equation and they're almost as important (the person's demeanor, confidence, etc.). By letting the person do most of the talking—explaining things about themselves and their work—you can learn a great deal. Stating the obvious, interviewers ought to be on top of the job they're hiring for but from this story and from other anecdotal info I know that's not always so. In my case as head of the department I was pretty much up to speed on most aspects of the work, so it was clear to me when an applicant was spinning me BS. If they knew more than I did then they almost certainly got the job (life's easier when you've smart employees, for starters one can ask them questions instead of vice versa). What's that leading up to? Well, when interviewing it was obvious to me (and to the other interviewers) within just the first few minutes whether the person was worth considering. It's uncanny how good one can become at doing this (and my record speaks for itself, I was pretty pleased with those employees I hired). Let me illustrate with an example: some decades ago I had an applicant apply for an electronics job. He spoke only Chinese and a few broken words in English and none of us on the interview panel spoke Chinese. BTW, he turned up with a Chinese/English dictionary in hand. I dispelled with many of the questions I usually asked and showed him a somewhat complex circuit diagram with many subsections that performed very different functions (if I showed this at all then it was usually towards the end of an interview). Within seconds of looking at the diagram he progressed logically through the processes from beginning to the end (note the processes/stages were not drawn in linear progression and consisted of many branches). He did this mainly by pointing. It was immediately obvious to both my independent technical assessor and me that this guy knew his subject very well. The only point was a minor objection from the nontechnical employment officer who questioned if we'd manage to communicate adequately with someone who didn't speak English. I didn't see that as a significant issue. He turned out to be an excellent employee who knew his work very well. And within six months his English was good enough not to be an issue. In summary, if AI is a problem now then it will be more so in the future and that needs to be tackled. That said, I find it hard to accept that an experienced interviewer can't quickly cut to the core even when an interviewee puts up smoke screens and distracting noise. Like, Popper's falsifiability it doesn't take much to knock a BS or inconsistent argument off its pedestal when one has enough data, refutability usually works well. That said, one must always keep in mind that just because an applicant can get some things wrong it doesn't mean that he/she isn't the best person for the job. All that tells me is that I have to probe a little deeper. Edit: asking awkward or trick questions usually ought to be avoided. An interviewee is usually already under considerable stress and can easily be flummoxed when asked them (even if they know the answers they can bind up and not spell them out). Usually, it's immediately obvious when an interviewee doesn't know an answer to a question or is struggling with it because of stress—body language alone usually makes that abundantly clear. It's then best to gently segue to another question. If the question is really relevant to the job then one can return to it at the end of the interview when the person is more at ease. By then, it's pretty clear to the panel whether the applicant is in the running or not. If not, then do not return to the question.

pockmarked19 2y ago

There’s an argument to be made that by the time your business runs out of vetted candidates to hire you’re already staffed at some of the highest levels by corporate parasites who got there by gaming the system, and you can tolerate adding more elsewhere.

BeFlatXIII 2y ago

The pushback is precisely because paying the high standards up front pulls up the ladder that made tech such an attractive option to people without traditional academic skills who managed to be computer whizzes. Take home projects and personal portfolios are the options that are the best. Those are the ones where candidates are given the time to show their best work and don't require the debt of a college degree in a specific major (whose knowledge is probably largely irrelevant to your day-to-day coding).

cleanerbob 2y ago

[dead]

jaredklewis 2y ago

I prefer our current system over the industry standards exam system you seem to be advocating for. It’s more flexible. Different companies care about different things. Different employees have different strengths. It all works out.

lhamil64 2y ago

I'm a little confused about your reasoning that take home projects are too much work to do for free. Aren't lawyers and doctors expected to spend a ton of time studying and taking the licensing exams without being paid (I assume even having to pay for it)? My gut feeling is that's more work than a take home project.

fragmede 2y ago
Licensing exam scores are usable for applying generally to all places. A take home I do for company A isn't useful in applying at company B.

deleted 2y ago

[deleted]

gopher_space 2y ago

You're complaining about a process you've implemented yourself. Have you tried hiring in a completely different manner? At what point do you question an approach so easily gamed, or any derived metric?

RadiozRadioz 2y ago

> Biased against people who don't perform well under pressure. I think performance under pressure is a virtue. An important job is never going to be entirely free from stress - you want somebody reliable who will take that in their stride, you don't want someone who will buckle when things get difficult. I'm all for in-person live collaborative problem solving, be it on a whiteboard or just a discussion. You see first-hand exactly what the person has in their memory, their problem solving techniques, how quickly they get things, their communication skills, and yes, their ability to work under pressure. If done correctly and collaboratively, you get a dry-run of a real design meeting and have some feeling of what it would be like to work with the person.

rootsofallevil 2y ago
> I think performance under pressure is a virtue. An important job is never going to be entirely free from stress - you want somebody reliable who will take that in their stride, you don't want someone who will buckle when things get difficult. Some people experience extraordinary high levels of stress in an interview setting. From your comment, I suspect that you are not one of them and might have trouble imagining what these levels of stress feel like.
- RadiozRadioz 2y ago
  Fair enough. I don't want to be mean, but I wonder, if they are the type of person to experience this in interviews, what other situations would set them off in a similar way? It won't be everything, but I'll bet there's some comorbidity; there is a combination of stressors in an interview setting that triggers them, where else can these occur? From my experience, there is more overlap with people who generally let stress get to them, than people who are solely stressed out by interviews specifically and nothing else. I have a relative like this. They struggle in interviews due to stress, they also struggle in exams due to stress. That's two things, I don't want to be surprised by the third at a critical and stressful time in my team.

kazinator 2y ago

> Require a formal degree in CS? That's gatekeeping. Selecting people for a job is literally gatekeeping. The company has the equivalent of a gate (or perhaps an actual one), and most people are kept out of it by the fact that they are not employees or authorized visitors. No part of would-be employee screening can be faulted for being gatekeeping; i.e. being that which it is.

medicore_duck 2y ago

Shut the fuck up. We'll stop cheating when these corps stop making us solve binary search algorithms to interview for front end React roles. Probably 80% of all technical interviews I've had were completely irrelevant to the actual daily work. They engineered this shitty game, we're just playing the game. No one's making you play this game.

aguaviva 2y ago

Live coding session? Biased against people who don't perform well under pressure. No, it's biased against people who perform perfectly well (sometimes exceptionally well) under real, actual pressure. But who do not take kindly to the bullshit / theatrical pressure typical of the standard interview context, these days. And to companies that just can't tell the difference (or are under the delusion that performance in the latter category is in some way predictive of performance in the former). Take home project? It's too much work to do for free. It's more that the companies simply abuse the process in various callous and careless ways -- either by assigning projects with inadequate specification and/or unknown goalposts (or plainly unrealistic expectations of the actual time required, given these ambiguities); or simply assigning scores of these assignments (often automatically to every candidate who applies) when in fact they have no intention of even looking at the vast majority of the submissions. Easily ameliorated by simply paying people for their time.