4 ms·

Meta Is Going to Let Job Candidates Use AI During Coding Tests

55 points by geox 1y ago

joshstrange 1y ago
I'm currently running the interview process for the company I work for and I allow AI on the coding tests for the same reason I allow googling/etc: I want to simulate as close to the real environment you will be coding in if you come and work here. My "rules" on AI/LLMs at work is that you can use whatever tools you want (much like SO before) but any code you commit, you are responsible for (also unchanged from before LLMs). I never want to hear "the LLM wrote that" when asked about how something works. If you can explain, modify, build on top of the code that an LLM spits out then fine by me. We don't require LLM usage (no token quotas or silly things like that) nor do we disallow LLMs. At the end of the day they are all tools. I've always found leet coding, whiteboarding, memorizing algorithms, etc to just be a silly form of hazing and a bad indicator of how well the employee will perform if you hire them. In the same way I still think my college was stupid for making us write, on paper, a C program with all the correct headers/syntax/etc for an exam or how I got a 90/100 on my final SQL exam because I didn't end my, otherwise perfect, queries with a semicolon.
- freedomben 1y ago
  I run our interviewing process as well and fully agree with you. An interview process that most closely mirrors "real work" gets much better results. Some of the best people we've had were really terrible at algorithms and leet code, but wrote quality code (that they tested!), had strong work ethics, were willing to be around off-hours for the occasional production outage, took ownership/accountability for their tasks, and were enjoyable people to work with/around. It's good to have at least one person on the team who is good with algorithms and stuff for those times when that comes up, but having that be a requirement for everyone on the team is an anti-pattern IMHO
  - joshstrange 1y ago
    I completely agree. We've had almost every candidate (even some we _didn't_ hire) thank us for the interview process and how we went out of our way to make it stress-free and representative of the job. I'm not interested in tricking people or trying to show off "look how smart we are and how hard our test was"... puke. I want to work with solid performers who don't play games, why would I start that relationship off by playing a bunch of stupid games (silly interview tests/process)? And here's the thing, even with our somewhat easier/less-stress testing process, we still got high quality hires and have had very few cases where we have later had to let someone go due to not being able to do the job. This isn't a "hire fast, fire fast" situation.
- jghn 1y ago
  I agree with you that the real rubric is that I'd expect the candidate to be able to explain everything they did and why. For the most part I don't care how they got there. Was thinking about this recently in a conversation about whether or not the candidate should have a full screen share so that interviewers could see what they're looking up, etc. I realized that I'd reached the point where I now trust my fellow interviewers less than I do the candidate at these bits. Personally I think it's nice to see how people are leveraging tools, but too often I see interviewers dig candidates anyways on the specifics. I've now found this across multiple companies now has been that when coding exercises are "open book" like this that other interviewers *are* actively judging people based on what they're googling, and these days LLMing. If you're going to allow google but then fall back to "I can't believe they had to look *that* up!", that's not how it works.
  - sokoloff 1y ago
    I think “how to write for loop in $LANGUAGE” is something that it’s fair to judge a candidate on if they claim (or the job requires) recent experience in $LANGUAGE.
    - jghn 1y ago
      Maybe. I've been coding professionally for a few decades now. I have always looked stuff up constantly. It's a lot more seamless now due to first search engines and later the ever improving autocomplete. To be honest, I think half the time I just do it to do it. But for instance I bounce between languages so often that I can never remember the fine details of syntax. To use your example, things like for loops, matches, and the like are things that I tend to double check. And that's the point. Everyone has their quirks, and if you're going to claim you want them to work they way they normally work, who cares? If they're able to turn out quality code at a reasonable clip, who cares what they need to look up.
zamalek 1y ago
Makes sense. It's like we were prohibited from using calculators in school when I was growing up. If you're testing for on-the-job performance then you should have on-the-job tools.
- lesuorac 1y ago
  Well, while I agree with the second half of your statement. It only applies to when you're testing on-the-job performance. If you want to test if somebody learned multiplication then you do need to take away a calculator otherwise you're just testing if they can multiply. I'm not sure when an employer should really care about whether their cashier can add/subtract correctly vs just use the output from the machine. But in the education setting where you're making a claim that students will learn X you should be testing X.
  - psunavy03 1y ago
    The entertaining thing for at least Xennials and before is how our teachers used to tell us "out in the world, you won't just always have a calculator in your pocket!" And yet here we are . . .
- tracker1 1y ago
  I'm mixed on this one... In terms of schooling, it's about understanding how math works... not how to operate a calculator. Similar to graphing calculators vs basic scientific calculators... if you're trying to demonstrate you understand an equation, having the calculator do that for you doesn't show you understand what it is doing. Where in a job, you're probably not going to need to rebalance a B-Tree in any practical sense. For that matter, in my own experience, most actual development rarely has people optimizing their RDBMS usage in any meaningful way, for better or worse. It's the difference between learning and comprehension and delivering value. Someone has to understand how the machine works to fix it when it breaks. And there are jobs at every level. Most of them aren't that constrained in practice though.
  - zamalek 1y ago
MichaelNolan 1y ago
I'm curious what they will come up with. For smaller companies, leetcode style tests probably aren't the best, but for large companies that hire 10s of thousands of devs, leetcode style tests have a lot of good qualities. People who criticize leetcode usually prefer take home projects, pair programming with a current employee, "just a conversation" or something else, but these all have serious drawbacks at scale. Despite leetcode's flaws it has a lot of benefits: * Objective and clear grading criteria. * Unlimited supply of questions. * Easily time bound to 30 min or 1 hr. * Legal - i.e., no worries about disparate impact. * Programming language/framework agnostic.
- sp527 1y ago
  I can't even begin to imagine what sort of mind could observe the quality of Big Tech's software output and conclude that there's nothing wrong with their hiring process.
  - qwerpy 1y ago
    My problem with Big Tech software isn’t code quality, it’s the deliberate user-hostile decisions they make. Leetcode-style interviews appear to be doing fine at getting people who can write code.
  - paxys 1y ago
    Now look at their market cap.
    - saubeidl 1y ago
      It's almost as if capitalism wasn't a good system for deciding.. well, anything, really.
  - triceratops 1y ago
    Big Tech's software is faster and less buggy than the median software product.
    - 1y ago
reverendsteveii 1y ago
interviews should be about determining whether you can use the tools available to deliver the desired product, not some sort of purity test to determine whether you can build without tools. also I want to see you interact w AI because as of right now it's both an incredible and deeply flawed tool and your ability to recognize when it's about to walk you off a cliff is of increasing importance as we discover the limits of what it can do.
bgwalter 1y ago
Zuckerberg has no clue about software development. He always wants young people, huge bullpen offices, moonshot projects like the Metaverse that fail. Facebook open source software does not have great code quality. In some projects that always have been a huge mess they are now adding claude.md files to guide their beloved "AI". They did not add these files for humans before. I think Facebook software is a lost case where it does not matter if you perform the weekly rewriting by LLM or by kLOC driven humans.
- dangus 1y ago
  And yet they’re wildly successful, including the wide level of adoption of their open source projects like React. Meta’s profit per employee is over $250,000, higher than Qualcomm. There is no Meta competitor in any of their verticals that has a larger customer base. It seems to me that your definition of “software quality” and “lost cause” is factually wrong based on the metrics of success that matter. And in any event, it is an engineer’s tunnel vision fallacy to believe that software quality is the most important factor to stakeholders. People will prefer buggy/slow software that solves their problem over fast/stable software that fails to solve their problem.
  - mepian 1y ago
    > higher than Qualcomm That’s a pretty low bar for a software company.
    - dangus 1y ago
      Qualcomm basically has a monopoly in many of their product lines. They’re like the Nvidia of 5G modems and high end ARM chips. They’re also an insane patent licensing machine.
  - thefaux 1y ago
    And are almost never offered fast/stable software that solves their problem. When they are, it will come with aggressive fud by the competition or they will just lobby the government to mandate the incidental complexity of their own software so that the competition cannot even enter the market without being brought down to their level.
bwfan123 1y ago
Meta interviews in the past have been straight regurgitation of leetcode. They want to measure your flow and typing speed (sarcasm). So, the interview can now be 2 leetcode hards in 20 min. Earlier, it was typing solution code from rote memory. Now it is furious vibe-coding copy-pasta from one browser window to another. More seriously, what will the new questions look like ? In the age of LLMs how does one measure ability objectively. Is it with extreme underspecification of the probem ?
- dangus 1y ago
  It’s going to be just like the Google/StackOverflow era where your value as an engineer is based on your resourcefulness. The value of an employee who says “I don’t know how to do that” or “I’ll need to ask my coworkers for help” versus one that says “I am sure I can figure out just about anything by googling” is night and day, and I think the same is true with AI. Half of the battle is going to be knowing what to ask for. Lastly I’d like to point out that it makes general sense to test people on the real tools they’ll be using to get their work done. E.g., you wouldn’t get much value testing a bus driver on using a manual transmission if all your buses are automatic. Most corporate leaders are expecting and even demanding that their employees use AI tools to get their work done.
  - deleted 1y ago
    [deleted]
  - thefaux 1y ago
    I don't know how to do that is a great answer that I wish people would give more often. > Most corporate leaders are expecting and even demanding that their employees use AI tools to get their work done. Imagine for a second that I am an aspiring monopolist. Wouldn't a great way to achieve this be to make people believe that I am their agent, when I am really their adversary? Would I really offer them in good faith a tool that they could use to compete against me? Or would I try to get them to accept a trojan horse that destroys their profits from within. Once, I have siphoned enough profit to significantly damage the business, I can come in, borrow money to buy the company that I basically destroyed and then sell off the parts to an ignorant buyer who doesn't realize who badly I have wounded this once good business, or I just write off the loss for tax purposes.
garciasn 1y ago
https://archive.is/9faUx https://archive.is/9faUx
thr0w 1y ago
This makes sense in principle, but then how do you do technical evaluation? I'm generally most interested in hearing the candidate think out loud about the problem, and explain their approach. LLMs do the "thinking", and also explain the approach.
- add-sub-mul-div 1y ago
  It feels like a slow generational evolution beyond considering one employee different from another. They're biding their time until the AI can babysit itself.
- ndriscoll 1y ago
  As always, it requires a knowledgeable interviewer. Is the candidate using it as a slightly advanced autocomplete? That's fine. Are they relying on it to design things for them/can't articulate out loud what they're hoping it will produce for them/can't identify and articulate when and how it does something stupid or wrong or differs from how they were thinking of doing it? Then they don't know what they're doing. I do peer coding with people at work and we have copilot on. We'll discuss what we're trying to accomplish and make little comments out loud along the lines of "sure, almost" or "not quite, copilot" when it generates something. It's obvious whether someone is working in a way where they know what they want the tool to do.
  - tracker1 1y ago
    It's funny but I've been turning Github CoPilot off more often than on... it's absolutely been fantastic with some boilerplate SQL, and other tedious things, but any time I've tried doing complex things it seems to just get in the way more than help. As to interviews, I'd be happier if whatever tool they're using actually had working API code sense working... so many of the tools I've had to use in interviews just had broken LSP results. I've also had issue when a given question requires specific domain knowledge over a third party library as part of the question.
- marssaxman 1y ago
  For sure - I don't care about the code, I care about watching the applicant work, so I can decide whether I would want to work with them - that being the whole purpose of a job interview. If the "work" consists of delegating everything to a robot, I learn nothing and the interview is a waste of time.
ryandv 1y ago
[flagged]
gronglo 1y ago
Why is Meta still interviewing SWEs, is my question. Surely, at this point, an agent can beat any human candidate at any task. What am I missing?
- paxys 1y ago
  You are missing the fact that an agent cannot beat any human candidate at any task.
  - gronglo 1y ago
    Mark Zuckerberg himself predicted that AI would replace all mid-level engineers at Meta by the end of this year, with senior engineers continuing to work on tasks such as planning and architecture. With the release of Claude 4 it feels like, if anything, he was too conservative with his prediction — we're already there, and Mark must know it. So what's he playing at?
    - paxys 1y ago
      Look at Zuckerberg's actions, not what he says in podcasts to shill his AI tools and boost his company's stock price. Meta is currently hiring hundreds of software engineers every month and paying $200-300k/yr for entry level and $700k+ for senior roles. Why do you think that is?
    - saubeidl 1y ago
      AI snake oil salesman talks up AI snake oil, news at 11.
paxys 1y ago
This is born out of necessity. I have seen some of the coding interview "cheating" tools in the market today and it is ridiculous just how good they are at helping a candidate fake not just the answer but their entire thought process. Leetcode style interviewing is basically dead, at least if done remotely. The only way to fight this from the employer side is to embrace these tools and change your evaluation criteria.
- supportengineer 1y ago
  This escalating war will only result in them hiring those most skillful at cheating.
- msgodel 1y ago
  If you're good at interviewing I really don't think AI tools should make a difference tbh.
SkyPuncher 1y ago
I do interviews for my company. We allow AI (and even encourage it). Use or lack of use of AI has zero correlation with candidate performance. In fact, I’d say there’s a slight negative trend against those who use AI. We have candidates build a _very_ simple fullstack app. Think something like a TODO app, but only like 2 core functions. This is basic CRUD 101 stuff. I’ve seen a boatload of candidates use AI only to flame out after the first round of prompting. They literally get stuck and can’t move forward. The good candidates clearly know their fundamentals. They’re making intentional decisions, telling the AI scoped actions to take, and reviewing pretty much everything the AI generates. They reject code they don’t like. They explain why code is good or bad. They debug easily and quickly. LLMs are very, very talented but context still matters. They can’t build proper understanding of the non-technical components. They can’t solve things in the simplest way possible. They can’t decide that to explain to the interviewer “if performance is a concern, I’d do X but it’s a time-bound interview so I’m going to Y. Happy to do X if you need”.
dekhn 1y ago
Interesting. I interviewed at Meta a decade ago and was put off by the process (inexperienced devs who memorized leetcode medium problems). But I still get daily requests from their recruiters to try again (I am a senior leader in ML engineering). I would need to spend a fair amount of time thinking about what it would be like to do an interview with LLMs. But honestly, I'd rather spend that time on figuring out how to use LLMs to interview people better (for example I already had an LLM write a collaboritive web editor with built-in code runner, so I don't need to license coderpad). I could see updating my prompt to have the coding agent generate a text box for entering prompts during the interview. Either way, I still expect candidates to be able to explain what a hash table is in their own words.

55 points by geox 1y ago

I'm currently running the interview process for the company I work for and I allow AI on the coding tests for the same reason I allow googling/etc: I want to simulate as close to the real environment you will be coding in if you come and work here. My "rules" on AI/LLMs at work is that you can use whatever tools you want (much like SO before) but any code you commit, you are responsible for (also unchanged from before LLMs). I never want to hear "the LLM wrote that" when asked about how something works. If you can explain, modify, build on top of the code that an LLM spits out then fine by me. We don't require LLM usage (no token quotas or silly things like that) nor do we disallow LLMs. At the end of the day they are all tools. I've always found leet coding, whiteboarding, memorizing algorithms, etc to just be a silly form of hazing and a bad indicator of how well the employee will perform if you hire them. In the same way I still think my college was stupid for making us write, on paper, a C program with all the correct headers/syntax/etc for an exam or how I got a 90/100 on my final SQL exam because I didn't end my, otherwise perfect, queries with a semicolon.

freedomben 1y ago
I run our interviewing process as well and fully agree with you. An interview process that most closely mirrors "real work" gets much better results. Some of the best people we've had were really terrible at algorithms and leet code, but wrote quality code (that they tested!), had strong work ethics, were willing to be around off-hours for the occasional production outage, took ownership/accountability for their tasks, and were enjoyable people to work with/around. It's good to have at least one person on the team who is good with algorithms and stuff for those times when that comes up, but having that be a requirement for everyone on the team is an anti-pattern IMHO
- joshstrange 1y ago
  I completely agree. We've had almost every candidate (even some we _didn't_ hire) thank us for the interview process and how we went out of our way to make it stress-free and representative of the job. I'm not interested in tricking people or trying to show off "look how smart we are and how hard our test was"... puke. I want to work with solid performers who don't play games, why would I start that relationship off by playing a bunch of stupid games (silly interview tests/process)? And here's the thing, even with our somewhat easier/less-stress testing process, we still got high quality hires and have had very few cases where we have later had to let someone go due to not being able to do the job. This isn't a "hire fast, fire fast" situation.
jghn 1y ago
I agree with you that the real rubric is that I'd expect the candidate to be able to explain everything they did and why. For the most part I don't care how they got there. Was thinking about this recently in a conversation about whether or not the candidate should have a full screen share so that interviewers could see what they're looking up, etc. I realized that I'd reached the point where I now trust my fellow interviewers less than I do the candidate at these bits. Personally I think it's nice to see how people are leveraging tools, but too often I see interviewers dig candidates anyways on the specifics. I've now found this across multiple companies now has been that when coding exercises are "open book" like this that other interviewers *are* actively judging people based on what they're googling, and these days LLMing. If you're going to allow google but then fall back to "I can't believe they had to look *that* up!", that's not how it works.
- sokoloff 1y ago
  I think “how to write for loop in $LANGUAGE” is something that it’s fair to judge a candidate on if they claim (or the job requires) recent experience in $LANGUAGE.
  - jghn 1y ago
    Maybe. I've been coding professionally for a few decades now. I have always looked stuff up constantly. It's a lot more seamless now due to first search engines and later the ever improving autocomplete. To be honest, I think half the time I just do it to do it. But for instance I bounce between languages so often that I can never remember the fine details of syntax. To use your example, things like for loops, matches, and the like are things that I tend to double check. And that's the point. Everyone has their quirks, and if you're going to claim you want them to work they way they normally work, who cares? If they're able to turn out quality code at a reasonable clip, who cares what they need to look up.

zamalek 1y ago

Makes sense. It's like we were prohibited from using calculators in school when I was growing up. If you're testing for on-the-job performance then you should have on-the-job tools.

lesuorac 1y ago
Well, while I agree with the second half of your statement. It only applies to when you're testing on-the-job performance. If you want to test if somebody learned multiplication then you do need to take away a calculator otherwise you're just testing if they can multiply. I'm not sure when an employer should really care about whether their cashier can add/subtract correctly vs just use the output from the machine. But in the education setting where you're making a claim that students will learn X you should be testing X.
- psunavy03 1y ago
  The entertaining thing for at least Xennials and before is how our teachers used to tell us "out in the world, you won't just always have a calculator in your pocket!" And yet here we are . . .
tracker1 1y ago
I'm mixed on this one... In terms of schooling, it's about understanding how math works... not how to operate a calculator. Similar to graphing calculators vs basic scientific calculators... if you're trying to demonstrate you understand an equation, having the calculator do that for you doesn't show you understand what it is doing. Where in a job, you're probably not going to need to rebalance a B-Tree in any practical sense. For that matter, in my own experience, most actual development rarely has people optimizing their RDBMS usage in any meaningful way, for better or worse. It's the difference between learning and comprehension and delivering value. Someone has to understand how the machine works to fix it when it breaks. And there are jobs at every level. Most of them aren't that constrained in practice though.
- zamalek 1y ago

MichaelNolan 1y ago

I'm curious what they will come up with. For smaller companies, leetcode style tests probably aren't the best, but for large companies that hire 10s of thousands of devs, leetcode style tests have a lot of good qualities. People who criticize leetcode usually prefer take home projects, pair programming with a current employee, "just a conversation" or something else, but these all have serious drawbacks at scale. Despite leetcode's flaws it has a lot of benefits: * Objective and clear grading criteria. * Unlimited supply of questions. * Easily time bound to 30 min or 1 hr. * Legal - i.e., no worries about disparate impact. * Programming language/framework agnostic.

sp527 1y ago
I can't even begin to imagine what sort of mind could observe the quality of Big Tech's software output and conclude that there's nothing wrong with their hiring process.
- qwerpy 1y ago
  My problem with Big Tech software isn’t code quality, it’s the deliberate user-hostile decisions they make. Leetcode-style interviews appear to be doing fine at getting people who can write code.
- paxys 1y ago
  Now look at their market cap.
  - saubeidl 1y ago
    It's almost as if capitalism wasn't a good system for deciding.. well, anything, really.
- triceratops 1y ago
  Big Tech's software is faster and less buggy than the median software product.
  - 1y ago

reverendsteveii 1y ago

interviews should be about determining whether you can use the tools available to deliver the desired product, not some sort of purity test to determine whether you can build without tools. also I want to see you interact w AI because as of right now it's both an incredible and deeply flawed tool and your ability to recognize when it's about to walk you off a cliff is of increasing importance as we discover the limits of what it can do.

bgwalter 1y ago

Zuckerberg has no clue about software development. He always wants young people, huge bullpen offices, moonshot projects like the Metaverse that fail. Facebook open source software does not have great code quality. In some projects that always have been a huge mess they are now adding claude.md files to guide their beloved "AI". They did not add these files for humans before. I think Facebook software is a lost case where it does not matter if you perform the weekly rewriting by LLM or by kLOC driven humans.

dangus 1y ago
And yet they’re wildly successful, including the wide level of adoption of their open source projects like React. Meta’s profit per employee is over $250,000, higher than Qualcomm. There is no Meta competitor in any of their verticals that has a larger customer base. It seems to me that your definition of “software quality” and “lost cause” is factually wrong based on the metrics of success that matter. And in any event, it is an engineer’s tunnel vision fallacy to believe that software quality is the most important factor to stakeholders. People will prefer buggy/slow software that solves their problem over fast/stable software that fails to solve their problem.
- mepian 1y ago
  > higher than Qualcomm That’s a pretty low bar for a software company.
  - dangus 1y ago
    Qualcomm basically has a monopoly in many of their product lines. They’re like the Nvidia of 5G modems and high end ARM chips. They’re also an insane patent licensing machine.
- thefaux 1y ago
  And are almost never offered fast/stable software that solves their problem. When they are, it will come with aggressive fud by the competition or they will just lobby the government to mandate the incidental complexity of their own software so that the competition cannot even enter the market without being brought down to their level.

bwfan123 1y ago

Meta interviews in the past have been straight regurgitation of leetcode. They want to measure your flow and typing speed (sarcasm). So, the interview can now be 2 leetcode hards in 20 min. Earlier, it was typing solution code from rote memory. Now it is furious vibe-coding copy-pasta from one browser window to another. More seriously, what will the new questions look like ? In the age of LLMs how does one measure ability objectively. Is it with extreme underspecification of the probem ?

dangus 1y ago
It’s going to be just like the Google/StackOverflow era where your value as an engineer is based on your resourcefulness. The value of an employee who says “I don’t know how to do that” or “I’ll need to ask my coworkers for help” versus one that says “I am sure I can figure out just about anything by googling” is night and day, and I think the same is true with AI. Half of the battle is going to be knowing what to ask for. Lastly I’d like to point out that it makes general sense to test people on the real tools they’ll be using to get their work done. E.g., you wouldn’t get much value testing a bus driver on using a manual transmission if all your buses are automatic. Most corporate leaders are expecting and even demanding that their employees use AI tools to get their work done.
- deleted 1y ago
  [deleted]
- thefaux 1y ago
  I don't know how to do that is a great answer that I wish people would give more often. > Most corporate leaders are expecting and even demanding that their employees use AI tools to get their work done. Imagine for a second that I am an aspiring monopolist. Wouldn't a great way to achieve this be to make people believe that I am their agent, when I am really their adversary? Would I really offer them in good faith a tool that they could use to compete against me? Or would I try to get them to accept a trojan horse that destroys their profits from within. Once, I have siphoned enough profit to significantly damage the business, I can come in, borrow money to buy the company that I basically destroyed and then sell off the parts to an ignorant buyer who doesn't realize who badly I have wounded this once good business, or I just write off the loss for tax purposes.

garciasn 1y ago

https://archive.is/9faUx https://archive.is/9faUx

thr0w 1y ago

This makes sense in principle, but then how do you do technical evaluation? I'm generally most interested in hearing the candidate think out loud about the problem, and explain their approach. LLMs do the "thinking", and also explain the approach.

add-sub-mul-div 1y ago
It feels like a slow generational evolution beyond considering one employee different from another. They're biding their time until the AI can babysit itself.
ndriscoll 1y ago
As always, it requires a knowledgeable interviewer. Is the candidate using it as a slightly advanced autocomplete? That's fine. Are they relying on it to design things for them/can't articulate out loud what they're hoping it will produce for them/can't identify and articulate when and how it does something stupid or wrong or differs from how they were thinking of doing it? Then they don't know what they're doing. I do peer coding with people at work and we have copilot on. We'll discuss what we're trying to accomplish and make little comments out loud along the lines of "sure, almost" or "not quite, copilot" when it generates something. It's obvious whether someone is working in a way where they know what they want the tool to do.
- tracker1 1y ago
  It's funny but I've been turning Github CoPilot off more often than on... it's absolutely been fantastic with some boilerplate SQL, and other tedious things, but any time I've tried doing complex things it seems to just get in the way more than help. As to interviews, I'd be happier if whatever tool they're using actually had working API code sense working... so many of the tools I've had to use in interviews just had broken LSP results. I've also had issue when a given question requires specific domain knowledge over a third party library as part of the question.
marssaxman 1y ago
For sure - I don't care about the code, I care about watching the applicant work, so I can decide whether I would want to work with them - that being the whole purpose of a job interview. If the "work" consists of delegating everything to a robot, I learn nothing and the interview is a waste of time.

ryandv 1y ago

[flagged]

gronglo 1y ago

Why is Meta still interviewing SWEs, is my question. Surely, at this point, an agent can beat any human candidate at any task. What am I missing?

paxys 1y ago
You are missing the fact that an agent cannot beat any human candidate at any task.
- gronglo 1y ago
  Mark Zuckerberg himself predicted that AI would replace all mid-level engineers at Meta by the end of this year, with senior engineers continuing to work on tasks such as planning and architecture. With the release of Claude 4 it feels like, if anything, he was too conservative with his prediction — we're already there, and Mark must know it. So what's he playing at?
  - paxys 1y ago
    Look at Zuckerberg's actions, not what he says in podcasts to shill his AI tools and boost his company's stock price. Meta is currently hiring hundreds of software engineers every month and paying $200-300k/yr for entry level and $700k+ for senior roles. Why do you think that is?
  - saubeidl 1y ago
    AI snake oil salesman talks up AI snake oil, news at 11.

paxys 1y ago

This is born out of necessity. I have seen some of the coding interview "cheating" tools in the market today and it is ridiculous just how good they are at helping a candidate fake not just the answer but their entire thought process. Leetcode style interviewing is basically dead, at least if done remotely. The only way to fight this from the employer side is to embrace these tools and change your evaluation criteria.

supportengineer 1y ago
This escalating war will only result in them hiring those most skillful at cheating.
msgodel 1y ago
If you're good at interviewing I really don't think AI tools should make a difference tbh.

SkyPuncher 1y ago

I do interviews for my company. We allow AI (and even encourage it). Use or lack of use of AI has zero correlation with candidate performance. In fact, I’d say there’s a slight negative trend against those who use AI. We have candidates build a _very_ simple fullstack app. Think something like a TODO app, but only like 2 core functions. This is basic CRUD 101 stuff. I’ve seen a boatload of candidates use AI only to flame out after the first round of prompting. They literally get stuck and can’t move forward. The good candidates clearly know their fundamentals. They’re making intentional decisions, telling the AI scoped actions to take, and reviewing pretty much everything the AI generates. They reject code they don’t like. They explain why code is good or bad. They debug easily and quickly. LLMs are very, very talented but context still matters. They can’t build proper understanding of the non-technical components. They can’t solve things in the simplest way possible. They can’t decide that to explain to the interviewer “if performance is a concern, I’d do X but it’s a time-bound interview so I’m going to Y. Happy to do X if you need”.

dekhn 1y ago

Interesting. I interviewed at Meta a decade ago and was put off by the process (inexperienced devs who memorized leetcode medium problems). But I still get daily requests from their recruiters to try again (I am a senior leader in ML engineering). I would need to spend a fair amount of time thinking about what it would be like to do an interview with LLMs. But honestly, I'd rather spend that time on figuring out how to use LLMs to interview people better (for example I already had an LLM write a collaboritive web editor with built-in code runner, so I don't need to license coderpad). I could see updating my prompt to have the coding agent generate a text box for entering prompts during the interview. Either way, I still expect candidates to be able to explain what a hash table is in their own words.