6 ms·

All this AI coding stuff is scaring the shit out of me. a few months ago my team were hiring for a new engineer. of the 9 candidates we ran technical interviews

All this AI coding stuff is scaring the shit out of me. a few months ago my team were hiring for a new engineer. of the 9 candidates we ran technical interviews with, only two could work without the ai. The rest literally just vibe coded their way though the app. as soon as it was taken away, they couldn't even write a basic sql query in ecto (we're a phoenix app). when questioned about tradeoffs inherent in the ai generated implementation, all but one was completely in the dark.

danielbln 1y ago
Now take Google away, and LSP. And the computer. Write CTEs with a pencil or bust. I'm exaggerating of ourse, and I hear what you're saying, but I'd rather hire someone who is really really good at squeezing the most out of current day AI (read: not vibe coding slop) than someone who can do the work manually without assistance or fizz buzz on a whiteboard.
- dnoberon 1y ago
  I think the point is how can you squeeze anything out of the AI without knowing the stuff at a deep enough level?
  - deleted 1y ago
    [deleted]
  - kenjackson 1y ago
    Ask most folks about the code generated by the compiler or interpreter and you’ll get blank stares. Even game devs now barely know assembly, much less efficient assembly. There is still a place for someone who is going to rewrite your inner-loops with hand-tuned assembly, but most coding is about delivering on functional requirement. And using tools to do this, AI or not, tend to be the prudent path in many if not most cases.
    - jamesmccann 1y ago
      I don't think these are comparable though. Compiler generation is deterministic and more or less provably correct. LLM code is a mile away from that.
    - skydhash 1y ago
      Apart from the whole argument about compilers being deterministic and not LLMS. You don't collaborate on compiled code. They are artifacts. But you're collaborating on source code, so whatever you write, someone else (or you in the future) will need to understand it and alter it. That's what the whole maintainability, testability,... is about. And that's why code is a liability, because it takes times for someone else to understand it. So the less you write, the better it is (there's some tradeoffs about complexity).
      - kenjackson 1y ago
        You can make LLMs deterministic, but that's not a priority right now. In the same way we used to not capture dev environments and end up in situations where you couldn't rebuild a binary exactly because the OS version, the compiler version, the CRT version, etc... all changed -- of course that's a 20 year old problem now, but was a legitimate problem as recently as 2000. And again, we're at a point in time where we do collaborate on the source code artifacts. But maybe we won't in the future. It assumes that we see AI progress, I can see a world where asking questions of the AI about the code is better than 99% of developers. There will be the John Carmack's of the world though who know better than the AI, but the common case is that we eventually move away from looking at code directly. But this does rely on continued progress that we may not get.
  - jama211 1y ago
    Being able to memorise things that are easily looked up (like syntax) doesn’t demonstrate deep knowledge. It’s a bad interview question.
    - myaccountonhn 1y ago
      I mean maybe these juniors are geniuses, but I often find it very non-obvious why LLM-generated code it wrong and it requires me to have an even deeper knowledge. Sometimes the code is correct, but overly complicated. One small example was a coworker that generated random numbers with AI using `dd count=30 if=/dev/urandom | tr -c "[a-z][A-Z]" | base64 | head -c20` instead of just `head -c20 /dev/urandom | base64`. I didn't actually know `dd` beyond that it's used for writing to usb-sticks, but I suddenly became really unsure if I was missing something and needing to double check the documentation. All that to say that I think if you vibe-code, you really need to know what you're generating and to keep in mind that other will need to be able to read and understand what you've written.
- instig007 1y ago
  > I'd rather hire someone [...] than someone who can do the work manually without assistance or fizz buzz on a whiteboard and the reason for you to do that would be to punish the remaining bits of competence in the name of "the current thing"? What's your strategy?
- timeon 1y ago
  Used to write Perl scripts with pencil while waiting at the airport.
  - christophilus 1y ago
    And then later, when you couldn’t read your own handwriting, you said, “This damn thing is illegible— so it’s probably valid Perl.”
    - yen223 1y ago
      Perl is just Python with bad handwriting
- kfajdsl 1y ago
  For your examples, honestly yeah. A dev should familiar with the basic concepts of their language and tech stack. So yes, they should be able to understand a basic snippet of code without Google, an LSP, or even a computer. They should even be able to "write CTEs with a pencil and paper". I don't expect them to get the syntax perfect, but they should just know the basic tools and concepts enough to have something at least semantically correct. And they certainly should be able to understand the code produced by an AI tool for a take home toy project. I say this as someone who would definitely be far less productive without Google, LSP, or Claude Code.
  - jama211 1y ago
    I’ve written huge queries and CTE’s in my career. But I haven’t done it recently. Personally, I’d need 10 minutes of google time to refresh my memory before being able to write much sql on paper, even with bad syntax. It doesn’t mean I’m a bad engineer because I don’t bother to memorise stuff that’s easily googleable.
runako 1y ago
> couldn't even write a basic sql query Not the point at all, but I have found it quite common among younger professional engineers to not know SQL at all. A combination of specialization (e.g. only work on microservices that do not directly touch a database) and NoSQL has made the skill of SQL more obscure than I would have thought possible as recently as 5 years ago.
- ElCapitanMarkla 1y ago
  I started to notice this in a big way at my last job which I started in 2013. We were a rails shop and by about 2016 I was noticing most new hires would have no idea how to write a SQL query.
  - cultofmetatron 1y ago
    > most new hires would have no idea how to write a SQL query. probably why people think rails is slow. our integration partners and our customers are constantly amazed by how fast and efficient our system is. The secret is I know how to write a damn query. you can push a lot of logic that would otherwise be done in the api layer into a query. if done properly with the right indexes, its going to be WAY faster than pulling the data into the api server and doing clumsy data transformations there.
    - hollowturtle 1y ago
      You actually confirmed that rails is slow if the optimization is on the database server and doing data mangling in ruby is less efficient
      - richwater 1y ago
        You've correctly identified that filtering a list is slower than looking up from an index. Congratulations.
        hollowturtle 1y ago
withinboredom 1y ago
Same. One candidate out of 6. I use claude code quite liberally, but I very often tell it why I won't accept it's changes and why; sometimes I just do it myself if it doesn't "get it".
pryelluw 1y ago
This was my experience prior to any of the llm tools. It’s hard to find people with all around knowledge. Plus someone good in one context is awful in another. Your hiring process should find people who are a good fit and not look for people with just certain technical skills. The basics of SQL can be learned quickly. Fit cannot be learned.
- jama211 1y ago
  Well said. Some of the best engineers I know looked up syntax whenever they needed it because there’s not much point in wrote learning everything. As long as they understand what they’re doing, that’s the main point. I’m honestly so sick of interviews filled with gotcha questions that if you’d happened to study the right thing you could outperform a great experienced engineer who hadn’t brushed up on a couple of specific googlable things before the interview. It’s such a bad practice.
deleted 1y ago
[deleted]
lvl155 1y ago
AI can also help you learn new things much faster. It’s just a tool.
- hollowturtle 1y ago
  I'd say "Learn the wrong things much faster". But I'd actually argue that learning isn't a fast process, it's rather a very slow journey, takes time and dedication to master deep knowledge. You won't learn anything that will stay with llms, if they got the output correct
  - lvl155 1y ago
    Sorry to be harsh but that just sounds ignorant. LLM can be a great tool to speed up your learning process especially for devs. It can help me get over blocks that used to take me half a day of digging through docs and codes.
    - skydhash 1y ago
      Having a quick answer is different than internalizing some domain, especially when you may need to argue about it or propose some novel method. The former can work if you know the right question, but that's just as difficult as knowing the answer. And most things that are useful daily is not pure knowledge. It's adapting the latter to the current circumstances (aka making tradeoffs). Coding is where pure knowledge shines and it's the easiest part. Before that comes designing a solution and fitting it to the current architecture and that's where judgement and domain knowledge are important.
    - hollowturtle 1y ago
      I may sound ignorant but if you delegate your knowledge to LLMs you'll stay ignorant. Learning is a never ending journey, you can chat about a problem with a non deterministic stocastic compressor with stale knowledge, no problem but don't call it education. Core domain knowledge still makes people successful not chatting skills. Sure if you have deep domain knowledge you can still benefinit AI filling the tedious details, I'm not anti AI
paffdragon 1y ago
We have also seen this about a year ago when hiring. But only a couple of them made it to the live interview and then it was evident. Most of them were quickly filtered out based on the coding submissions. We are soon about to hire again, with the uptick in LLM usage and newer more up to date models, I'm not looking forward too much having to deal with all of this.
- hopelite 1y ago
  Maybe it’s time for you to update your perspective. You strike me as the old guy who curses all the young’ens who use the fancy electric power tools and never learn how to use a manual saw … and you don’t like it much.
  - paffdragon 1y ago
    I don't know why you make up such stories for yourself. It's completely pointless and has zero value. Let me share you something, maybe it helps to update your perspective. We reject people not because they help themselves with AI, everyone on the team uses AI in some form. Candidates are mostly rejected, because they don't understand what they write and can't explain what they just dumped into the editor from their AI assistant. We don't need colleagues who don't know what they ship and can't reason about code and won't be able to maintain it and troubleshoot issues. I can get the same level from an AI assistant without hiring. It's not old vs. young, we have plenty of young people on the team, it's about our time and efforts spent on people trying to fake their skills with AI help and then eventually fail and we wasted our time. This is the annoying part, the waste, because AI makes it easier to fake the process longer for people without the required skills.
jama211 1y ago
I’ve worked for years in the past on huge complex sql. I wouldn’t have been able to remember exactly what that looks like in sql without a quick search. Your interview questions are bad if they require wrote learned syntax. Great programmers exist who barely bother to remember anything they can’t just look up.
trenchpilgrim 1y ago
I know SQL but rarely have to use it because my career has mostly been nonrelational data (etcd, prometheus, object storage, dataframes). So for anything more than the basics I have to read the manual to refresh my memory.

danielbln 1y ago

Now take Google away, and LSP. And the computer. Write CTEs with a pencil or bust. I'm exaggerating of ourse, and I hear what you're saying, but I'd rather hire someone who is really really good at squeezing the most out of current day AI (read: not vibe coding slop) than someone who can do the work manually without assistance or fizz buzz on a whiteboard.

dnoberon 1y ago
I think the point is how can you squeeze anything out of the AI without knowing the stuff at a deep enough level?
- deleted 1y ago
  [deleted]
- kenjackson 1y ago
  Ask most folks about the code generated by the compiler or interpreter and you’ll get blank stares. Even game devs now barely know assembly, much less efficient assembly. There is still a place for someone who is going to rewrite your inner-loops with hand-tuned assembly, but most coding is about delivering on functional requirement. And using tools to do this, AI or not, tend to be the prudent path in many if not most cases.
  - jamesmccann 1y ago
    I don't think these are comparable though. Compiler generation is deterministic and more or less provably correct. LLM code is a mile away from that.
  - skydhash 1y ago
    Apart from the whole argument about compilers being deterministic and not LLMS. You don't collaborate on compiled code. They are artifacts. But you're collaborating on source code, so whatever you write, someone else (or you in the future) will need to understand it and alter it. That's what the whole maintainability, testability,... is about. And that's why code is a liability, because it takes times for someone else to understand it. So the less you write, the better it is (there's some tradeoffs about complexity).
    - kenjackson 1y ago
      You can make LLMs deterministic, but that's not a priority right now. In the same way we used to not capture dev environments and end up in situations where you couldn't rebuild a binary exactly because the OS version, the compiler version, the CRT version, etc... all changed -- of course that's a 20 year old problem now, but was a legitimate problem as recently as 2000. And again, we're at a point in time where we do collaborate on the source code artifacts. But maybe we won't in the future. It assumes that we see AI progress, I can see a world where asking questions of the AI about the code is better than 99% of developers. There will be the John Carmack's of the world though who know better than the AI, but the common case is that we eventually move away from looking at code directly. But this does rely on continued progress that we may not get.
- jama211 1y ago
  Being able to memorise things that are easily looked up (like syntax) doesn’t demonstrate deep knowledge. It’s a bad interview question.
  - myaccountonhn 1y ago
    I mean maybe these juniors are geniuses, but I often find it very non-obvious why LLM-generated code it wrong and it requires me to have an even deeper knowledge. Sometimes the code is correct, but overly complicated. One small example was a coworker that generated random numbers with AI using `dd count=30 if=/dev/urandom | tr -c "[a-z][A-Z]" | base64 | head -c20` instead of just `head -c20 /dev/urandom | base64`. I didn't actually know `dd` beyond that it's used for writing to usb-sticks, but I suddenly became really unsure if I was missing something and needing to double check the documentation. All that to say that I think if you vibe-code, you really need to know what you're generating and to keep in mind that other will need to be able to read and understand what you've written.
instig007 1y ago
> I'd rather hire someone [...] than someone who can do the work manually without assistance or fizz buzz on a whiteboard and the reason for you to do that would be to punish the remaining bits of competence in the name of "the current thing"? What's your strategy?
timeon 1y ago
Used to write Perl scripts with pencil while waiting at the airport.
- christophilus 1y ago
  And then later, when you couldn’t read your own handwriting, you said, “This damn thing is illegible— so it’s probably valid Perl.”
  - yen223 1y ago
    Perl is just Python with bad handwriting
kfajdsl 1y ago
For your examples, honestly yeah. A dev should familiar with the basic concepts of their language and tech stack. So yes, they should be able to understand a basic snippet of code without Google, an LSP, or even a computer. They should even be able to "write CTEs with a pencil and paper". I don't expect them to get the syntax perfect, but they should just know the basic tools and concepts enough to have something at least semantically correct. And they certainly should be able to understand the code produced by an AI tool for a take home toy project. I say this as someone who would definitely be far less productive without Google, LSP, or Claude Code.
- jama211 1y ago
  I’ve written huge queries and CTE’s in my career. But I haven’t done it recently. Personally, I’d need 10 minutes of google time to refresh my memory before being able to write much sql on paper, even with bad syntax. It doesn’t mean I’m a bad engineer because I don’t bother to memorise stuff that’s easily googleable.

runako 1y ago

> couldn't even write a basic sql query Not the point at all, but I have found it quite common among younger professional engineers to not know SQL at all. A combination of specialization (e.g. only work on microservices that do not directly touch a database) and NoSQL has made the skill of SQL more obscure than I would have thought possible as recently as 5 years ago.

ElCapitanMarkla 1y ago
I started to notice this in a big way at my last job which I started in 2013. We were a rails shop and by about 2016 I was noticing most new hires would have no idea how to write a SQL query.
- cultofmetatron 1y ago
  > most new hires would have no idea how to write a SQL query. probably why people think rails is slow. our integration partners and our customers are constantly amazed by how fast and efficient our system is. The secret is I know how to write a damn query. you can push a lot of logic that would otherwise be done in the api layer into a query. if done properly with the right indexes, its going to be WAY faster than pulling the data into the api server and doing clumsy data transformations there.
  - hollowturtle 1y ago
    You actually confirmed that rails is slow if the optimization is on the database server and doing data mangling in ruby is less efficient
    - richwater 1y ago
      You've correctly identified that filtering a list is slower than looking up from an index. Congratulations.
      - hollowturtle 1y ago

withinboredom 1y ago

Same. One candidate out of 6. I use claude code quite liberally, but I very often tell it why I won't accept it's changes and why; sometimes I just do it myself if it doesn't "get it".

pryelluw 1y ago

This was my experience prior to any of the llm tools. It’s hard to find people with all around knowledge. Plus someone good in one context is awful in another. Your hiring process should find people who are a good fit and not look for people with just certain technical skills. The basics of SQL can be learned quickly. Fit cannot be learned.

jama211 1y ago
Well said. Some of the best engineers I know looked up syntax whenever they needed it because there’s not much point in wrote learning everything. As long as they understand what they’re doing, that’s the main point. I’m honestly so sick of interviews filled with gotcha questions that if you’d happened to study the right thing you could outperform a great experienced engineer who hadn’t brushed up on a couple of specific googlable things before the interview. It’s such a bad practice.

deleted 1y ago

[deleted]

lvl155 1y ago

AI can also help you learn new things much faster. It’s just a tool.

hollowturtle 1y ago
I'd say "Learn the wrong things much faster". But I'd actually argue that learning isn't a fast process, it's rather a very slow journey, takes time and dedication to master deep knowledge. You won't learn anything that will stay with llms, if they got the output correct
- lvl155 1y ago
  Sorry to be harsh but that just sounds ignorant. LLM can be a great tool to speed up your learning process especially for devs. It can help me get over blocks that used to take me half a day of digging through docs and codes.
  - skydhash 1y ago
    Having a quick answer is different than internalizing some domain, especially when you may need to argue about it or propose some novel method. The former can work if you know the right question, but that's just as difficult as knowing the answer. And most things that are useful daily is not pure knowledge. It's adapting the latter to the current circumstances (aka making tradeoffs). Coding is where pure knowledge shines and it's the easiest part. Before that comes designing a solution and fitting it to the current architecture and that's where judgement and domain knowledge are important.
  - hollowturtle 1y ago
    I may sound ignorant but if you delegate your knowledge to LLMs you'll stay ignorant. Learning is a never ending journey, you can chat about a problem with a non deterministic stocastic compressor with stale knowledge, no problem but don't call it education. Core domain knowledge still makes people successful not chatting skills. Sure if you have deep domain knowledge you can still benefinit AI filling the tedious details, I'm not anti AI

paffdragon 1y ago

We have also seen this about a year ago when hiring. But only a couple of them made it to the live interview and then it was evident. Most of them were quickly filtered out based on the coding submissions. We are soon about to hire again, with the uptick in LLM usage and newer more up to date models, I'm not looking forward too much having to deal with all of this.

hopelite 1y ago
Maybe it’s time for you to update your perspective. You strike me as the old guy who curses all the young’ens who use the fancy electric power tools and never learn how to use a manual saw … and you don’t like it much.
- paffdragon 1y ago
  I don't know why you make up such stories for yourself. It's completely pointless and has zero value. Let me share you something, maybe it helps to update your perspective. We reject people not because they help themselves with AI, everyone on the team uses AI in some form. Candidates are mostly rejected, because they don't understand what they write and can't explain what they just dumped into the editor from their AI assistant. We don't need colleagues who don't know what they ship and can't reason about code and won't be able to maintain it and troubleshoot issues. I can get the same level from an AI assistant without hiring. It's not old vs. young, we have plenty of young people on the team, it's about our time and efforts spent on people trying to fake their skills with AI help and then eventually fail and we wasted our time. This is the annoying part, the waste, because AI makes it easier to fake the process longer for people without the required skills.

jama211 1y ago

I’ve worked for years in the past on huge complex sql. I wouldn’t have been able to remember exactly what that looks like in sql without a quick search. Your interview questions are bad if they require wrote learned syntax. Great programmers exist who barely bother to remember anything they can’t just look up.

trenchpilgrim 1y ago

I know SQL but rarely have to use it because my career has mostly been nonrelational data (etcd, prometheus, object storage, dataframes). So for anything more than the basics I have to read the manual to refresh my memory.