7 ms·

Atrophy has really been an issue in my recent hiring cycles for good senior engineers. 80% of senior candidates I interview now aren’t able to do junior level

• k__ 1y ago
  It's a matter of opinion what you should know and what you can easily google or ask an LLM.
  • dagmx 1y ago
    If someone needs to continuously google how to use the basic data structures in a language they use every day, then I worry about their ability for knowledge retention as a whole.
    • Herring 1y ago
      “A human being should be able to change a diaper, plan an invasion, butcher a hog, conn a ship, design a building, write a sonnet, balance accounts, build a wall, set a bone, comfort the dying, take orders, give orders, cooperate, act alone, solve equations, analyze a new problem, pitch manure, program a computer, cook a tasty meal, fight efficiently, die gallantly. Specialization is for insects.” ― Robert A. Heinlein (It's a matter of opinion)
      • dagmx 1y ago
        Except I’m not hiring someone to do 20 things. I’m hiring them to do the one thing they say they can do. Would I hire a taxi driver who can’t drive to drive me somewhere? Why would I hire a software engineer who can’t demonstrate their abilities.
        • Herring 1y ago
          edit: wasting my time
          • dagmx 1y ago
            Please re-read the original comment. All your suggestions are addressed there and I explain why this is inadequate.
        • johnnyanmac 1y ago
          >I’m hiring them to do the one thing they say they can do. If you hire an amazing programmer but then ask for a quick report on their implementation and it's gibberish. Is that satisfactory? Their job isn't to write reports. If you ask them during lunch to grab your meal from the desk and they spill it everywhere, is that satisfactory? You didn't hire then for their ability to carry a plate. EDIT: I should add here that I agree with your hiring approach. I just think you're critiquing this quote in the wrong way. The point of the quote is that humans have a capacity to learn and perform many tasks, even if they do some better than others. LLMs hinder this ability.
      • 0x696C6961 1y ago
        I'm not sure I understand the point you're trying to make. Do you think a senior developer needs a basic understanding of the language they use or not?
        • ben_w 1y ago
          > Do you think a senior developer needs a basic understanding of the language they use 4 years ago, I'd have said "obviously". At this point? Only for specialist languages the LLMs suck at. (Last I tried, "suck at" included going all the way from yacc etc. upwards when trying to make a custom language). For most of us, what's now important is: Design patterns; architectures; knowing how to think of big-O consequences; and the ability to review what the AI farts out, which sounds like it should need an understanding of the programming language in question, but it's surprisingly straightforward to read code in languages you're not fmiliar with, and it gets even easier when your instruction to the AI includes making the code easy to review.
    • raincole 1y ago
      I think it depends on what you consider "basic data structures"... If it's the List and Dict (or whatever they are called in that language) then maybe. But I'd not expect someone to spell the correct API by heart to use Stack, Queue, Deque, etc, even they're text book examples of "basic" data structures.
      • dagmx 1y ago
        For me, in a coding test, basic data structures are (using Python terms for brevity): - lists - dictionaries - sets - nested versions of the above - strings (not exactly a data structure) strings are tenuously in my data structures list because I let people treat them as ascii arrays to avoid most of the string footguns.
    • jstummbillig 1y ago
      The question is: What are they better at? If you think the answer is "Nothing", I would be suspicious.
      • dagmx 1y ago
        It’s never nothing, but it’s a matter of whether that something is enough of a differentiator. More so, can they demonstrate that in interviews. We specifically structure our interviews so coding is only one part of them, and try and suss out other aspects of a candidate (problem solving, project leading, working with different disciplines) and then weight those against the needs of the role to avoid over indexing on any specific aspect. It also lets us see if there are other roles for them that might be better fits. So a weak coder has opportunities to show other qualities. But generally someone interviewing for a coding heavy role who isn’t a strong coder tends to not sufficiently demonstrate those other qualities. Though of course there are exceptions.
• Narciss 1y ago
  I was actually thinking about this the other day while vibe coding for a side project. I am a lead engineer, but I’ve been using AI in much of my code recently. If you were to ask me to code anything manually right now, I could do it, but it would take a bit to acclimate to writing code line by line. By “a while”, I mean maybe a few days. Which means that if we were to do a coding interview without LLMs, I would probably flop without me doing a bit of work beforehand, or at least struggle. But hire me regardless, and I would get back on track in a few days and be better than most from then on. Careful not to lose talent just because you are testing for little used but latent capabilities.
  • dagmx 1y ago
    In your scenario though, how do you avoid hiring based on blind faith? How do I know you aren’t just a lead with a very good team to pick up the slack? How do I separate you from the 20 other people saying they’re also good? Why would I hire someone who can’t hit the ground running faster than someone else who can? Furthermore, why would I hire someone who didn’t prepare at all for an interview, even if just mentally? How do you avoid just hiring based on vibes? Bear in mind every candidate can claim they’re part of impressive projects so the resume is often not your differentiator.
    • htrp 1y ago
      We're gonna have to reinvent swe hiring
    • zelphirkalt 1y ago
      You invest time into actually looking at applications properly and look at their previous work and ask them questions about that. The curious and learning mind will probably have some projects, because they are not only into the job for the money, but because they have a passion for the job. That's not a requirement, but it distinguishes the great ones from the mediocre ones.
      • dagmx 1y ago
        That’s also something any number of people can fake easily. There’s a ton of smooth talkers who can hype up a project and talk in abstract about their previous projects but weren’t on the side of delivering it.
• alchemism 1y ago
  Pair a senior with an agent LLM, pair a junior with an agent LLM, measure the output over some cycles. You will find your answer in the data, one way or another.
  • dagmx 1y ago
    Truthfully, in my experience, they both end up performing near the level of the LLM. It’s an averaging factor not an uplifting one.
    • alchemism 1y ago
      Right now we are seeing junior hiring fall off a cliff, whether or not LLMs are responsible is hard to say. But if it does turn out that everyone performs to the level of the LLM, then a vast amount of business dollars can be saved by hiring starving English majors exclusively and dispensing with STEM majors altogether.
      • dagmx 1y ago
        I said it’s an averaging factor not an absolute match. A senior with a LLM will likely still outperform a junior. A CS major with an LLM will out perform an English major. But is the senior out performing the junior at a level that warrants the difference in salary? Then people will point to the intangibles of experience, but will forget the intangibles of being fresh and new with regards to being malleable and gung-ho.
      • jasonthorsness 1y ago
        I keep hearing this however I was just at a career fair for University of Washington CSE new grads (representing my company) and it was packed with companies hiring. Obviously, this is just an anecdote, is there any good data showing the real situation?
        • mschuster91 1y ago
          It's anecdata from layoff reports and whatever people see from their own employer / industry. To add my two cents of anecdotes from what I can tell in the creative agency world: shit sucks hard at the moment. Clients are cutting back budgets hard and a lot of them explicitly ask for AI strategies - obviously in terms of saving money. Blockchain craze was similarly bad when it comes to client requests for stuff that Just Did Not Make Sense, but at least there was money pouring in. And it's that way across the industry, everyone I know feels the same. There's no money from clients that would pay for juniors, so no juniors are getting hired even if it might blow up in five, six years when there are no fresh juniors/intermediates around. Personally, I decided to part ways. There's many places I want to be, but the creative industry when the AI bubble and hiring anti-bubble finally pops? Oh hell no.
• athrowaway3z 1y ago
  What do you consider senior? We've seem to have had a significant title-inflation in the last 5 years, and everybody seems to be at least a senior. With no new junior positions opening up, I'm not even sure I blame them.
  • dagmx 1y ago
    A senior to me is someone who can tackle complex problems without significant supervision , differentiated from a lead in that they still need guidance on what the overall tasks are. They should be familiar enough with their tech stacks that I can send them to meetings (after the requisite on-boarding time) to represent the team if needed (though I try and not overload my engineers with meetings) and answer feasibility questions for our projects. They don’t need constant check ins on their work. I should be able to bounce ideas of them readily on how to approach bigger problems. A junior being someone who needs more granular direction or guidance. I’d only send them to meetings paired with a senior. They need close to daily check ins on their work. I include them in all the same things the seniors do for exposure, but do not expect the same level of technical strength at this time in their careers. I try not to focus on years of experience necessarily, partly because I was supervising teams at large companies very early in my career.
• spaceballbat 1y ago
  Grade inflation has spilled over into the corporate world. I’ve interviewed people titled “principal” who would barely qualify as “senior” a few decades ago.
  • ben_w 1y ago
    "Senior" was already a weird title, given it could have been anything from 3-10 years of experience even back in 2021. I've seen people with 10 years experience blindly duplicate C++ classes rather than subclass them, and when questioned they seemed to think the mere existence of `private:` access specifiers justified it. There were two full time developers including him, and no code review, so it's not like any of the access specifiers even did anything useful.
    • spaceballbat 1y ago
      The jump from junior to senior means you can self start and have created enough of a network to seek out help. Junior used to be a 1-3 year training period. Senior to principal means you have signififcant positive impact across the company: upper management relies on you to define the roadmap. Most people hang out in ‘senior’ for their entire careers because they never have that drive to stand out. Thats why there are titles like “staff” and “senior staff” to promote people who don’t have what it takes to get to principal.
      • Ekaros 1y ago
        I always wondered why don't we have just developer level. That is for those with say 5-7 years of experience. That are skilled enough to do their bit on their own, but might not yet fully see the big picture on whole application scale for whole thing... Straight from junior to senior just feels weird jump. Junior and Senior sound like adjectives to me. Qualifiers. And there should be some middle point in between where bulk of the workforce doing the actual job should be.
        • dagmx 1y ago
          A lot of companies do have levels. See http://www.levels.fyi http://www.levels.fyi
• erichocean 1y ago
  > Firstly, we’re making new technologies and APIs that LLMs really struggle LLMs absolutely excel at this task. Source: Me, been doing it since early July with Gemini Pro 2.5 and Claude Opus. So good, in fact, that I have no plans to hire software engineers in the future. (I have hired many over my 25 years developing software.)
  • dagmx 1y ago
    So you’ve made a decision based on three months of use. I am legitimately interested in your experience though. What are you creating where you can see the results in that time frame to make entire business decisions like that? I would really like to see those kinds of productivity gains myself.
    • erichocean 1y ago
      I'll give you an easy example. LMDB has a (very) long-standing bug where "DUPSORT" databases become corrupted over time. Separately, I wanted to make some changes to LMDB but the code is so opaque that it's hard to do anything with it (safely). So I gave the entire codebase to Gemini Pro 2.5 and had it develop a glossary for local variable renames and for structure member renames. I then hand-renamed all of the structures (using my IDE's struct member refactoring tools). Then I gave the local variable glossary and each function to Gemini and had it rewrite the code. Finally, I had a separate Gemini Pro 2.5 context and a Claude Opus context validate that the new code was LOGICALLY IDENTICAL the previous code (i.e. that only local variables were renamed, and that the renaming was consistent). Most of the time, GPro did the rewrite correctly the first time, but other times, it took 3-4 passes before GPro and Opus agreed. Each time, I simply pasted the feedback from one of the LLMs back into the original context and told it to fix it. The largest function done this way was ~560 LOC. Anyway, the entire process took around a day. However, at one point, GPro reported: "Hey, this code is logically identical BUT THERE IS A SERIOUS BUG." Turns out, it had found the cause of the DUPSORT corruption, without any prompting—all because the code was much cleaner than it was at the start of the day. That is wild to me! (It actually found another, less important bug too.) Without LLMs, I would have never even attempted this kind of refactoring. And I certainly wouldn't pay a software engineer to do it. > What are you creating where you can see the results in that time frame to make entire business decisions like that? I've developed new libraries (using GPro) that have APIs it can reliably use. What's easy for LLMs can be hard for humans, and what's easy for humans can be hard for LLMs. If you want to use LLMs for coding, it pays to need code they are really good at writing. AI-first development is the big win here. (This is all in Clojure BTW, which isn't really trained for by vendors. The idea that LLMs are ONLY good at, e.g. Python, is absurd.)
• drdaeman 1y ago
  > aren’t able to do junior level tasks without GenAI helping them I’m assuming “unable” means not complete lack of knowledge how to approach it, but lack of detail knowledge. E.g. a junior likely remembers some $algorithm in detail (from all the recent grind), while a senior may no longer do so but only know that it exists, what properties it has (when to use, when to not use), and how to look it up. If you don’t think of something regularly, memory of that fades away, becomes just a vague remembrance, and you eventually lose that knowledge - that’s just how we are. However, consider that not doing junior-level tasks means it was unnecessary for the position and the position was about doing something else. It’s literally a matter of specialization and nomenclature mismatch: “junior” and “senior” are frequently not different levels of same skill set, but somewhat different skill sets. A simple test: if at your place you have juniors - check if they do the same tasks as seniors do, or if they’re doing something different. Plus the title inflation - demand shifts and title-catching culture had messed up the nomenclature.
  • dagmx 1y ago
    I don’t test rote algorithmic knowledge in our coding tests. Candidates can pick their language Candidates can ask for help, and can google/llm as well if they can’t recall methods. I just do not allow them to post the whole problem in an LLM and need to see them solve through the problem themselves to see how they think and approach issues. This therefore also requires they know the language that they picked to do simple tasks , including iterating iterables
    • drdaeman 1y ago
      That’s weird. Any senior developer worth their salt surely should know that LLMs produce a lot of weird nonsense with one-shot prompts, so they need to talk design first, then code the implementation. This said, IMHO one-shot is worth a try because it’s typically cheap nowadays - but if it’s not good (or, for interview reasons, unavailable) any developer should have the skills to break the problem down and iterate on it, especially if all learning/memory-refreshing resources are so available. That’s the skill that every engineer should have. I guess I must take my words back - if that’s how nowadays “seniors” are then I don’t know what’s going on. My only guess is that you must’ve met a bunch of scammers/pretenders who don’t know anything but trying to pass for a developer.
• sktrdie 1y ago
  > then why am I paying for a senior ? Because they know how to talk to the AI. That's literally the skill that differentiates seniors from juniors at this point. And a skill that you gain only by knowing about the problem space and having banged your head at it multiple times.
  • htrp 1y ago
    Except most junior devs will be better than sr devs at wholehearted ai adoption
  • johnnyanmac 1y ago
    >That's literally the skill that differentiates seniors from juniors at this point. If your product has points where llms falter, this use a useless metric. >and having banged your head at it multiple times. And would someone who relied on an LLM be doing this?
  • zelphirkalt 1y ago
    The actual skill is having knowledge and knowing when to not trust the AI, because it is hallucinating or bullshitting you. Having worked on enough projects to have a good idea about how things should be structured and what is extensible and what not. What is maintainable and what not. The list goes on and on. A junior rarely has these skills and especially not, when they rely on AI to figure things out.
• rcxdude 1y ago
  Are you sure you're not just mostly seeing the candidates who are using LLMs to pass through the earlier screening phases with flying colors despite their lack of skills? (i.e., they haven't atrophied, they just weren't very good to begin with). There's always a lot of unqualified applicants to a job but LLMs can make them way more effort to filter out.
  • dagmx 1y ago
    We have the coding test as the second phase now for specifically that reason, and might have more once they’re doing the full interview set.
• bongodongobob 1y ago
  That's what I said about compiled languages. No one knows how to optimize assembly anymore, they just let the compiler do it.
• wyre 1y ago
  Why not let them use LLMs? LLM's are a tool for the job so you wind the find the candidate that can most effectively use that tool in your role. If LLM's struggle with your technologies and API's then a developer that can use an LLM for development with good results should be a desirable thing, right? Can the senior developer understand and internalize your codebase? Can they solve complex problems? If you're paying them to be a senior developer, it likely isn't worth their time to concern themselves with basic data structures when they are trying to solve more complex problems.
  • johnnyanmac 1y ago
    You literally asked the question the GP pre-emptively answered. Read their last paragraph. >Can the senior developer understand and internalize your codebase? Would you trust someone who needs llms in the hiring phase to be able to do this higher order tasks if they can't nail down the fundamentals?
• selinkocalar 1y ago
  This is a real concern. We use AI for a lot of our development work now and I've noticed people will be less likely to dig deep into problems before asking Claude. The trick is using AI to handle the grunt work while still maintaining the critical thinking skills. But it's so easy to slip into autopilot mode.