17 ms·

Bug squash: An underrated interview question

434 points by jez 2y ago

JohnFen 2y ago
I like this approach far, far more than coding tests! > It’s fun. It’s fun in the same way an escape room is fun. It’s fun because of the dopamine you get when the test suite blinks green. Keep in mind that for lots of people in a job interview setting (even excellent candidates) this is not fun, this is stressful. It may be fun on the job or at home, but the stress of a job interview both eliminates any "fun" aspect and makes even simple tasks more difficult. Just mentioning that to encourage some amount of empathy and understanding for the candidate.
- draw_down 2y ago
  The discomfort and stress is a given in any interview. What I think is nice about this is it doesn’t require you to emanate fully-formed, nicely designed code from your brain on the spot. Instead you just have to navigate the codebase, read it and understand it. It’s a better demonstration of knowing how software works imo. But I am biased because this sort of thing is one of my strengths.
- nine_k 2y ago
  This approach then selects those who are relaxed. The candidates that have five more interviews at advanced stages, and likely to receive several offers, as opposed to candidates who badly need a job. The former kind of candidate may be more desirable for the employer :-/
  - shermantanktop 2y ago
    Depends very much on the company. Fishing in the unemployed-and-desperate end of the pool is a real tactic. The odds are good but the goods are odd, as they say.
    - AlexCoventry 2y ago
      What's the goal? Increase headcount at any cost?
      - danielheath 2y ago
        Find the occasional spark of brilliance someplace where others aren't looking?
        shermantanktop 2y ago
        That’s it. Not everyone lives a life that leads to a perfect resume.
        Pikamander2 2y ago
        And for that matter, some people do live that kind of live but still fall through the cracks. Not everyone is lucky enough to graduate during a strong economy, apply to exactly the right jobs at the right time, etc. There's a lot of wasted talent out there that many businesses have little interest in seeking out.
        ajmurmann 2y ago
        This! One of the best developers I've worked with used to be a carpenter who had a kid with Down syndrome. He taught himself programming in his spare time. Smart, extremely driven and knew how good we all had it.
      - ericjmorey 2y ago
        Finding talent
      - Exoristos 2y ago
        Improve the ratio of compensation to workload in the hiring employer's favor.
        2y ago
  - teaearlgraycold 2y ago
    Even if I have no other offers I’m pretty good at maintaining a calm demeanor in an interview. I think it’s served me well over the years.
    - darepublic 2y ago
      I've completely flubbed questions but thinking out loud in a personable way nets a lot of bonus points
  - burnished 2y ago
    Relaxation during an interview can also be a function of practice. Obviously when you're squeezed its more difficult to get that practice in, but if you can it can help tremendously.
  - deleted 2y ago
    [deleted]
- b20000 2y ago
  for me cracking the coding interview is stressful i prefer the bug squash question. i have 20 years of experience. i will probably do well. a fresh grad might fail. do i care? no. leetcode has benefited fresh grads and is being used to discriminate against those with anxiety and other disorders and those who are older. i think it is refreshing to try a different interview method, one that benefits those with experience.
  - ta_1138 2y ago
    I administered bug squash questions a few dozen times. Experience definitely trumped the smartest, most prepared young programmers, because no matter how much they studied, they hadn't really faced a real test suite in a large codebase. We had to rate them on a different scale, because otherwise very few of them passed.
    - b20000 2y ago
      why do young programmers need to pass? nobody gives a shit whether experienced people pass leetcode so give me one good reason why you should rate young programmers differently.
      - strken 2y ago
        A) I, and a lot of other people, do give a shit whether experienced candidates pass leetcode, and try not to give leetcode interviews unless forced to by HR fiat B) As an industry and as a society, it's not a good thing to refuse to hire new grads, since that leads to the kind of labour shortages we see in e.g. medicine and pulls up one of the best ladders for wealth mobility. Tech therefore needs some kind of funnel to evaluate candidates with less experience. A test that almost none of them can pass is not very useful for evaluation purposes
        b20000 2y ago
        it’s not a good thing to refuse to hire older people and most older people do not pass leetcode without significant grinding
- Hamuko 2y ago
  Yeah, I've had a live bug squash interview (SSH into a server where a script is not functioning) and I didn't find it particularly fun. I did manage to get through it and the interviewer seemed impressed, but it's still stressful to do it with someone silently judging you. I'd much prefer take-home assignment and do it at my own pace, but I guess that might be susceptible to "cheating" (whatever you consider it being).
  - sqeaky 2y ago
    Doesn't a take-home task also prevent any chance to ask questions in the middle? And I mean questions both ways. An interviewee can ask questions about the code base that might be common knowledge to people on the team and an interviewer can ask questions about why a thing happened and gain insight into the interviewees thought processes. I get that interviewing is a skill on its own, and not everyone has skills that work in an interview that transfer to the day-to-day job but don't most day-to-day jobs have a large communication component?
    - Hamuko 2y ago
      >Doesn't a take-home task also prevent any chance to ask questions in the middle? Probably yeah. I don't remember my last take-home assignment having communication between the start and the end, although I would have probably been able to send them a question if there was something unclear. >don't most day-to-day jobs have a large communication component? I wouldn't know if mine has a "large" communication component, especially in terms of synchronous communication, since everyone's remote, choosing their own hours and so on. A lot of our work is very independent.
- brainzap 2y ago
  thanks for pointing this out
- StefanBatory 2y ago
  Isn't that what companies would look for, though? They want people who can work under stress. If you had two equal programmers, and one who was stressed and the second who was not, any company would pick the second one without thinking.
  - chii 2y ago
    That assumes that during regular work, you'd be under stress. If it were the military, then yes, i would agree - working while you're being shot at means you need to be able to work under stress. However, a lot of people don't do well under stress, and in many corporate settings, there's usually no real source of stress (except perhaps artificial stress induced by management). Someone might work much better in a stress free environment than someone else under a stressful environment, but you'd then easily miss these candidates by structuring the interview to be stressful.
  - lapcat 2y ago
    All stress is not the same. Interview stress is actually very different from working stress. Ask a firefighter who rushes into burning buildings for a living to give a public speech in front of a large crowd of people in a non-burning building. Which is more stressful? For the firefighter, it may be the latter. Fighting fires is what they train for, what they're experienced with, whereas giving speeches is not. A lot of people, no matter the profession—firefighter, programmer, etc.—have a great fear of public speaking. Likewise, a lot of people, no matter the profession, also have a great fear of job interviews, especially the crazy audition-style interviews of programmers. It's simply human nature, with little bearing on job performance in general. We shouldn't be giving auditions to people who aren't stage performers. Personally, I never work with someone watching me—definitely not a judgmental stranger who will fire me for any perceived peccadillo—nor do I ever have some specific narrow time limit on my work in the range of 15 minutes or so.
    - starkparker 2y ago
      Interview stress is nigh indistinguishable in my experience from a bad customer call. Indeed, if I replaced the bug fix part with writing an effective ticket or test for the issue, this becomes an even better method for tech support, technical account managers, or sales engineers than for developers, unless it's a company where devs regularly interact directly with customers.
- nailer 2y ago
  > It may be fun on the job or at home, but the stress of a job interview both eliminates any "fun" aspect and makes even simple tasks more difficult. Suggestion here: leave the room and come back in 40 minutes. Let them debug, fix the problem, and present to you when they're done.
harimau777 2y ago
I wonder if this is an improvement over a conventional white boarding question. It seems to me that debugging in particular is often dependent on the developer realizing some edge case or scenario where things circumstances line up to produce a bug. In that case, wouldn't a "bug squash" end up being similar to a "gotcha style" white boarding question. The author, quite correctly, says that the interview should be scored not on whether the candidate can immediately vomit up a solution but on whether they can demonstrate their an organized thought process. However, isn't that true of white board questions as well? Overall, it seems to me that the real problem with conventional white board questions is when the interviewer focuses on whether the candidate gets the right answer rather than on whether they demonstrate the ability to problem solve. While it sounds like the author is a good interviewer who focuses on assessing the candidate's thought process, it's not clear to me that giving debugging questions is actually what causes that.
- simonw 2y ago
  "It seems to me that debugging in particular is often dependent on the developer realizing some edge case or scenario where things circumstances line up to produce a bug." I don't think that's the case if there's a test failing. If a well-written test fails, that means there's a completely scoped out requirement for how the feature should work. The candidate should know how to use a debugger, so they should be able to start diving in and figuring out how the code works and what might be going wrong. I don't think that's likely to produce gotchas.
berikv 2y ago
Although I like this interview question, it has a big downside. You make the candidate spend quite a lot of effort while you as an interviewer don’t learn much. The answer to “Can this candidate find and fix this bug?” is not necessarily equal to “Is this candidate a good expansion of the team. What do they bring that we need?”
- ryandrake 2y ago
  It's kind of a fizzbuzz-ish weed-out question: If they can do it, sure, you haven't learned that much, but if they cannot, then you know with pretty high certainty that they're not going to add to the team.
  - aaronbrethorst 2y ago
    Astonishingly, I’ve found fizzbuzz to be a shockingly good weeder question for tech screens.
- Noumenon72 2y ago
  To me this sounds like a relaxing break from the effortful part of the interview.
- ozr 2y ago
  It rarely makes sense to hire for a specific need. I want people that are smart and high agency. Seeing how they approach problems like this is generally enough to tell. I've done similar interviews in the past and they are remarkably high signal.
- jez 2y ago
  The point of the interview is not to answer "Can this candidate find and fix this bug?" but rather "what is the candidate's approach to fixing unknown problems?" A good performance looks like making a hypothesis for where the bug is, testing that hypothesis, and repeatedly narrowing in closer and closer. Finding and fixing the bug is irrelevant! A bad performance might look like - running out of hypotheses for what might be causing the bug - making hypotheses, but never testing them - failing to interpret the result of their test of the hypotheses (thinking it confirms one thing, when it doesn't actually confirm that, or it confirms the opposite)
yas_hmaheshwari 2y ago
This is one of the interview questions at Stripe, and I agree that it was the most "fun" part of the interview (It is still pretty stressful giving an interview, but magnitude times much less so than converting a BST to doubly linked list, and converting it back )
ajbt200128 2y ago
> You’ll need at least one question per language that you want to allow people to interview in. ... “Was this performance poor because they’re bad at debugging, or just unfamiliar with this language’s tools?” Not necessarily a con. Different languages/tech stacks have very different tools. I.e. debugging a GC issue is different than debugging a network issue is different than debugging an embedded issue. If you chose a language for a very good reason, then it's not a con to filter out those who aren't familiar with that language enough to debug an issue in it
- deathanatos 2y ago
  > If you chose a language for a very good reason, then it's not a con to filter out those who aren't familiar with that language enough to debug an issue in it The language is based on/chosen by the candidate. We want to map the candidate to whatever language in our repository of "this question in different langs" is closest, to control for the candidate's familiarity as much as possible. If you're screening to only candidates who can code in the language your company primarily works with, you're missing good candidates. The best are going to be able to pick up a new language rapidly, particularly if your language is a mainstream one that's probably imperative or OO-ish, or both; adding a non-esoteric language to one's repertoire is just not that hard a thing to do. But in the interview, I don't want them struggling with syntactic bull, as it's not a useful signal; I want to know how they think, whether they've seen code before, and can they reason from problem statement to debugged bug.
  - ajbt200128 2y ago
    > If you're screening to only candidates who can code in the language your company primarily works with, you're missing good candidates. I agree with that. I work somewhere that uses a lot of OCaml, we don't screen out those who don't know OCaml, but it usually helps them, since static analysis is easier with an ML family language than say, javascript. > We want to map the candidate to whatever language in our repository of "this question in different langs" is closest If there's not a close language in any of your repositories for the candidate, then I think that's a good signal that candidate may not be a great fit. As mentioned before, we don't necessarily ask candidates to write OCaml, most functional languages will have a similar bug + fix. > problem statement to debugged bug what if the problem you might normally run into is perf issues related to the language? I.e. HFTs usually use C++ and perf = $$$. I would agree that yes, those who have worked on say Rust could be good candidates, but if someone has years of experience in writing highly performant C++, and that's what the job requires, probably easier to look for those candidates. I definitely agree with you in general, just pointing out that there are times where being familiar with the language used is desirable.
Noumenon72 2y ago
I was thinking, wouldn't it be fairer to do the test in an online code sandbox where being able to build and install isn't an issue? Then I thought, isn't letting the candidate use their own machine and debugging tools about as biased as hiring a limo driver based on how fast they can drive their own car? It does tell you who's a real "car guy" or "Unix guy", it just has a slight disparate impact by screening for people with money and hobbies. I mean, I think you should be allowed to hire these people because they will be better at the job, but hopefully they could also show that in a sandbox test.
- singleshot_ 2y ago
  Decent product idea? I’ll stay tuned for your Show HN.
- dingnuts 2y ago
  your job doesn't let you configure your own environment and tools?
- ElectricSpoon 2y ago
  I find editors to be deeply personal things. Throw a vim user in vscode and you might be disappointed, and vice-versa. I don't care about your tools, I care that the sum of you and your tools make you good. I think the good middle ground is having both a canned environment and allowing candidates to use their own... Unless the job is about debugging production systems where only vi is available, in which case that interview might as well represent the actual job.
- rtpg 2y ago
  We would run stuff in repl.it, you get the shared codespace and also don't waste time on the ops issues (which are maybe interesting and you really want devs to be able to handle but also entirely incidental complexity for most devs). In abstract I think it's great to have people use tools their comfortable with, but I dislike dinging people because they write good code but are not good at unborking Python venvs (less of a problem now) (yes you can be a good programmer and still lose time on dumb environment stuff)
- 2y ago
Mistletoe 2y ago
Thought it was going to be about squashing a bug or releasing it outside your house, which would tell me a lot more about a potential employee than this.
franciscop 2y ago
I had a very similar intro experience to dev. I started with some friends as a hobby, and my first paid jobs were when I was hired to do a bunch of small tasks in different projects. Add some animations in this Ember project, fix some bugs in this Wordpress project, add a very specific feature in this Angular project, etc. I probably did a terrible job at that time and I'll always be grateful, but it also taught me A LOT about jumping into a foreign codebase and start working on it almost right away, especially since I was paid hourly and I was super-aware I was paid to fix the problems and not to learn the code. IMHO what this article explains is one of the most valuable bits I retain from that time.
kopirgan 2y ago
I could be wrong but given how fast things change in this business, this seems to test very micro level skills not the learning and adaptability skills. What if a guy could, given the business or technical issue the code solves, write something faster than he could fix others code? May be the job is about heads down coding so this bug fix test is appropriate..
- from-nibly 2y ago
  Then you could do that and showcase that you can write code incredibly fast.
- __float 2y ago
  How is debugging a "micro level skill"? Jumping into an unknown project/system/whatever is something you will likely have to do at some point. (Sometimes it might be code you wrote yourself years ago, but revisiting without any context makes it feel unknown.) Is this hypothetical guy always writing 100% correct code?
vandyswa 2y ago
I've done the equivalent of this by asking them to describe an interesting bug they've encountered in past lives; how it came up, how they hunted it, how they fixed it. By listening to them describe the work, asking questions, and following their thought processes, you can come to a fairly good hire/no from this single walk-through. I know, it's short and humane, so not a good fit for current Sillycon Valley culture.
- sk11001 2y ago
  There’s no chance that I can come up with a good bug story on demand without any preparation.
  - tdeck 2y ago
    For past experience I reviews like this it's a good idea to give the candidate advance notice.
  - serpix 2y ago
    compare this with the technical interview bomb of demanding you write a full blown program right fuking now, complete with tests. Which one do you prefer?
    - 2rsf 2y ago
      I prefer to describe a bug in details, but it is hard to come up with the details on the fly without prior preparation
    - burnished 2y ago
      Writing the program. Recalling a bug for the purpose of impressing a stranger does not sound fun.
  - izacus 2y ago
    Ok, but why do you think you should be hired in that place though? :)
deleted 2y ago
[deleted]
aray07 2y ago
I did a bug squash interview when I was interviewing for new grad positions and it was one of my favorite interviews (both as an interviewee and an interviewer) - People were allowed to use their favorite IDEs - so you could see how proficient some people were - Great engineers are really really good at debugging - it was great to see how people debugged and it helped me pick up a few things as well - People couldn't leetcode their way out of this
boredtofears 2y ago
I’ve always liked the Gilded Rose refactoring[1] which is similar but about improvements/feature requests instead. [1] https://github.com/emilybache/GildedRose-Refactoring-Kata/blob/main/GildedRoseRequirements.md https://github.com/emilybache/GildedRose-Refactoring-Kata/bl...
jpmoral 2y ago
Had an interview question like this where the interviewers were very impressed. Said it was the fastest and cleanest they'd seen anyone do it, and in an unfamiliar framework no less. Unfortunately it wasn't enough for me to get the job. I had a couple of answers in the next section they didn't quite like.
- __float 2y ago
  > I had a couple of answers in the next section they didn't quite like. Could you say more about this? Were they technical or behavioral interviews?
  - jpmoral 2y ago
    Technical. IIRC: - I did the system design at the wrong level of abstraction (too low) - Got asked about lowish-level database details. I took the question at face value and said I didn't know. What I should have done was explain what I did know (which was at a level slightly higher than the question), make an educated guess, and say how I would find the information. I still don't understand why Id didn't do that as I've always done it in interviews before or since. This is my own assessment, they didn't give detailed feedback. Just that I wasn't strong enough in some areas (paraphrasing).
michaelteter 2y ago
This is indeed a good interview challenge, and as a candidate I’ve enjoyed taking it the couple of times I have encountered it. I also like the related challenge of, “add a feature that does X” to a codebase.
lr4444lr 2y ago
How much time do you allot for this thing? Cloning the repo, dependency installs, server/DB setups, possible build steps, configuring the IDE to the project ... even if the whole thing is containerized, just going from nothing to running code in an IDE alone could be expected to take 20-30 minutes of the interview.
- ibash 2y ago
  You just need to rewrite the dev tools in rust! Joking aside, modern fast tools like bun can make this type of interview way more viable. I once conducted this type of interview where the candidate had to install node on their windows laptop and... we spent most of the time debugging their install :/
  - photon_lines 2y ago
    FYI just small tip on the Windows / Node front, you should just install & use NVM instead of Node directly - will make many headaches go away.
- s17n 2y ago
  In my experience doing these interviews in Node and Python, it takes about 5 minutes to get running. If it takes more than that... maybe use a different codebase.
  - lr4444lr 2y ago
    What exactly is the interviewer looking for? Familiarity with tools, or actually debugging skill? I like the idea of debugging questions, I'm just not convinced all this setup and "use your own tooling" is necessary,and that a long enough snippet of given whiteboard code sample can morr effciently test for debugging skill. I can always teach (and learn!) new tools, and frankly, there is value to me as an interviewer to see someone manage without his tools.
- __float 2y ago
  The bugs are chosen in projects that don't require this much setup. (There is no server/database, they use standard tools for the given language, and they are generally solvable without needing an elaborate IDE/debugger setup. Of course that may help, and it's why you bring your own computer and choose the language ahead of time.)
tdeck 2y ago
I somewhat disagree that this is reflective of a person's everyday work. Usually we're only ever working on at most handful of codebases at a time, and getting oriented in a completely new application (including getting it to build and run on your machine) doesn't happen every day. I've done interviews like this and one major pitfall is the start-up time. It's possible to spend an unreasonable amount of time debugging cold-start issues with getting the repo set up, dependencies installed, and the app to build while the interview slips away. These things are representative of the first week on a new team or project, maybe. Figuring out why bundler can't seem to download the dependency in the gemfile isn't my idea of a good use of interview time.
rtpg 2y ago
At one place there was a bug squash interview like this. Rough idea was we wrote a very small version of a system we had in our app (some data-syncing-then-displaying service), with a handful of bugs and a very simple feature request. It was very helpful for sanity checking if a person was able to be in front of a computer. There's a bit of a challenge because I think there's a pretty low ceiling of performance (strace on a Python program is fun, but our bugs are not that deep!), but we could use it to also talk about new features on this mini-system and whatnot. General feeling on it was "this is a great filter", because ultimately we needed people to just pass above a minimum skill bar, not be maximally good at coding. And having the exercise really helped to communicate the day-to-day (which can be tedious!)
- bluGill 2y ago
  I've interviewed a few people who have a great resume and came off well in the interview, but I was left wondering if they could write code. (HR only allows me to ask specific research based questions in an interview and so it is easy to answer well without giving any indication you can write code). We now have some filter exercises that are about if you can write code at all.
- Izkata 2y ago
  > (strace on a Python program is fun, but our bugs are not that deep!) They don't need to be deep to be useful. I once used it on nodejs on a hunch, looking for anything that felt "off", and discovered it was accessing settings files I had no idea even existed. Turned out the other dev was having problems because of one in his home directory the rest of us didn't have.
- tharibo 2y ago
  This. Keep in mind people that a job interview is to check wether you want to work with the person, not to judge if they "pass" the test. Or to say it differently, they don't have to find and fix the bug, they have to demonstrate ability with the tools, a good attitude, a match with your work culture.
flurie 2y ago
I'm not too proud to admit I took the general concept from https://sadservers.com/ https://sadservers.com/ and turned it into something I could use for interactive debugging interviews. I had golden images with some scenarios I wrote myself, and the images automatically shared a tmux session over http so I could follow along without requiring the candidates to screen share. I did have to ask for IPs so I could ensure the machines were just available to me and the candidate, but it was otherwise pretty seamless! Though now that https://sadservers.com/ https://sadservers.com/ has a paid service it might be worth looking into.
- fduran 2y ago
  Hello, SadServers author here. Zero shame in adapting an idea you found somewhere; I'm super happy you built something useful. I've thought of adding programming debug scenarios (I even got sadbugs.com lol), may implement in the future. I'd love to see what you've done, please feel free to connect :-)
dangsux 2y ago
[dead]
deleted 2y ago
[deleted]
zug_zug 2y ago
I've only had one "find the bug" interview, and it was awful: - Didn't set you up with a way to reliably run/test the code, nor a step-through-debugger which, jeez is it 1980? Like setting up a repo in a language you aren't familiar with (say getting your py-env exactly right) can take a whole hour if it's not your main language. - Didn't have standardized questions, which is hugely problematic (some bugs are 2 orders of magnitude harder than others) - It also just seems like there's a huge random element, like am I debugging code written by somebody who thinks at the same level of abstraction as me? Did they write comments I understand?
- trevor-e 2y ago
  Yea there's a huge element of randomness to these kinds of interviews. I wasn't a big fan of them at Stripe. You can get some signal like: - do they methodically approach the problem? - do they understand the bug? - do they know how to use debugging tools? - do they know advanced debugging tools? - do they work well in an unfamiliar codebase? But in practice I'd say most candidates check those boxes, at which point it becomes an arbitrary evaluation unless you pass/fail them based on whether they solved the bug.
  - zug_zug 2y ago
    As it happens the place I had a really bad experience with it was stripe. I had pretty much aced every other question, but basically was setup with a problem where I didn't have a way to reliably get it running (and I wasn't even clear what the app was supposed to do, it was just some random project off of github). Fortunately I already had an offer onhand from facebook.
  - leoqa 2y ago
    I gave this interview 100+ times and my criteria boiled down to: did they propose a hypothesis and then follow it to a conclusion? Doesn’t matter if it was wrong, just mattered that they were able to falsify it or find the solution. At the end, I would have them walk me through the bug, its cause, and the fix in very high level terms to make sure they could articulate the path we took.
- 2y ago
0x20cowboy 2y ago
If you can’t sit down with someone, have a simple tech conversation with them and be able to tell if they know what they are doing… if you need to “give someone a test”… the problem is likely not the candidate.
- tedunangst 2y ago
  If all I needed to do was express my thoughts out loud, I wouldn't need to hire anyone. I can do that myself.
- cheeze 2y ago
  On the flipside, I've spent an hour with a candidate who seemed great, but could hardly _actually program_. It was a bizarre dichotomy. Dude was clearly smart as hell but was so caught up in building perfect abstractions in parts of our software that _didn't matter all that much_ using languages that nobody else knew or wanted to use. I get it, your fun language is fun. But if the dev shop is 60% one language and 39% another, I don't really care about the small improvements. You need a strong reason that it actually _helps the business_
  - caseyy 2y ago
    Ask them to explain what 4-5 different bits of code do. When they do, ask them to explain it like they would to a junior and to a senior/principal/fellow. It works very well and takes only an extra 10 minutes on top of the conversation.
- marcinzm 2y ago
  That biases heavily to candidates with good people skills and the skill to take over a conversation. Yes, you are not in fact immune to this even if you think you are, that’s why it’s so dangerous.
  - 0x20cowboy 2y ago
    I disagree. That would indicate conversations skills equate to knowing what you are talking about. "Yes, you are not in fact immune to this even if you think you are, that’s why it’s so dangerous." I think that I am. I'll think about that a bit more. I don't care for popular people so, in fact, my bias might be the other direction. However, I don't think I have ever met a person who I thought was technical and proficient, but turned out not to be. I have been forced to hire people I knew were not proficient because they passed a test, but not the other way around.
dave333 2y ago
A similar exercise but much easier to do in an interview is to give a short piece of code that does something a bit non-trivial and ask them to write (paper or whiteboard) test cases for it. Can then either run the test cases and see if they can find (all) the bug(s), or simply review vs a known list. Classic example in Myers The Art Of Software Testing is a method that takes 3 numbers as the length of the sides of a triangle and prints out whether it is scalene, isosceles, or equilateral. Many people miss a lot of corner cases.
doawoo 2y ago
Triplebyte (one of the best hiring experiences I ever had) gave me this during their initial interview. They dropped an archive of a medium-ish codebase to my machine, that had failed unit tests. My task was to simply fix the code so the tests passed. Not only did I feel engaged with the interview because I could speak aloud as I debugged, I also found it fun!
- Etheryte 2y ago
  Too bad that Triplebyte then turned around and decided to sell everyone's data after it turned out their business was not scalable after all.
- awkii 2y ago
  You're describing the parallelized web-scraper that they pointed to their own internal site? Yeah that was fun. Too bad everyone got the same question.
deleted 2y ago
[deleted]
halfcat 2y ago
”what we’re observing is X, but we see Y instead.” What? Anyway, I’ve found these kind of interview questions rarely helpful because the scenario is either overly simplistic, or it’s some obscure bug they identified (which they only caught because it caused an outage, i.e. they also didn’t “solve” this when they wrote it). A similarly poor interview question happens in IT operations when the interviewer asks super specific details about some CVE from 5 years ago that he’s particularly proud of pulling an all nighter to patch a bunch of servers.
- __float 2y ago
  > what we’re observing is X, but we see Y instead. I think this might be a typo - we're observing X but want Y, or some similar variation. In any case, there's a bug.
- jez 2y ago
  Sorry about that! edited to > What we're observing is X, but we want to see Y instead.
datavirtue 2y ago
It's easy to write this question. I can cut a branch with gnarly UI bugs in it, no sweat.
ryoshu 2y ago
One of my favorite interviews was like this, but it was a long time ago and they printed out the code and told me there was a problem. Here's a pencil. What's wrong? Most were a function call or two. In one case I remember someone was doing a loop that incremented the index of a SQL query in code and using each result, instead of querying the set and looping through it in the code. The fun part was it was a discussion. Two people, paper and pencil, talking about the code. And the examples were from bugs they'd already squashed in a code base they inherited. It was a project that I ended up working on that had... quite a few interesting bugs like that.
mistercow 2y ago
> It’s easy for the candidate to self-assess their own progress.⊕ When the candidate isn’t doing well on it, they probably already knew it without needing to be told as much by the recruiter. This is a much better candidate experience than the whiplash of thinking you solved a question perfectly only to realize that the interviewer was looking for something else entirely. Not to say that I think this is a bad type of question overall, but IMO, this is an anti-feature. The candidate does not need to accurately self-assess their performance on the day. They need to have their confidence preserved so they don’t tilt and tank the signal for the entire interview round.
- bawolff 2y ago
  I don't know, i think the best questions are the ones where candidates fully understand what is being asked about them. If they aren't able to self-assess did they really understand the question?
  - kenschu 2y ago
    Candidates walking away pissed is itself also a problem. A significant percentage of candidates avoid buying from companies they're rejected from [1]. They also love sharing their poor interview experience with future potential applicants. [1] https://www.wayup.com/employers/blog/how-a-positive-candidate-experience-affects-company-revenue https://www.wayup.com/employers/blog/how-a-positive-candidat...
    - neilv 2y ago
      > A significant percentage of candidates avoid buying from companies they're rejected from [1]. When rejected, or from an otherwise negative experience. It's not necessarily negative emotional associations. Usually it's that I picked up on signal that the company has serious problems -- either overall, or with a key person -- which suggests I shouldn't depend on the company as a vendor.
  - mistercow 2y ago
    What you want is to prevent floundering death spirals. They need to understand the criteria, but if you need to hint and nudge them, they need to not feel like that means they’re bombing. Of course, you can absolutely do that in a bug squash interview, but my point is that it defeats this claimed advantage.
bawolff 2y ago
In computer security, this sort of interview often takes the form of find the vulnerability in this app. I like it. It feels like much more directly measuring skills than most interview questions.
zgs 2y ago
Kind of like the "install this software" as part of the interview process. "My hourly rating for figuring out your bug is $X".
maxwelljoslyn 2y ago
My kingdom for this approach to take the place of (some/all) leetcodes.
Brystephor 2y ago
I had a bug squash interview today. I found it nice, but also frustrating. It was nice because I didn't need to practice and I knew exactly how to debug the thing. It was frustrating because my personal laptop is from 7 years ago (from college), is slow, and the dependencies and editor don't work out of the box for an a new repo. Additionally, I'd prefer to use IntelliJ like I do at work but again, that's too heavy for my computer to handle so I resort to vscode and have to figure out how to use it. So then the interview becomes debugging my environment instead of debugging the problem. Maybe that's a useful signal, but it's not really bug squashing anymore then. So overall, it was still requiring learning but there was not a very good way to test in advance (how do you test all possible repo structures?)
jdlyga 2y ago
This sounds like a lot of fun to me. A lot of people dislike debugging, but I always enjoyed it. It's a bit of a murder mystery. You not only need to understand what's happening several layers beyond the initial problem, but why the code may have been written the way it is even when you find the issue.
Twirrim 2y ago
My favourite thing to do for "coding" interviews is to give the candidate a piece of absolutely awful python code that a friend of mine came up with for use for interviews. There are code smells, side effects, errors, confusing syntax, a whole slew of things in it. I give them the code, tell them I'll help with every bit of python syntax (and really emphasise that I don't mark people down _at all_ for any lack of familiarity with python / python syntax), and ask them to work through the code and do a code review. There's some python specific quirks that I largely consider bonus points at best, but anyone with any appreciable familiarity with at least one language should be able to spot a number of things. As they call stuff out, I'll dig in (or not) as seems appropriate. So far it seems to be working out great for me as an interviewer. Candidates seem to be way less stressed than they are with a "whiteboard coding" exercise. Good discussions are showing me their development skills, etc.
- bigbong 2y ago
  Would you mind sharing it?
- esperent 2y ago
  It does seem a bit unfair - ok sure, anyone familiar with C-family syntax should be able to work through it and spot errors. But anyone already familiar with Python would be able to do so a lot faster. I think the idea is good, but to be equitable it should be given in a language the person claims to be already familiar with. Or alternatively, only give it to people in a language they are not familiar with.
  - phil-martin 2y ago
    If the goal was to objectively evaluate ability in a broad population, I 100% agree. However, if my dev shop primarily uses Python then selecting for people familiar with Python is pretty reasonable. Thankfully there is a giant corpus of terrible code out there for any language. Especially that code written by the most evil terrible coders of all time: past-self :D
- naught0 2y ago
  I'm dying to see this code
AsthmaBoy 2y ago
I like this approach. We did something similar, but for a position as support engineer, where coding ability was one of the requirements, though not the only one. They were given a piece of code on a toy, easy to read, english-like pseudo-language, and were asked to examine it and explain what the output to a specific input would be. They were told that we would answer any question, except what the answer was or what the code was doing, and encouraged them to explain their reasoning as they went ahead. We wanted to asses the candidates ability to problem solve, communicate, how would they react under preassure, and how they approached the issue. We were not necessarily looking for perfect answers, we rather rated the candidates willingness to ask for help when stuck, how well they communicated while doing the task, whether they tried different approaches, their ability to grasp what an unknown piece of code was doing and so on... all important properties for a role supporting mission critical solutions with near to zero down-time requirements. Our reasoning was that technical knowledge is easy to acquire over time, but problem solving skills, how candidates tackle difficult and unknown challanges were more important for the role. In my current role I try to design interview questions like that, looking both for the candidates current skills as well as their future potential. Not easy, but IMHO more rewarding for us and them in the long run.
RomanPushkin 2y ago
The only downside I see is that I need to share my screen with a person I don't know. Don't get me wrong, but I normally have access to some crypto on my laptop, banking notifications turned on for my 2FAs, private messages I'm getting from my spouse, maybe some borderline NSFW chatgpt dialogs, screenshots of my ideas, emails, private notes, personal projects, whole bunch of tabs I am not willing to close or share, etc. I don't have issues to show the screen to a friend of mine. Or showing the screen of a work computer to a colleague. However, demoing the whole screen is violation of privacy. Please don't do that, unless you grant VNC access to a machine with a set of popular tools, code editors, etc. How about reverse bug squash? Interviewer shares the screen with all his/her private info, notes, tabs, development environment settings, shell command history (with maybe autosuggestions turned on), and the interviewee needs to guide the interviewer towards finding the bug?
- yonrg 2y ago
  True, this would discomfort me, too. Also in webconferences, I never share the full screen, only one window. This should be enough for bug squash as well.
- tdeck 2y ago
  This is common in many interviews, not just "bug squash" interviews. I recommend having a plan for it if you're interviewing. Perhaps dedicate a virtual desktop to interviews if your system supports it.
  - RomanPushkin 2y ago
    Thanks for recommendation. I just finished my rounds of interviews, gotten a decent offer (SF company), being 20 years in industry. I have my own rules, and luckily they align very well with industry standards. So I never had any issues not following the scenario when I need to share something to land a job. I also don't do any take-home assessments, and moreover, I only use one programming language while interviewing. Never had any issues with that. I think I don't need a plan or virtual desktop. I just say "no" if it doesn't work for me. Unless I am really desperate, I do not share.
aaronbrethorst 2y ago
This is essentially what I do with candidates. I ask them to pair with me on adding features to a codebase, and also to offer a code review to a PR. 100% success rate on candidates thus far.
gecko6 2y ago
I have an interview question: step 1: the candidate is shown the specification for a method and the results of running the test suite on an obfuscated version of the method. All tests pass. The test suite is minimal, and test coverage is abysmal. step 2: the candidate is asked to come up with more test cases, based on the specification. The code is run against the updated test suite - most new tests will fail because the method's implementation has several known bugs. step 3: The un-obfuscated source is provided and the candidate is asked to correct any bugs they discover. step 4: the changed source is run against a full test suite. I like this because the candidate's ability to think of test cases, debug, and fix existing code are all tested for.
- pastage 2y ago
  This has failed me because code get pretty gnarly fast, many things I thought was obvious was missed by people smarter than me. I think as long as it is a tool for discussions rather than grades it is good though.
- saagarjha 2y ago
  Curious if any interviewee has tried to unobfuscate the code.
- eru 2y ago
  Do you allow property based testing?
ReleaseCandidat 2y ago
> It’s fun because fixing self-contained, reproducible bugs like this is what so many of us enjoy the most about software engineering in the first place. Do really "many of us" enjoy that? In the "day-to-day business", not interviews.
jb1991 2y ago
I am quite skeptical of item 6 in this list, but otherwise it looks pretty good: > Cheating effectively is indistinguishable from debugging skill. If you know the code or problem ahead of time and how to fix it, you can certainly be convincing in your walk-through of how to do it.
pyromaker 2y ago
Would love for others to share non-technical questions that can filter bad candidates!
globular-toast 2y ago
I like this idea but I'm curious how others would implement it. Does the developer have to write the failing test themselves? Are they instructed to do it first, or is that behaviour worthy of bonus points? (Most write a passing test afterwards, which is worse). > I’ve seen candidates wield their text editor like it was an extension of their fingertips. So am I allowed to install my own text editor and other tools (Emacs with my own config that requires some build time)? Or is it on my own laptop (I don't have one)? It seems unfair if some candidates get to use their favourite tools and others don't.
eru 2y ago
Compare and contrast https://sockpuppet.org/blog/2015/03/06/the-hiring-post/ https://sockpuppet.org/blog/2015/03/06/the-hiring-post/
em1sar 2y ago
[dead]
steventhedev 2y ago
Having done nearly 100 technical interviews (with multiple questions each), the bug squash question gave us the biggest signal on candidates. Our question was far simpler: it was a simple class (java + python variants, no fancy syntax) and ask them to describe what it does, then find the bug, and finally ask them what they would change. It reflects a true test of what the day to day is, and whether or not the candidate would succeed in the role.
Pikamander2 2y ago
I've written some interview questions before and can confirm that having people find and fix simple issues in code is a surprisingly great way to weed out people who have no clue what they're doing. > Cheating effectively is indistinguishable from debugging skill. Even with knowledge of the exact bug ahead of time, if you just open the file with the bug, barf out the code to fix it, and run the tests, you’re going to fail the interview. Did you mean "distinguishable" here, by chance? The first sentence seems to contradict the second.
sethammons 2y ago
"Work Sample Test" Our best received interview question was in this style. Pick something your team fixed or did, distilled down do a 15min thing a teammate could do. Give the candidate 3x the time. For us, it was a db value not being set as expected during a cron. The candidate got our forged bug reports, cloned a repo, ran the script, verified the unexpected db state, and off to the races
ChrisMarshallNY 2y ago
I agree. Great question. Academic LeetCode exercises are really not what I consider useful in evaluating candidates. However, I'm biased. I'm really, really good at finding and fixing bugs, and pretty much stink at LeetCode, so take my support with a grain of salt. One problem is that, if the exercise gets known, expect an underground economy of solution cheats. Same with LeetCode, but that's sort of expected. Bug fix solutions hit harder.
janaagaard 2y ago
I agree. The best interview experience that I had was to be sat in front of a small app that had lots issues, both big and small, and also a lot an not-really-issues-but-still like inconsistent code styles and names, and code comments and code that weren't aligned. This was a great starting point for both me and the interviewer. The interviewer had a list of things to ask into about the code, so that we didn't get stuck.
bluGill 2y ago
There is academic research on how to interview. Most comments here give no indication the commenter is even aware it exists much less what it says. I'm aware this exists, but I'll admit to not having read it directly. (HR gives me a list of questions I'm allowed to ask in an interview and they tell me those questions are based on research but I only have their word they are)
giantg2 2y ago
I've never done a bug squash interview. I have done PR review interviews that are similar - given a PR that technically runs but has tons of problems, then see how many of the problems the candidate identities. I really liked that interview.
rurp 2y ago
I was given a similar interview problem and thought it was one of the better interviews I've had. In my case the interviewer pulled up a web app and said this particular page is loading too slow, how would you approach the problem? We went from opening dev tools to look at the requests driving the page all the way through database optimizations and every layer in between. This kind of interview didn't feel like a gotcha and was much much closer to real world work than the toy and/or algorithm problems that I have encountered. More companies should adopt these types of interview approaches.
aeternum 2y ago
>Cheating is indistinguishable from debugging skill. Even with knowledge of the exact bug ahead of time, if you just open the file with the bug, barf out the code to fix it, and run the tests, you’re going to fail the interview. Umm hopefully you don't fail for this. I've definitely known and hired devs that can do this with surprising frequency. Just because the interviewer may have taken hours to find the bug doesn't mean someone else must.
nailer 2y ago
> I’ve seen candidates drop into strace to debug a Ruby program strace is great. 99% of the time, your program is looking for a missing file. A pity it's so hard to use in modern macOS due to OS integrity protection.
mark-r 2y ago
I ended up doing this for an entire week, once. I was hired and presented with a system that didn't work. I knew nothing whatsoever about their proprietary system and wasn't even given access to their codebase. I wondered why they were being so spectacularly unhelpful with onboarding tasks. At the end of the week, I found out - the whole thing had been a test to see how well I would do, and needless to say I failed miserably. They told me I was a bad fit, and I had to agree, but probably not for the same reasons they were thinking. I really dodged a bullet with that one.
rudnevr 2y ago
it's kind of strange to me that ad-hoc debugging is considered such a valuable skill. I thought it's mostly a juniors' perspective. I typically set up extended loggers and logger methods, write everything to the some formatted file, and have a diffable, persistent, provable, multithread-friendly and versioned bug demonstration, which scales well to bugs of any complexity. (I once found some 10 bugs in a pretty old and tested bond calculation engine while migrating it to the cloud, which nobody could initially believe.)
GoToRO 2y ago
Unrelated: why there are no stories about interview questions for managers?
darepublic 2y ago
Ah the humble 2 point bug ticket

434 points by jez 2y ago

I like this approach far, far more than coding tests! > It’s fun. It’s fun in the same way an escape room is fun. It’s fun because of the dopamine you get when the test suite blinks green. Keep in mind that for lots of people in a job interview setting (even excellent candidates) this is not fun, this is stressful. It may be fun on the job or at home, but the stress of a job interview both eliminates any "fun" aspect and makes even simple tasks more difficult. Just mentioning that to encourage some amount of empathy and understanding for the candidate.

draw_down 2y ago
The discomfort and stress is a given in any interview. What I think is nice about this is it doesn’t require you to emanate fully-formed, nicely designed code from your brain on the spot. Instead you just have to navigate the codebase, read it and understand it. It’s a better demonstration of knowing how software works imo. But I am biased because this sort of thing is one of my strengths.
nine_k 2y ago
This approach then selects those who are relaxed. The candidates that have five more interviews at advanced stages, and likely to receive several offers, as opposed to candidates who badly need a job. The former kind of candidate may be more desirable for the employer :-/
- shermantanktop 2y ago
  Depends very much on the company. Fishing in the unemployed-and-desperate end of the pool is a real tactic. The odds are good but the goods are odd, as they say.
  - AlexCoventry 2y ago
    What's the goal? Increase headcount at any cost?
    - danielheath 2y ago
      Find the occasional spark of brilliance someplace where others aren't looking?
      - shermantanktop 2y ago
        That’s it. Not everyone lives a life that leads to a perfect resume.
        Pikamander2 2y ago
        And for that matter, some people do live that kind of live but still fall through the cracks. Not everyone is lucky enough to graduate during a strong economy, apply to exactly the right jobs at the right time, etc. There's a lot of wasted talent out there that many businesses have little interest in seeking out.
        ajmurmann 2y ago
        This! One of the best developers I've worked with used to be a carpenter who had a kid with Down syndrome. He taught himself programming in his spare time. Smart, extremely driven and knew how good we all had it.
    - ericjmorey 2y ago
      Finding talent
    - Exoristos 2y ago
      Improve the ratio of compensation to workload in the hiring employer's favor.
      - 2y ago
- teaearlgraycold 2y ago
  Even if I have no other offers I’m pretty good at maintaining a calm demeanor in an interview. I think it’s served me well over the years.
  - darepublic 2y ago
    I've completely flubbed questions but thinking out loud in a personable way nets a lot of bonus points
- burnished 2y ago
  Relaxation during an interview can also be a function of practice. Obviously when you're squeezed its more difficult to get that practice in, but if you can it can help tremendously.
- deleted 2y ago
  [deleted]
b20000 2y ago
for me cracking the coding interview is stressful i prefer the bug squash question. i have 20 years of experience. i will probably do well. a fresh grad might fail. do i care? no. leetcode has benefited fresh grads and is being used to discriminate against those with anxiety and other disorders and those who are older. i think it is refreshing to try a different interview method, one that benefits those with experience.
- ta_1138 2y ago
  I administered bug squash questions a few dozen times. Experience definitely trumped the smartest, most prepared young programmers, because no matter how much they studied, they hadn't really faced a real test suite in a large codebase. We had to rate them on a different scale, because otherwise very few of them passed.
  - b20000 2y ago
    why do young programmers need to pass? nobody gives a shit whether experienced people pass leetcode so give me one good reason why you should rate young programmers differently.
    - strken 2y ago
      A) I, and a lot of other people, do give a shit whether experienced candidates pass leetcode, and try not to give leetcode interviews unless forced to by HR fiat B) As an industry and as a society, it's not a good thing to refuse to hire new grads, since that leads to the kind of labour shortages we see in e.g. medicine and pulls up one of the best ladders for wealth mobility. Tech therefore needs some kind of funnel to evaluate candidates with less experience. A test that almost none of them can pass is not very useful for evaluation purposes
      - b20000 2y ago
        it’s not a good thing to refuse to hire older people and most older people do not pass leetcode without significant grinding
Hamuko 2y ago
Yeah, I've had a live bug squash interview (SSH into a server where a script is not functioning) and I didn't find it particularly fun. I did manage to get through it and the interviewer seemed impressed, but it's still stressful to do it with someone silently judging you. I'd much prefer take-home assignment and do it at my own pace, but I guess that might be susceptible to "cheating" (whatever you consider it being).
- sqeaky 2y ago
  Doesn't a take-home task also prevent any chance to ask questions in the middle? And I mean questions both ways. An interviewee can ask questions about the code base that might be common knowledge to people on the team and an interviewer can ask questions about why a thing happened and gain insight into the interviewees thought processes. I get that interviewing is a skill on its own, and not everyone has skills that work in an interview that transfer to the day-to-day job but don't most day-to-day jobs have a large communication component?
  - Hamuko 2y ago
    >Doesn't a take-home task also prevent any chance to ask questions in the middle? Probably yeah. I don't remember my last take-home assignment having communication between the start and the end, although I would have probably been able to send them a question if there was something unclear. >don't most day-to-day jobs have a large communication component? I wouldn't know if mine has a "large" communication component, especially in terms of synchronous communication, since everyone's remote, choosing their own hours and so on. A lot of our work is very independent.
brainzap 2y ago
thanks for pointing this out
StefanBatory 2y ago
Isn't that what companies would look for, though? They want people who can work under stress. If you had two equal programmers, and one who was stressed and the second who was not, any company would pick the second one without thinking.
- chii 2y ago
  That assumes that during regular work, you'd be under stress. If it were the military, then yes, i would agree - working while you're being shot at means you need to be able to work under stress. However, a lot of people don't do well under stress, and in many corporate settings, there's usually no real source of stress (except perhaps artificial stress induced by management). Someone might work much better in a stress free environment than someone else under a stressful environment, but you'd then easily miss these candidates by structuring the interview to be stressful.
- lapcat 2y ago
  All stress is not the same. Interview stress is actually very different from working stress. Ask a firefighter who rushes into burning buildings for a living to give a public speech in front of a large crowd of people in a non-burning building. Which is more stressful? For the firefighter, it may be the latter. Fighting fires is what they train for, what they're experienced with, whereas giving speeches is not. A lot of people, no matter the profession—firefighter, programmer, etc.—have a great fear of public speaking. Likewise, a lot of people, no matter the profession, also have a great fear of job interviews, especially the crazy audition-style interviews of programmers. It's simply human nature, with little bearing on job performance in general. We shouldn't be giving auditions to people who aren't stage performers. Personally, I never work with someone watching me—definitely not a judgmental stranger who will fire me for any perceived peccadillo—nor do I ever have some specific narrow time limit on my work in the range of 15 minutes or so.
  - starkparker 2y ago
    Interview stress is nigh indistinguishable in my experience from a bad customer call. Indeed, if I replaced the bug fix part with writing an effective ticket or test for the issue, this becomes an even better method for tech support, technical account managers, or sales engineers than for developers, unless it's a company where devs regularly interact directly with customers.
nailer 2y ago
> It may be fun on the job or at home, but the stress of a job interview both eliminates any "fun" aspect and makes even simple tasks more difficult. Suggestion here: leave the room and come back in 40 minutes. Let them debug, fix the problem, and present to you when they're done.

harimau777 2y ago

I wonder if this is an improvement over a conventional white boarding question. It seems to me that debugging in particular is often dependent on the developer realizing some edge case or scenario where things circumstances line up to produce a bug. In that case, wouldn't a "bug squash" end up being similar to a "gotcha style" white boarding question. The author, quite correctly, says that the interview should be scored not on whether the candidate can immediately vomit up a solution but on whether they can demonstrate their an organized thought process. However, isn't that true of white board questions as well? Overall, it seems to me that the real problem with conventional white board questions is when the interviewer focuses on whether the candidate gets the right answer rather than on whether they demonstrate the ability to problem solve. While it sounds like the author is a good interviewer who focuses on assessing the candidate's thought process, it's not clear to me that giving debugging questions is actually what causes that.

simonw 2y ago
"It seems to me that debugging in particular is often dependent on the developer realizing some edge case or scenario where things circumstances line up to produce a bug." I don't think that's the case if there's a test failing. If a well-written test fails, that means there's a completely scoped out requirement for how the feature should work. The candidate should know how to use a debugger, so they should be able to start diving in and figuring out how the code works and what might be going wrong. I don't think that's likely to produce gotchas.

berikv 2y ago

Although I like this interview question, it has a big downside. You make the candidate spend quite a lot of effort while you as an interviewer don’t learn much. The answer to “Can this candidate find and fix this bug?” is not necessarily equal to “Is this candidate a good expansion of the team. What do they bring that we need?”

ryandrake 2y ago
It's kind of a fizzbuzz-ish weed-out question: If they can do it, sure, you haven't learned that much, but if they cannot, then you know with pretty high certainty that they're not going to add to the team.
- aaronbrethorst 2y ago
  Astonishingly, I’ve found fizzbuzz to be a shockingly good weeder question for tech screens.
Noumenon72 2y ago
To me this sounds like a relaxing break from the effortful part of the interview.
ozr 2y ago
It rarely makes sense to hire for a specific need. I want people that are smart and high agency. Seeing how they approach problems like this is generally enough to tell. I've done similar interviews in the past and they are remarkably high signal.
jez 2y ago
The point of the interview is not to answer "Can this candidate find and fix this bug?" but rather "what is the candidate's approach to fixing unknown problems?" A good performance looks like making a hypothesis for where the bug is, testing that hypothesis, and repeatedly narrowing in closer and closer. Finding and fixing the bug is irrelevant! A bad performance might look like - running out of hypotheses for what might be causing the bug - making hypotheses, but never testing them - failing to interpret the result of their test of the hypotheses (thinking it confirms one thing, when it doesn't actually confirm that, or it confirms the opposite)

yas_hmaheshwari 2y ago

This is one of the interview questions at Stripe, and I agree that it was the most "fun" part of the interview (It is still pretty stressful giving an interview, but magnitude times much less so than converting a BST to doubly linked list, and converting it back )

ajbt200128 2y ago

> You’ll need at least one question per language that you want to allow people to interview in. ... “Was this performance poor because they’re bad at debugging, or just unfamiliar with this language’s tools?” Not necessarily a con. Different languages/tech stacks have very different tools. I.e. debugging a GC issue is different than debugging a network issue is different than debugging an embedded issue. If you chose a language for a very good reason, then it's not a con to filter out those who aren't familiar with that language enough to debug an issue in it

deathanatos 2y ago
> If you chose a language for a very good reason, then it's not a con to filter out those who aren't familiar with that language enough to debug an issue in it The language is based on/chosen by the candidate. We want to map the candidate to whatever language in our repository of "this question in different langs" is closest, to control for the candidate's familiarity as much as possible. If you're screening to only candidates who can code in the language your company primarily works with, you're missing good candidates. The best are going to be able to pick up a new language rapidly, particularly if your language is a mainstream one that's probably imperative or OO-ish, or both; adding a non-esoteric language to one's repertoire is just not that hard a thing to do. But in the interview, I don't want them struggling with syntactic bull, as it's not a useful signal; I want to know how they think, whether they've seen code before, and can they reason from problem statement to debugged bug.
- ajbt200128 2y ago
  > If you're screening to only candidates who can code in the language your company primarily works with, you're missing good candidates. I agree with that. I work somewhere that uses a lot of OCaml, we don't screen out those who don't know OCaml, but it usually helps them, since static analysis is easier with an ML family language than say, javascript. > We want to map the candidate to whatever language in our repository of "this question in different langs" is closest If there's not a close language in any of your repositories for the candidate, then I think that's a good signal that candidate may not be a great fit. As mentioned before, we don't necessarily ask candidates to write OCaml, most functional languages will have a similar bug + fix. > problem statement to debugged bug what if the problem you might normally run into is perf issues related to the language? I.e. HFTs usually use C++ and perf = $$$. I would agree that yes, those who have worked on say Rust could be good candidates, but if someone has years of experience in writing highly performant C++, and that's what the job requires, probably easier to look for those candidates. I definitely agree with you in general, just pointing out that there are times where being familiar with the language used is desirable.

Noumenon72 2y ago

I was thinking, wouldn't it be fairer to do the test in an online code sandbox where being able to build and install isn't an issue? Then I thought, isn't letting the candidate use their own machine and debugging tools about as biased as hiring a limo driver based on how fast they can drive their own car? It does tell you who's a real "car guy" or "Unix guy", it just has a slight disparate impact by screening for people with money and hobbies. I mean, I think you should be allowed to hire these people because they will be better at the job, but hopefully they could also show that in a sandbox test.

singleshot_ 2y ago
Decent product idea? I’ll stay tuned for your Show HN.
dingnuts 2y ago
your job doesn't let you configure your own environment and tools?
ElectricSpoon 2y ago
I find editors to be deeply personal things. Throw a vim user in vscode and you might be disappointed, and vice-versa. I don't care about your tools, I care that the sum of you and your tools make you good. I think the good middle ground is having both a canned environment and allowing candidates to use their own... Unless the job is about debugging production systems where only vi is available, in which case that interview might as well represent the actual job.
rtpg 2y ago
We would run stuff in repl.it, you get the shared codespace and also don't waste time on the ops issues (which are maybe interesting and you really want devs to be able to handle but also entirely incidental complexity for most devs). In abstract I think it's great to have people use tools their comfortable with, but I dislike dinging people because they write good code but are not good at unborking Python venvs (less of a problem now) (yes you can be a good programmer and still lose time on dumb environment stuff)
2y ago

Mistletoe 2y ago

Thought it was going to be about squashing a bug or releasing it outside your house, which would tell me a lot more about a potential employee than this.

franciscop 2y ago

I had a very similar intro experience to dev. I started with some friends as a hobby, and my first paid jobs were when I was hired to do a bunch of small tasks in different projects. Add some animations in this Ember project, fix some bugs in this Wordpress project, add a very specific feature in this Angular project, etc. I probably did a terrible job at that time and I'll always be grateful, but it also taught me A LOT about jumping into a foreign codebase and start working on it almost right away, especially since I was paid hourly and I was super-aware I was paid to fix the problems and not to learn the code. IMHO what this article explains is one of the most valuable bits I retain from that time.

kopirgan 2y ago

I could be wrong but given how fast things change in this business, this seems to test very micro level skills not the learning and adaptability skills. What if a guy could, given the business or technical issue the code solves, write something faster than he could fix others code? May be the job is about heads down coding so this bug fix test is appropriate..

from-nibly 2y ago
Then you could do that and showcase that you can write code incredibly fast.
__float 2y ago
How is debugging a "micro level skill"? Jumping into an unknown project/system/whatever is something you will likely have to do at some point. (Sometimes it might be code you wrote yourself years ago, but revisiting without any context makes it feel unknown.) Is this hypothetical guy always writing 100% correct code?

vandyswa 2y ago

I've done the equivalent of this by asking them to describe an interesting bug they've encountered in past lives; how it came up, how they hunted it, how they fixed it. By listening to them describe the work, asking questions, and following their thought processes, you can come to a fairly good hire/no from this single walk-through. I know, it's short and humane, so not a good fit for current Sillycon Valley culture.

sk11001 2y ago
There’s no chance that I can come up with a good bug story on demand without any preparation.
- tdeck 2y ago
  For past experience I reviews like this it's a good idea to give the candidate advance notice.
- serpix 2y ago
  compare this with the technical interview bomb of demanding you write a full blown program right fuking now, complete with tests. Which one do you prefer?
  - 2rsf 2y ago
    I prefer to describe a bug in details, but it is hard to come up with the details on the fly without prior preparation
  - burnished 2y ago
    Writing the program. Recalling a bug for the purpose of impressing a stranger does not sound fun.
- izacus 2y ago
  Ok, but why do you think you should be hired in that place though? :)

deleted 2y ago

[deleted]

aray07 2y ago

I did a bug squash interview when I was interviewing for new grad positions and it was one of my favorite interviews (both as an interviewee and an interviewer) - People were allowed to use their favorite IDEs - so you could see how proficient some people were - Great engineers are really really good at debugging - it was great to see how people debugged and it helped me pick up a few things as well - People couldn't leetcode their way out of this

boredtofears 2y ago

I’ve always liked the Gilded Rose refactoring[1] which is similar but about improvements/feature requests instead. [1] https://github.com/emilybache/GildedRose-Refactoring-Kata/blob/main/GildedRoseRequirements.md https://github.com/emilybache/GildedRose-Refactoring-Kata/bl...

jpmoral 2y ago

Had an interview question like this where the interviewers were very impressed. Said it was the fastest and cleanest they'd seen anyone do it, and in an unfamiliar framework no less. Unfortunately it wasn't enough for me to get the job. I had a couple of answers in the next section they didn't quite like.

__float 2y ago
> I had a couple of answers in the next section they didn't quite like. Could you say more about this? Were they technical or behavioral interviews?
- jpmoral 2y ago
  Technical. IIRC: - I did the system design at the wrong level of abstraction (too low) - Got asked about lowish-level database details. I took the question at face value and said I didn't know. What I should have done was explain what I did know (which was at a level slightly higher than the question), make an educated guess, and say how I would find the information. I still don't understand why Id didn't do that as I've always done it in interviews before or since. This is my own assessment, they didn't give detailed feedback. Just that I wasn't strong enough in some areas (paraphrasing).

michaelteter 2y ago

This is indeed a good interview challenge, and as a candidate I’ve enjoyed taking it the couple of times I have encountered it. I also like the related challenge of, “add a feature that does X” to a codebase.

lr4444lr 2y ago

How much time do you allot for this thing? Cloning the repo, dependency installs, server/DB setups, possible build steps, configuring the IDE to the project ... even if the whole thing is containerized, just going from nothing to running code in an IDE alone could be expected to take 20-30 minutes of the interview.

ibash 2y ago
You just need to rewrite the dev tools in rust! Joking aside, modern fast tools like bun can make this type of interview way more viable. I once conducted this type of interview where the candidate had to install node on their windows laptop and... we spent most of the time debugging their install :/
- photon_lines 2y ago
  FYI just small tip on the Windows / Node front, you should just install & use NVM instead of Node directly - will make many headaches go away.
s17n 2y ago
In my experience doing these interviews in Node and Python, it takes about 5 minutes to get running. If it takes more than that... maybe use a different codebase.
- lr4444lr 2y ago
  What exactly is the interviewer looking for? Familiarity with tools, or actually debugging skill? I like the idea of debugging questions, I'm just not convinced all this setup and "use your own tooling" is necessary,and that a long enough snippet of given whiteboard code sample can morr effciently test for debugging skill. I can always teach (and learn!) new tools, and frankly, there is value to me as an interviewer to see someone manage without his tools.
__float 2y ago
The bugs are chosen in projects that don't require this much setup. (There is no server/database, they use standard tools for the given language, and they are generally solvable without needing an elaborate IDE/debugger setup. Of course that may help, and it's why you bring your own computer and choose the language ahead of time.)

tdeck 2y ago

I somewhat disagree that this is reflective of a person's everyday work. Usually we're only ever working on at most handful of codebases at a time, and getting oriented in a completely new application (including getting it to build and run on your machine) doesn't happen every day. I've done interviews like this and one major pitfall is the start-up time. It's possible to spend an unreasonable amount of time debugging cold-start issues with getting the repo set up, dependencies installed, and the app to build while the interview slips away. These things are representative of the first week on a new team or project, maybe. Figuring out why bundler can't seem to download the dependency in the gemfile isn't my idea of a good use of interview time.

rtpg 2y ago

At one place there was a bug squash interview like this. Rough idea was we wrote a very small version of a system we had in our app (some data-syncing-then-displaying service), with a handful of bugs and a very simple feature request. It was very helpful for sanity checking if a person was able to be in front of a computer. There's a bit of a challenge because I think there's a pretty low ceiling of performance (strace on a Python program is fun, but our bugs are not that deep!), but we could use it to also talk about new features on this mini-system and whatnot. General feeling on it was "this is a great filter", because ultimately we needed people to just pass above a minimum skill bar, not be maximally good at coding. And having the exercise really helped to communicate the day-to-day (which can be tedious!)

bluGill 2y ago
I've interviewed a few people who have a great resume and came off well in the interview, but I was left wondering if they could write code. (HR only allows me to ask specific research based questions in an interview and so it is easy to answer well without giving any indication you can write code). We now have some filter exercises that are about if you can write code at all.
Izkata 2y ago
> (strace on a Python program is fun, but our bugs are not that deep!) They don't need to be deep to be useful. I once used it on nodejs on a hunch, looking for anything that felt "off", and discovered it was accessing settings files I had no idea even existed. Turned out the other dev was having problems because of one in his home directory the rest of us didn't have.
tharibo 2y ago
This. Keep in mind people that a job interview is to check wether you want to work with the person, not to judge if they "pass" the test. Or to say it differently, they don't have to find and fix the bug, they have to demonstrate ability with the tools, a good attitude, a match with your work culture.

flurie 2y ago

I'm not too proud to admit I took the general concept from https://sadservers.com/ https://sadservers.com/ and turned it into something I could use for interactive debugging interviews. I had golden images with some scenarios I wrote myself, and the images automatically shared a tmux session over http so I could follow along without requiring the candidates to screen share. I did have to ask for IPs so I could ensure the machines were just available to me and the candidate, but it was otherwise pretty seamless! Though now that https://sadservers.com/ https://sadservers.com/ has a paid service it might be worth looking into.

fduran 2y ago
Hello, SadServers author here. Zero shame in adapting an idea you found somewhere; I'm super happy you built something useful. I've thought of adding programming debug scenarios (I even got sadbugs.com lol), may implement in the future. I'd love to see what you've done, please feel free to connect :-)

dangsux 2y ago

[dead]

deleted 2y ago

[deleted]

zug_zug 2y ago

I've only had one "find the bug" interview, and it was awful: - Didn't set you up with a way to reliably run/test the code, nor a step-through-debugger which, jeez is it 1980? Like setting up a repo in a language you aren't familiar with (say getting your py-env exactly right) can take a whole hour if it's not your main language. - Didn't have standardized questions, which is hugely problematic (some bugs are 2 orders of magnitude harder than others) - It also just seems like there's a huge random element, like am I debugging code written by somebody who thinks at the same level of abstraction as me? Did they write comments I understand?

trevor-e 2y ago
Yea there's a huge element of randomness to these kinds of interviews. I wasn't a big fan of them at Stripe. You can get some signal like: - do they methodically approach the problem? - do they understand the bug? - do they know how to use debugging tools? - do they know advanced debugging tools? - do they work well in an unfamiliar codebase? But in practice I'd say most candidates check those boxes, at which point it becomes an arbitrary evaluation unless you pass/fail them based on whether they solved the bug.
- zug_zug 2y ago
  As it happens the place I had a really bad experience with it was stripe. I had pretty much aced every other question, but basically was setup with a problem where I didn't have a way to reliably get it running (and I wasn't even clear what the app was supposed to do, it was just some random project off of github). Fortunately I already had an offer onhand from facebook.
- leoqa 2y ago
  I gave this interview 100+ times and my criteria boiled down to: did they propose a hypothesis and then follow it to a conclusion? Doesn’t matter if it was wrong, just mattered that they were able to falsify it or find the solution. At the end, I would have them walk me through the bug, its cause, and the fix in very high level terms to make sure they could articulate the path we took.
2y ago

0x20cowboy 2y ago

If you can’t sit down with someone, have a simple tech conversation with them and be able to tell if they know what they are doing… if you need to “give someone a test”… the problem is likely not the candidate.

tedunangst 2y ago
If all I needed to do was express my thoughts out loud, I wouldn't need to hire anyone. I can do that myself.
cheeze 2y ago
On the flipside, I've spent an hour with a candidate who seemed great, but could hardly _actually program_. It was a bizarre dichotomy. Dude was clearly smart as hell but was so caught up in building perfect abstractions in parts of our software that _didn't matter all that much_ using languages that nobody else knew or wanted to use. I get it, your fun language is fun. But if the dev shop is 60% one language and 39% another, I don't really care about the small improvements. You need a strong reason that it actually _helps the business_
- caseyy 2y ago
  Ask them to explain what 4-5 different bits of code do. When they do, ask them to explain it like they would to a junior and to a senior/principal/fellow. It works very well and takes only an extra 10 minutes on top of the conversation.
marcinzm 2y ago
That biases heavily to candidates with good people skills and the skill to take over a conversation. Yes, you are not in fact immune to this even if you think you are, that’s why it’s so dangerous.
- 0x20cowboy 2y ago
  I disagree. That would indicate conversations skills equate to knowing what you are talking about. "Yes, you are not in fact immune to this even if you think you are, that’s why it’s so dangerous." I think that I am. I'll think about that a bit more. I don't care for popular people so, in fact, my bias might be the other direction. However, I don't think I have ever met a person who I thought was technical and proficient, but turned out not to be. I have been forced to hire people I knew were not proficient because they passed a test, but not the other way around.

dave333 2y ago

A similar exercise but much easier to do in an interview is to give a short piece of code that does something a bit non-trivial and ask them to write (paper or whiteboard) test cases for it. Can then either run the test cases and see if they can find (all) the bug(s), or simply review vs a known list. Classic example in Myers The Art Of Software Testing is a method that takes 3 numbers as the length of the sides of a triangle and prints out whether it is scalene, isosceles, or equilateral. Many people miss a lot of corner cases.

doawoo 2y ago

Triplebyte (one of the best hiring experiences I ever had) gave me this during their initial interview. They dropped an archive of a medium-ish codebase to my machine, that had failed unit tests. My task was to simply fix the code so the tests passed. Not only did I feel engaged with the interview because I could speak aloud as I debugged, I also found it fun!

Etheryte 2y ago
Too bad that Triplebyte then turned around and decided to sell everyone's data after it turned out their business was not scalable after all.
awkii 2y ago
You're describing the parallelized web-scraper that they pointed to their own internal site? Yeah that was fun. Too bad everyone got the same question.

deleted 2y ago

[deleted]

halfcat 2y ago

”what we’re observing is X, but we see Y instead.” What? Anyway, I’ve found these kind of interview questions rarely helpful because the scenario is either overly simplistic, or it’s some obscure bug they identified (which they only caught because it caused an outage, i.e. they also didn’t “solve” this when they wrote it). A similarly poor interview question happens in IT operations when the interviewer asks super specific details about some CVE from 5 years ago that he’s particularly proud of pulling an all nighter to patch a bunch of servers.

__float 2y ago
> what we’re observing is X, but we see Y instead. I think this might be a typo - we're observing X but want Y, or some similar variation. In any case, there's a bug.
jez 2y ago
Sorry about that! edited to > What we're observing is X, but we want to see Y instead.

datavirtue 2y ago

It's easy to write this question. I can cut a branch with gnarly UI bugs in it, no sweat.

ryoshu 2y ago

One of my favorite interviews was like this, but it was a long time ago and they printed out the code and told me there was a problem. Here's a pencil. What's wrong? Most were a function call or two. In one case I remember someone was doing a loop that incremented the index of a SQL query in code and using each result, instead of querying the set and looping through it in the code. The fun part was it was a discussion. Two people, paper and pencil, talking about the code. And the examples were from bugs they'd already squashed in a code base they inherited. It was a project that I ended up working on that had... quite a few interesting bugs like that.

mistercow 2y ago

> It’s easy for the candidate to self-assess their own progress.⊕ When the candidate isn’t doing well on it, they probably already knew it without needing to be told as much by the recruiter. This is a much better candidate experience than the whiplash of thinking you solved a question perfectly only to realize that the interviewer was looking for something else entirely. Not to say that I think this is a bad type of question overall, but IMO, this is an anti-feature. The candidate does not need to accurately self-assess their performance on the day. They need to have their confidence preserved so they don’t tilt and tank the signal for the entire interview round.

bawolff 2y ago
I don't know, i think the best questions are the ones where candidates fully understand what is being asked about them. If they aren't able to self-assess did they really understand the question?
- kenschu 2y ago
  Candidates walking away pissed is itself also a problem. A significant percentage of candidates avoid buying from companies they're rejected from [1]. They also love sharing their poor interview experience with future potential applicants. [1] https://www.wayup.com/employers/blog/how-a-positive-candidate-experience-affects-company-revenue https://www.wayup.com/employers/blog/how-a-positive-candidat...
  - neilv 2y ago
    > A significant percentage of candidates avoid buying from companies they're rejected from [1]. When rejected, or from an otherwise negative experience. It's not necessarily negative emotional associations. Usually it's that I picked up on signal that the company has serious problems -- either overall, or with a key person -- which suggests I shouldn't depend on the company as a vendor.
- mistercow 2y ago
  What you want is to prevent floundering death spirals. They need to understand the criteria, but if you need to hint and nudge them, they need to not feel like that means they’re bombing. Of course, you can absolutely do that in a bug squash interview, but my point is that it defeats this claimed advantage.

bawolff 2y ago

In computer security, this sort of interview often takes the form of find the vulnerability in this app. I like it. It feels like much more directly measuring skills than most interview questions.

zgs 2y ago

Kind of like the "install this software" as part of the interview process. "My hourly rating for figuring out your bug is $X".

maxwelljoslyn 2y ago

My kingdom for this approach to take the place of (some/all) leetcodes.

Brystephor 2y ago

I had a bug squash interview today. I found it nice, but also frustrating. It was nice because I didn't need to practice and I knew exactly how to debug the thing. It was frustrating because my personal laptop is from 7 years ago (from college), is slow, and the dependencies and editor don't work out of the box for an a new repo. Additionally, I'd prefer to use IntelliJ like I do at work but again, that's too heavy for my computer to handle so I resort to vscode and have to figure out how to use it. So then the interview becomes debugging my environment instead of debugging the problem. Maybe that's a useful signal, but it's not really bug squashing anymore then. So overall, it was still requiring learning but there was not a very good way to test in advance (how do you test all possible repo structures?)

jdlyga 2y ago

This sounds like a lot of fun to me. A lot of people dislike debugging, but I always enjoyed it. It's a bit of a murder mystery. You not only need to understand what's happening several layers beyond the initial problem, but why the code may have been written the way it is even when you find the issue.

Twirrim 2y ago

My favourite thing to do for "coding" interviews is to give the candidate a piece of absolutely awful python code that a friend of mine came up with for use for interviews. There are code smells, side effects, errors, confusing syntax, a whole slew of things in it. I give them the code, tell them I'll help with every bit of python syntax (and really emphasise that I don't mark people down _at all_ for any lack of familiarity with python / python syntax), and ask them to work through the code and do a code review. There's some python specific quirks that I largely consider bonus points at best, but anyone with any appreciable familiarity with at least one language should be able to spot a number of things. As they call stuff out, I'll dig in (or not) as seems appropriate. So far it seems to be working out great for me as an interviewer. Candidates seem to be way less stressed than they are with a "whiteboard coding" exercise. Good discussions are showing me their development skills, etc.

bigbong 2y ago
Would you mind sharing it?
esperent 2y ago
It does seem a bit unfair - ok sure, anyone familiar with C-family syntax should be able to work through it and spot errors. But anyone already familiar with Python would be able to do so a lot faster. I think the idea is good, but to be equitable it should be given in a language the person claims to be already familiar with. Or alternatively, only give it to people in a language they are not familiar with.
- phil-martin 2y ago
  If the goal was to objectively evaluate ability in a broad population, I 100% agree. However, if my dev shop primarily uses Python then selecting for people familiar with Python is pretty reasonable. Thankfully there is a giant corpus of terrible code out there for any language. Especially that code written by the most evil terrible coders of all time: past-self :D
naught0 2y ago
I'm dying to see this code

AsthmaBoy 2y ago

I like this approach. We did something similar, but for a position as support engineer, where coding ability was one of the requirements, though not the only one. They were given a piece of code on a toy, easy to read, english-like pseudo-language, and were asked to examine it and explain what the output to a specific input would be. They were told that we would answer any question, except what the answer was or what the code was doing, and encouraged them to explain their reasoning as they went ahead. We wanted to asses the candidates ability to problem solve, communicate, how would they react under preassure, and how they approached the issue. We were not necessarily looking for perfect answers, we rather rated the candidates willingness to ask for help when stuck, how well they communicated while doing the task, whether they tried different approaches, their ability to grasp what an unknown piece of code was doing and so on... all important properties for a role supporting mission critical solutions with near to zero down-time requirements. Our reasoning was that technical knowledge is easy to acquire over time, but problem solving skills, how candidates tackle difficult and unknown challanges were more important for the role. In my current role I try to design interview questions like that, looking both for the candidates current skills as well as their future potential. Not easy, but IMHO more rewarding for us and them in the long run.

RomanPushkin 2y ago

The only downside I see is that I need to share my screen with a person I don't know. Don't get me wrong, but I normally have access to some crypto on my laptop, banking notifications turned on for my 2FAs, private messages I'm getting from my spouse, maybe some borderline NSFW chatgpt dialogs, screenshots of my ideas, emails, private notes, personal projects, whole bunch of tabs I am not willing to close or share, etc. I don't have issues to show the screen to a friend of mine. Or showing the screen of a work computer to a colleague. However, demoing the whole screen is violation of privacy. Please don't do that, unless you grant VNC access to a machine with a set of popular tools, code editors, etc. How about reverse bug squash? Interviewer shares the screen with all his/her private info, notes, tabs, development environment settings, shell command history (with maybe autosuggestions turned on), and the interviewee needs to guide the interviewer towards finding the bug?

yonrg 2y ago
True, this would discomfort me, too. Also in webconferences, I never share the full screen, only one window. This should be enough for bug squash as well.
tdeck 2y ago
This is common in many interviews, not just "bug squash" interviews. I recommend having a plan for it if you're interviewing. Perhaps dedicate a virtual desktop to interviews if your system supports it.
- RomanPushkin 2y ago
  Thanks for recommendation. I just finished my rounds of interviews, gotten a decent offer (SF company), being 20 years in industry. I have my own rules, and luckily they align very well with industry standards. So I never had any issues not following the scenario when I need to share something to land a job. I also don't do any take-home assessments, and moreover, I only use one programming language while interviewing. Never had any issues with that. I think I don't need a plan or virtual desktop. I just say "no" if it doesn't work for me. Unless I am really desperate, I do not share.

aaronbrethorst 2y ago

This is essentially what I do with candidates. I ask them to pair with me on adding features to a codebase, and also to offer a code review to a PR. 100% success rate on candidates thus far.

gecko6 2y ago

I have an interview question: step 1: the candidate is shown the specification for a method and the results of running the test suite on an obfuscated version of the method. All tests pass. The test suite is minimal, and test coverage is abysmal. step 2: the candidate is asked to come up with more test cases, based on the specification. The code is run against the updated test suite - most new tests will fail because the method's implementation has several known bugs. step 3: The un-obfuscated source is provided and the candidate is asked to correct any bugs they discover. step 4: the changed source is run against a full test suite. I like this because the candidate's ability to think of test cases, debug, and fix existing code are all tested for.

pastage 2y ago
This has failed me because code get pretty gnarly fast, many things I thought was obvious was missed by people smarter than me. I think as long as it is a tool for discussions rather than grades it is good though.
saagarjha 2y ago
Curious if any interviewee has tried to unobfuscate the code.
eru 2y ago
Do you allow property based testing?

ReleaseCandidat 2y ago

> It’s fun because fixing self-contained, reproducible bugs like this is what so many of us enjoy the most about software engineering in the first place. Do really "many of us" enjoy that? In the "day-to-day business", not interviews.

jb1991 2y ago

I am quite skeptical of item 6 in this list, but otherwise it looks pretty good: > Cheating effectively is indistinguishable from debugging skill. If you know the code or problem ahead of time and how to fix it, you can certainly be convincing in your walk-through of how to do it.

pyromaker 2y ago

Would love for others to share non-technical questions that can filter bad candidates!

globular-toast 2y ago

I like this idea but I'm curious how others would implement it. Does the developer have to write the failing test themselves? Are they instructed to do it first, or is that behaviour worthy of bonus points? (Most write a passing test afterwards, which is worse). > I’ve seen candidates wield their text editor like it was an extension of their fingertips. So am I allowed to install my own text editor and other tools (Emacs with my own config that requires some build time)? Or is it on my own laptop (I don't have one)? It seems unfair if some candidates get to use their favourite tools and others don't.

eru 2y ago

Compare and contrast https://sockpuppet.org/blog/2015/03/06/the-hiring-post/ https://sockpuppet.org/blog/2015/03/06/the-hiring-post/

em1sar 2y ago

[dead]

steventhedev 2y ago

Having done nearly 100 technical interviews (with multiple questions each), the bug squash question gave us the biggest signal on candidates. Our question was far simpler: it was a simple class (java + python variants, no fancy syntax) and ask them to describe what it does, then find the bug, and finally ask them what they would change. It reflects a true test of what the day to day is, and whether or not the candidate would succeed in the role.

Pikamander2 2y ago

I've written some interview questions before and can confirm that having people find and fix simple issues in code is a surprisingly great way to weed out people who have no clue what they're doing. > Cheating effectively is indistinguishable from debugging skill. Even with knowledge of the exact bug ahead of time, if you just open the file with the bug, barf out the code to fix it, and run the tests, you’re going to fail the interview. Did you mean "distinguishable" here, by chance? The first sentence seems to contradict the second.

sethammons 2y ago

"Work Sample Test" Our best received interview question was in this style. Pick something your team fixed or did, distilled down do a 15min thing a teammate could do. Give the candidate 3x the time. For us, it was a db value not being set as expected during a cron. The candidate got our forged bug reports, cloned a repo, ran the script, verified the unexpected db state, and off to the races

ChrisMarshallNY 2y ago

I agree. Great question. Academic LeetCode exercises are really not what I consider useful in evaluating candidates. However, I'm biased. I'm really, really good at finding and fixing bugs, and pretty much stink at LeetCode, so take my support with a grain of salt. One problem is that, if the exercise gets known, expect an underground economy of solution cheats. Same with LeetCode, but that's sort of expected. Bug fix solutions hit harder.

janaagaard 2y ago

I agree. The best interview experience that I had was to be sat in front of a small app that had lots issues, both big and small, and also a lot an not-really-issues-but-still like inconsistent code styles and names, and code comments and code that weren't aligned. This was a great starting point for both me and the interviewer. The interviewer had a list of things to ask into about the code, so that we didn't get stuck.

bluGill 2y ago

There is academic research on how to interview. Most comments here give no indication the commenter is even aware it exists much less what it says. I'm aware this exists, but I'll admit to not having read it directly. (HR gives me a list of questions I'm allowed to ask in an interview and they tell me those questions are based on research but I only have their word they are)

giantg2 2y ago

I've never done a bug squash interview. I have done PR review interviews that are similar - given a PR that technically runs but has tons of problems, then see how many of the problems the candidate identities. I really liked that interview.

rurp 2y ago

I was given a similar interview problem and thought it was one of the better interviews I've had. In my case the interviewer pulled up a web app and said this particular page is loading too slow, how would you approach the problem? We went from opening dev tools to look at the requests driving the page all the way through database optimizations and every layer in between. This kind of interview didn't feel like a gotcha and was much much closer to real world work than the toy and/or algorithm problems that I have encountered. More companies should adopt these types of interview approaches.

aeternum 2y ago

>Cheating is indistinguishable from debugging skill. Even with knowledge of the exact bug ahead of time, if you just open the file with the bug, barf out the code to fix it, and run the tests, you’re going to fail the interview. Umm hopefully you don't fail for this. I've definitely known and hired devs that can do this with surprising frequency. Just because the interviewer may have taken hours to find the bug doesn't mean someone else must.

nailer 2y ago

> I’ve seen candidates drop into strace to debug a Ruby program strace is great. 99% of the time, your program is looking for a missing file. A pity it's so hard to use in modern macOS due to OS integrity protection.

mark-r 2y ago

I ended up doing this for an entire week, once. I was hired and presented with a system that didn't work. I knew nothing whatsoever about their proprietary system and wasn't even given access to their codebase. I wondered why they were being so spectacularly unhelpful with onboarding tasks. At the end of the week, I found out - the whole thing had been a test to see how well I would do, and needless to say I failed miserably. They told me I was a bad fit, and I had to agree, but probably not for the same reasons they were thinking. I really dodged a bullet with that one.

rudnevr 2y ago

it's kind of strange to me that ad-hoc debugging is considered such a valuable skill. I thought it's mostly a juniors' perspective. I typically set up extended loggers and logger methods, write everything to the some formatted file, and have a diffable, persistent, provable, multithread-friendly and versioned bug demonstration, which scales well to bugs of any complexity. (I once found some 10 bugs in a pretty old and tested bond calculation engine while migrating it to the cloud, which nobody could initially believe.)

GoToRO 2y ago

Unrelated: why there are no stories about interview questions for managers?

darepublic 2y ago

Ah the humble 2 point bug ticket