11 ms·

We compiled a library of realistic engineering take-home tests and ranked them

223 points by alexyang21 4y ago

alexyang21 4y ago
Studies [1] show that a work sample test is the best predictor of candidate performance on the job, which is why many software engineering teams use take-home tests as one step in their hiring process. But designing an effective test is difficult and time-consuming. For example, candidates are reluctant to complete tests that are too long or not engaging enough. But make them too short and teams won’t get the signal they need for a proper evaluation. To encourage more thoughtful test design (and hopefully save future candidates from the worst offenders), my team compiled the largest library of non-“whiteboard” take-home tests that real engineering teams have used. You’ll find the challenges that Stripe and Microsoft gave to their full-stack candidates, front-end tests from Tailwind and Rivian, and back-end ones from Basecamp and Revolut. Whether you’re looking to evaluate an Android, DevOps, or Data Science candidate, a bootcamp grad, or senior engineer, we found a few options for each. Having built 20+ tests ourselves, we also rated the design of each test. The criteria for a 5-star rating: 1. Tests for skills highly relevant to those required for the position 2. Includes a well-written description of the prompt and even motivation for using a take-home test 3. Sets clear expectations for candidates (e.g. time requirements, evaluation criteria, submission details) 4. Asks for a reasonable time commitment from candidates (<4 hours) A few notes: - We found most of these test prompts in public GitHub repos, usually owned by the hiring team but occasionally in the candidate-owned submission. We sifted through hundreds of tests and filtered out those overly focused on algorithms (aka LeetCode), leaving us with 142 tests in the library. - The larger and more recognizable companies didn’t always have the best tests. Some of the most interesting prompts we found were from smaller teams (e.g. YC startups). This shouldn’t be surprising. Startups need to design candidate-friendly hiring experiences to compete for talent against more established players. - There were common themes among the tests we found. For example, front-end candidates were often given a Figma design + content feed to implement, while back-end candidates had to implement an API given a set of requirements. Data scientists were usually given a data set to clean, analyze, and submit a Jupyter notebook with their findings. - We’ll continue to update this library and add descriptions of each test so it’s easier to compare. Have feedback, or another take-home test we should add? We’d love to hear from you! [1] The Validity and Utility of Selection Methods in Personnel Psychology (https://www.researchgate.net/publication/232564809_The_Validity_and_Utility_of_Selection_Methods_in_Personnel_Psychology https://www.researchgate.net/publication/232564809_The_Valid...)
- noisenotsignal 4y ago
  While this certainly sounds like reasonable criteria for rating a take home, does this truly account for general candidate reluctance to do take homes due to the time commitment? Even 2-4 hours is more time than conventional Leetcode-style initial assessments, and many may rather use that time to interview with 2-4 companies instead of just one. I recently interviewed with Ramp and enjoyed their model of practical problems, like a take home, but condensed into just one hour with less of the end-to-end expectations of a true take home.
  - ryandrake 4y ago
    If you are interviewing for 10-20 jobs in a particular week, a 2-4 hour take home test is a full time job 40 hours worth of work. I would accept a 2-4 hour time commitment but only much later in the interview process, when much of the interviewing was done and the list was whittled down to 2-3 serious companies.
    - lost953 4y ago
      Seriously, if a company wants me to spend more than an hour of my time on something for them, they better be including a check as well.
      - dymk 4y ago
        Less competition for everybody else looking at the position, then.
  - mellavora 4y ago
    > Even 2-4 hours is more time than conventional Leetcode-style initial assessments, Hard disagree on this one. Unless your day-to-day work includes many leetcode style problems, you need to put in significant time training on leetcode if you want to pass the interview. You should be able to complete a take-home based on your current skill set. Yes, it may take 4-6 hours (or 8-10), and yes, that is a big ask of a candidate, but leetcode can take 5 to 10x prep time and you still can muff the interview.
pmonza 4y ago
It's painful to imagine how many teams miss out on candidates because their take-home or technical interviews are not extracting the right signals. Having access to a library like this will save so much time on getting new ideas and iterating. At the very least, I hope it saves the effort of running into useless leetcode libraries.
- SQueeeeeL 4y ago
  Or it'll just cause the leetcode bar to be moved to fit the tests with "stronger signals". Goodhart's Law will basically apply to any exam under capitalism: "When a measure becomes a target, it ceases to be a good measure." If these jobs didn't offer more compensation and prestige than other positions, matching candidate competency to job requirements would be much more achievable, but as it is now, it's an adverseal system that demands gamification (because when any candidate plays the game, if you don't, you are actively harming your material conditions with a lower position and smaller salary)
  - stephenhogsten 4y ago
    Everything will be gamed when there are stakes. Whatever is closest to the real skills you need will makes it harder to improve at the measure without also improving at the real job skills. At least that would be the dream.
- convolvatron 4y ago
  too many companies are using the take home as an initial filter. twice now I've spent a couple days really polishing the assignment only to discover later that there is absolutely no chance we will work together well. at least make me _want_ the job before you start placing demands on my time and those of who I might provide as references.
thinkmorebetter 4y ago
Very useful, thanks!
twawaaay 4y ago
As an interviewer, I believe take home tasks are unfair and also poor return on investment. Take home tasks incentivise candidates to burn as much time as possible on the task. Consequently, they are a test of who is willing to put a week of work even when the instructions say it should not take more than 3 hours to complete. I prefer pair programming with the candidate on the premise that if I ask you to spend 2 hours of your time it is only fair that I also spend this time with you. And also give you a chance to learn about me, your future boss. Not to mention I can learn more about you working with you for half an hour than I would ever learn by studying the code you might or might not have written yourself.
- stephenhogsten 4y ago
  These are great points. How do you feel about take-homes that have an enforced cap at 60-120 minutes to try to remove the competition on time investment? Do you learn more than it's possible to learn asynchronously? Most take-homes only have candidates write code, but it's possible to understand a candidate more deeply by asking questions about, for example, how they approached the decisions they made.
  - kevmo314 4y ago
    How would such a cap be enforced? If it's due 120 minutes from the time it's given, it's no longer a take home interview, it's a remote interview.
    - smegsicle 4y ago
      and however it was enforced, it would increase the stress a great deal
      - stephenhogsten 4y ago
        True. More stress than an untimed take-home, but I would think less stress than a normal interview (for most engineers, at least).
higherintel_io 4y ago
Thank you for this! Great work!
teeceetime2 4y ago
Hardware Reverse Engineer here. Would love to see some more tests geared towards my field. (Either reverse or hardware engineering)
- SketchySeaBeast 4y ago
  I've never run into someone who that is their career - do you work for companies that are trying to take the competitions stuff apart, or legacy hardware, or is there some other stuff that's obvious that I'm missing?
  - teeceetime2 4y ago
    Think more along the lines of "bad guys" vs "good guys" rather than corporate competition. Perfectly harmless systems are built everyday. But, the consumers of those systems may not always be harmless. Understanding - and eventually controlling - the inner workings of these custom systems [covertly in some circumstances] is often necessary to stop the bad guys.
lelandfe 4y ago
As there are no tests rated 1 star, I wonder if the scoring is too lax. I've taken some of these – shocked that Algolia's made it to a 3 star. Look at how much work this is: https://github.com/algolia/solutions-hiring-assignment https://github.com/algolia/solutions-hiring-assignment
- alexyang21 4y ago
  Yeah I agree that a test that's super long is awful for candidates. We did our best to be objective in grading, so we didn't want to dock more stars based on our subjective opinion on what's considered reasonable. FWIW my team believes that test designers should strive to scope tests to no more than 1-2 hours (ideally 1). The reason why there aren't any 1-star tests isn't because they don't exist, but because we didn't think anyone would want to see them. Cataloguing these was a ton of work (we sifted through hundreds of tests). Including the 1-star ones seemed like it would only shame companies who used them.
  - lelandfe 4y ago
    I guess what I’m saying is that if that is a 3 star, I’m not sure how valuable the rating is.
- jakearmitage 4y ago
  That's an absurd amount of work to do for free. Plus, it is useless: You won't learn anything new, it is going to impact no one, and it is going to sit stale at the inbox of 2 senior developers for 2 weeks until it is seen, as a chore. This vicious cycle of wasting time must stop.
  - yamtaddle 4y ago
    Holy shit, y'all weren't kidding. That Algolia task would be many thousands of dollars of work at agencies I've worked at, and I'd expect $1-2k even bidding as a freelancer (yes, just for a mockup of this size and complexity, not prod-ready). That's so far beyond reasonable, it can no longer even see its border. Also, WTF is this about?: > Important: Do not fork this repository to create your assignment. Doing so will wake a bot that prints out your code, immediately sends it to the shredder, and archives your application in our applicant tracking system. And anyway we’d rather give everyone an equal shot to show us what they can do. They want you to host it on GH Pages later on in the instructions, so what's the point of this restriction?
_odey 4y ago
Now take that list, throw it in the rubbish bin and wash your hands thoroughly. Take home challenges are simply an absurd waste of time and a huge disrespect towards your candidates personal time. You ask them to spend 4 or more hour of their free time on a useless, throwaway excersise that you "review" in 5 minutes and you conclude they're good enough because they laid out code the way you like it... You never see the really important things that you should be screening for: how they explain their approach, their though process, how well they communicate, the compromises they had to make to fit within the timeframe, and why they'd made those compromises, how they react and adapt to changing requirements and feedback... It's just flawed on so many levels... As a new Engineering Manager I've eliminated the take home challenge in my new company in favour of this process: - You have open source code or closed code that you can show? Walk me through it. Our best engineers passed the interview this way. - You don't have code to show, 1 hour live coding challenge, focus on communication, search want you want online, ask us any questions, and explain what you're doing. We had about 40 such interviewes this year, one person refused the live challenge, 11 passed this stage, and we ended up with 7 hires accepting the offer. If you lack inspiration, use this: https://github.com/guardian/coding-exercises https://github.com/guardian/coding-exercises
- m0llusk 4y ago
  As a software engineer with significant experience I think the option to explain some existing code is valuable. This option allows me to show my skill with something that is typically more interesting and demonstrative than a quick coding test. As a professional I also value my time and find explaining how I know I can help with what needs to be done takes far less time and effort than going through a coding test and then discussing that.
- time_to_smile 4y ago
  Another huge problem with take home challenges is that they are one time use. At least if you're grinding leet code questions you're spending time that is useful at multiple companies. I have a big folder on my computer of homeworks I've finished for various companies over the years. If I recall I passed all of them, it's sort of annoying that I can't simply hand over this list of other projects I've passed and say "here, all of these companies thought this was good". What I like about your proposal is that if many other companies did it, the result would be every few years I should take a month of evenings and build out a cool project for myself demoing my ability to write code. If life gets to busy, it's fine if the project ages a bit. This way you can scratch a personal itch and prep for interviews all the same time.
llimllib 4y ago
All of ours[1] are given three stars but I would love to know why! I feel like they meet your criteria, but who knows. Would love yours, or anybody else's feedback on our problems. We do care about them, and wrote about why we use homework here: https://adhoc.team/2018/02/26/why-we-use-homework-to-recruit-engineers/ https://adhoc.team/2018/02/26/why-we-use-homework-to-recruit... [1]: https://homework.adhoc.team/ https://homework.adhoc.team/
- alexyang21 4y ago
  Hey Bill! Rating aside, I'm a big fan of Ad Hoc's homework assignments. Here's what I love about them: 1. Your team has made them public for others to use. This is HUGE. 2. The library is organized and the prompts are generally clear and explicit about the requirements to complete. 3. You offer support for candidates with questions about the assignment. 4. Kathy Keating told me about the grading environment you all use with rubrics, blind grading, and rotating cohorts of graders. We actually created similar tooling for the teams we work with. I also want to explain why they're rated as 3 stars (and why I think this undersells how good they are). When we designed the rating criteria, it was really important to recognize tests that could extract hiring signal without requiring a ton of time from candidates, since that's one of the biggest issues that candidates face. So one of our criteria was "setting clear expectations for candidates (e.g. time expectations)" and another was that the time requested was "reasonable...(<4 hours)". So a test that stated upfront that it would take 8-10 hours would meet only one of those criteria. You can see the full rubric if you hover over the "5-star scale" text in the sub-header. Unfortunately, the Ad Hoc tests don't specify time anywhere (technically missing both of these criteria) while doing many other things well that aren't captured in our rubric (e.g. candidate chat tool, blind grading). I admit that our rubric isn't perfect and it feels like the Ad Hoc tests are being doubly penalized for something that's easy to fix. In fact, if you add this to the tests, I'd happily update these to 5-stars. Finally, if you're up for a chat sometime, I'd love to meet you. I really appreciate the work that your team has done to improve the hiring experience for candidates beyond those at Ad Hoc! You can reach me at alex@trytapioca.com
endisneigh 4y ago
I actually don’t mind take home tests, but I want the company to also burn resources as to disincentivize ghosting. Problem with take homes is that you can spend hours and then they can spend seconds or minutes reviewing. At least with traditional interviews the time spent is equal for both parties.
- bogomipz 4y ago
  Agreed. That's been exactly my experience on the few I did before I refused to them. The onus is all on you and your only interface is basically a flakey recruiter. There's no guarantee that any will every look at your submission.
  - alexyang21 4y ago
    As a candidate, it's an awful feeling to spend time only to get ghosted. It's usually not malicious, just disorganization. From a hiring manager's perspective, it's too easy for things to slip through the cracks. When the volume of candidates increases, it's hard to keep track of all the zip file submissions + individual repos while making sure the eng team reviews them all. This is one of the problems my team is hoping to solve through software. So far, we're seeing teams working with us getting back to 100% of candidates (often with personalized feedback), with median times as fast as 1 day. I hope we can reduce the frequency of terrible take-home experiences.
    - bogomipz 4y ago
      I didn't say or even imply that it was malicious. Regardless, it's disrespectful and unprofessional. I fail to see how adding yet another SaaS tool into the mix of the dozens that companies already use is somehow going to fix what is essentially a cultural problem. Applicant Tracking Systems have already been a thing for a long time now.
      - alexyang21 4y ago
        From talking to many hiring managers who use take-homes and using 3 ATSs myself, most engineering take-home processes don't run through the ATS. I agree with you that it's disrespectful to candidates. My point is that it's not just a cultural problem - it's also a tooling problem.
not_the_fda 4y ago
Why would I do any of these? They are a huge time sink and I have plenty of job opportunities without jump through these hoops.
- davidjfelix 4y ago
  So there are two possibilities: - There is no reason to take them and this process will slowly fail over time as companies who adopt these policies slowly push away talent options. - You are not the target audience of a take home assignment hiring process. I think it's easy to believe the first since it affirms your world view. Speaking as somebody who has a deep pocket of job opportunities and has taken one of these take home assignments, sometimes those opportunities become stale or you're in a tough market or need a job immediately. A number of companies who have bad hiring practices are still fairly acceptable to work at, even if it's not ideal. TLDR - to pay the bills.
- alfalfasprout 4y ago
  Because they weed out people that just do a bunch of leetcode from people that actually know how to write quality software.
- fernandotakai 4y ago
  i honestly prefer a take-home exercise to live-coding. at least with a take-home exercise, i can take my time, do in a few hours, without the pressure of having someone looking at what i'm doing.
  - deleted 4y ago
    [deleted]
- ArrayBoundCheck 4y ago
  While hiring I've done a 15min test before an interview (always looking through their portfolio first) that took some candidates 30-45mins. If they fail or took longer than 45mins it ends there. I then do a 30min interview that sometimes take 45min if they talk a lot or ask me questions about the company How does this sound? I have no idea if this is ideal but so far we didn't hire any more bad candidates than we did using a different process
bogomipz 4y ago
The entry for this company Haraj states: >"Just fyi, this submission make up to 50% from your overall hiring score. While your CV only make up to 5%. So do your best to create great submission!" >"UPDATE (2022-02-05): Due to many great candidates have applied to this vacancy, we decided to close this challenge on Friday (2022-02-11). So we will wait for the last submission until Thursday (2022-02-10) at 23:59 WIB." [1] This is like a cattle call audition with a hard deadline. Who would do this? Also there is absolutely no indication of how much time one should spend this. I'm guessing this is intentional. If you want to give me a deadline there better be pay involved. I hope people are smart enough to call BS on these companies. Hopefully a company making demands one's time without compensation is a huge red flag for people. [1] https://github.com/riandyrn/owldetect https://github.com/riandyrn/owldetect
- Jensson 4y ago
  People with a bad CV but good skills would do it. There are a lot of those people entering the market every year so it makes sense for some companies to look for gems among the people other companies considers unhireable.
  - bogomipz 4y ago
    That doesn't seem to make sense. If someone had good skills why wouldn't they just embellish their resume a bit? Also there's still a preliminary phone screen interview before even asking a candidate to complete this.
rsstack 4y ago
It's my turn today to complain about website building instead of content: When macOS developers write in CSS "overflow: scroll;" they get the same behavior as they do with "overflow: auto;", but they don't realize that viewers on Linux and Windows now see ugly default operating scrollbar at all times, especially ugly when the dropdowns aren't supposed to have horizontal scrolling. Test web software on two browser and two operating systems :)
- pmonza 4y ago
  Oh dang, thanks for flagging this. It's fixed now :)
  - rsstack 4y ago
    Beautiful!!
irf1 4y ago
Give candidates real work & pay them for their time. https://blog.algora.io/post/2022-04-pay-when-hiring-part-2/ https://blog.algora.io/post/2022-04-pay-when-hiring-part-2/
- morelisp 4y ago
  He calls it a "Ride Along": top candidates spend a day of coding with his team and receive $500. How do you figure out if they're top candidates? First we did screening & technical interviews to arrive at our 5-10 most promising candidates Right, so we're back at square one doing "technical interviews" again.
  - irf1 4y ago
    That's just how Thomas at Cloudthread.io run his process, he used real paid work for his final hiring decision. It's not the only way. We personally (Algora.io founders) jumped straight into paid work without prior interviews. We shared a hiring bounty with a dozen applicants purely based on their resumes, never spoke with anyone on the phone, received 5 PRs, awarded & hired 3 of them.
    - morelisp 4y ago
      You awarded 5 of them, or 3 of them? If you're only paying the people you hire, that's not paying people for their time. I mean sorry, but if you did a good job, we're going to make you an offer such that you won't need to quibble about $500. The entire difficult part is what you're asking from people you don't hire!
      - irf1 4y ago
        Received 5 PRs, awarded the top 3. You can actually see them https://app.algora.io/algora/challenge/pulls/5 https://app.algora.io/algora/challenge/pulls/5 We clarified it in our bounty issue that only the best-winning submissions would get the bounty. Other folks have chosen to award all PR submissions regardless of hiring outcome. The format is up to the company hiring. You're making a great point! It's exactly about what we ask from people who don't get hired. I think real-world coding, getting reviewed by the CTO / hiring manager and getting compensated beats standardized coding tests & imaginary take-home assignments any time of the day.
alfalfasprout 4y ago
I think a lot of people are getting hung up on the fact that these challenges take a long time. Yep, and that's a problem... But the goal of having a candidate work on a real problem is a good once. IMO a far better approach is to do a live exercise with them. Not a leetcode problem... take an existing codebase and add a feature to it, fix an issue with it, etc. Heck, you don't need them to complete everything, but you just need to get signals on how they approach the task, how they pick up working in an unfamiliar codebase, attention to detail, communication, etc. Anything to get rid of leetcode interviews that are a collosal waste of everyone's time and lead to awful hires. Over the years there has been a democratization of study material for "coding interviews" which has led to a huge influx of wildly inexperienced/unqualified candidates. It also means if you want to hop to another company, you have to waste months grinding leetcode problems completely irrelevant to anything you'll ever do.
- frumper 4y ago
  Why not, you know, interview them? See if you like them, see if they are reasonably knowledgable about what you need them to do. Putting people on the spot doesn't mean they can or can't develop software or solutions. Sending homework is even worse though.
- alexyang21 4y ago
  I'm a big fan of live exercises too, mainly because it's a great way to see how candidates think and how they collaborate with others. There are only a couple tradeoffs with this format (every approach has them): live exercises can be more stressful and teams with many applicants won't have the bandwidth to offer this to everyone who could be qualified. My favorite is a combination: short (1 hr) take-home followed by live discussion/pairing with anyone who does a half-decent job. It reduces stress because candidates will already be familiar with the code (they wrote it!) while being efficient with time.
mr_gibbins 4y ago
Picked one at random, the Zero5 assessment, technologies: SQL. The first question was a scenario where I would build a backend endpoint using an API/Lambda and a web frontend. Nothing to do with SQL except for the 'database' icon in the architecture diagram. Rated 5 stars.
- alexyang21 4y ago
  Sorry, looks like we didn't tag this one sufficiently! We're updating this now. There was a TON of content to filter through - we're doing our best :) I also think there's a surprising number of edge cases that need to be considered for this challenge. Consider cars that were already parked at the start of the time range, ones that were entered during the range and never left, etc. So I think it does require familiarity with SQL. Did you disagree with the tagging, the stars, or both?
stakkur 4y ago
There’s no such thing as a ‘realistic engineering take-home test’. When will we admit tests are not only unrealistic, but an utterly useless predictor of…anything?
francisofascii 4y ago
I like the assignments where a starting project is provided, and you are given a few enhancements and bug fix tasks within the project. That is probably what your first few months will be like at the job anyway. You can match the coding standard of the existing project rather than agonizing about naming standards or other subjective decisions. I have an issue with the more open ended tasks where you create a project from scratch. I saw the directive "make the interface as polished as possible". That is how a 3 hour project turns into a week of work.
- alexyang21 4y ago
  I agree! It's great when companies take the time to do this. Though it's tough when it's been a few months/years and the code hasn't been maintained. You'll see some repos in the library that are up to 8 years old. One of the ideas my team brainstormed is to create and maintain a library like this that engineering teams can rely on.
magicloop 4y ago
My attitude is that take-home tests should be given to candidates if they cannot share work they've already done. This creates a win-win incentive because it would become standard in our industry to have a portfolio of GitHub contributions, a stack overflow profile, etc. At least those artefacts are helpful to a wider community as well as showing your abilities. Couple this with an open ended technical conversation with the engineer, progressively drilling down on items they know about to see the depth of their skillset, you will understand the key skills they can present for the job opportunity.
- tharkun__ 4y ago
  I have never in my life done something you can find on Github and I doubt I ever will. I have worked on plenty of stuff that I can share generic 'war stories' on, stuff you might even have 'used' without knowing but that isn't actually anything I could really "show" to you. Other than said war stories, like 'did you ever ride a train and it was late? Yeah I worked on xyz backend thing which ultimately is involved in getting that info to that info screen at the train station'. And no I don't think you can put any incentives anywhere to get that code onto a Github repo, nevermind a public one.
d23 4y ago
These are great in theory. In practice, I've regretted the handful of times I've done them. It seems like some companies use this as a way to "waste" less of their interviewers' time. One particular dysfunctional company that everyone here would recognized gave me the feedback: you're one of the candidates out of nearly 200 that did this [7-10 hour take home test] and passed! Then they ghosted me for a month before asking me to come on for a full day onsite.
- alexyang21 4y ago
  That's awful! I'm sorry you had to go through that. The intent isn't usually malicious, but some teams (esp larger ones) are really disorganized and don't value candidates' time enough. My team is doing our best to create the "perfect" take-home experience and reduce the frequency of these horror stories.
  - fuzzythinker 4y ago
    Sorry to break this to you. If you believe there's "perfect" take-home, you will be wasting your time. The perfect take-home is no take-home. Take-homes may make some sense for engineer with little experience.
- NeverFade 4y ago
  As a technical hiring manager, I was involved in some teams that did use homework assignments. Their fatal issue on the hiring side is that they take too much time to properly score. If you ask a good candidate to work for 4+ hours, they'll produce code that will cost 2+ hours of a great engineer to thoroughly check. Lots of companies are very fond of giving candidates huge tasks and having them spend 4+ hours on these. But practically no company will have their best senior engineers (a very scarce resource) dedicate dozens of hours per month to check these tasks. Hence, these tasks effectively never receive the resources required to properly check them. Especially at the more senior level. If you're hiring a senior engineer, and giving them a 4 hour homework task, a staff-level engineer would have to spend 2-3 hours to properly check it. Forget about it, that's not happening. Instead, these teams do ask the senior engineer to take 4 hours to work on the task. However, once the hard part arrives, the team will never check it properly. They'll assign some junior engineer for an hour to check it. The team won't care, because it's the candidate's time being wasted, while theirs is efficiently preserved. Plus, our junior engineers are so amazing, they can certainly score the work of a senior in less than an hour.
rehanc 4y ago
Hugely helpful! Thank you for the work you did compiling these.
0x20cowboy 4y ago
Software engineering has been turned into Star Search: https://m.youtube.com/watch?v=FfRihaJxozo https://m.youtube.com/watch?v=FfRihaJxozo When is Musk finishing that rocket to Mars?
itronitron 4y ago
Did anyone actually read the article? Just kidding, there isn't actually anything to read...
rendall 4y ago
Once, when I started out, I did a take-home test and didn't get the job. The feedback was literally "In some of the files, you mixed tabs and spaces" - which was not true. The penny dropped a few years later: there never was a job. I did some kid's homework.
thisiswrongggg 4y ago
Unless I'm desperate I'll never do an unpaid take home test again in my life. For me it has been a waste of time and hugely demoralizing. Best/worst story and the one that broke camel's back: at one uber-copycat company I had my test failing on one reviewer. Then after a few months another reviewer interviewing me for another company told me that "this test would be fine by me. Had I reviewed it when I was back at the uber-copycat company I'd pass it". And my experience was more or less like that with all unpaid take home tests. So, bitter truth is I have more to my life than doing boring unpaid work with minimal chances of getting to an offer and TBH I doubt I'd enjoy working in a company that hires like that. So, my stance is give me as much whiteboard or pair programming or whatever _time_bounded_ test you want and let's go from there. PS: I looked one of the tests. It said "follow SOLID". I'm not gonna say what I think of SOLID here but it's good that they say what they adhere to beforehand. I mean if you're going to have people pouring hours and days in your tests at least be transparent of what exactly you expect.
- throwaway5959 4y ago
  What’s wrong with SOLID?
VoodooJuJu 4y ago
Take-home tests are incredibly disrespectful. Not only is it an implication my work and experience are illegitimate, but it's a request for me to do work for free. Not happening. My frame is always "I'm a professional meeting another professional to see if what I'm selling is fit for what they're seeking to buy, and at a fair price." You can imagine my shock when the interviewer's frame is something like "Do this work for free to prove to me you're worthy." Sorry to burst your bubble, but I have nothing to prove, my friend. I like to put things like this in mob terms because it distills things down to their simple essence. Mobsters are all about essence and simplicity. No Bullshit™. This humbles the nerd. Here: >You like steak? I deliver steak. I could deliver you steak. At a good price too. https://www.youtube.com/watch?v=ii5CiYxwuMo https://www.youtube.com/watch?v=ii5CiYxwuMo That's it. That's the essence of the ideal interview. Two professionals having a conversation. No Bullshit™. Like literally every other industry. I'm a respectable steak delivery man and you are a respectable potential client. That's the frame. None of this "prove to me you're worthy" nonsense. When I deliver, you pay. If I don't deliver, you don't pay. Simple as. >Before I hire you as a plumber, I just need you to install a section of piping free of charge, because I don't trust you or believe who you say you are. Sounds ridiculous, doesn't it? It is. You'll be installing your own pipe with this attitude. There's already a great deal of asymmetry in the hiring process. You're getting paid to interview me, but I'm not. Let's not make it any more asymmetric than it already is. I understand interviewing is difficult. A lot of this difficulty stems from the fact that many of us in this business are pretty nerdy, i.e. socially awkward, both interviewers and candidates alike. This makes it difficult to probe someone via conversation to see if they'd be a good fit. But difficulty with socializing is no excuse. If you're a socially awkward hiring manager, you either need a different role or need to improve your social skills. If you're a socially awkward candidate, same thing goes for you - improve your social skills. If a hiring manager tries to saddle you with unpaid work, you tell him "fuck you, pay me". Gabish? Have some self-respect people. The more candidates that set these managers straight, the more they'll be forced to start conducting normal human interviews.
speby 4y ago
I have a better idea. Don't do take-home assignments as a candidate and don't give them as part of your interview process as an employer. Problem solved.
porknubbins 4y ago
Just a nitpick but when I see engineering without software in front of it I think EE or mech. This is a useful resource I almost skipped over. I realize certain industry segments are probably so steeped in the jargon that its obvious to people on the inside.
porknubbins 4y ago
Just a nitpick but when I see engineering without software in front of it I think EE or mech. This is a useful resource I almost skipped over. I realize certain industry segments are probably so steeped in the jargon that its obvious to people on the inside but its not to everyone.
Omnipresent 4y ago
This is a phenomenal resource. I wish it also included the DevOps category -- k8s, aws, terraforms of the world.
- alexyang21 4y ago
  You'll see DevOps as one of the options in the 'Roles' filter with 13 examples in the library.
hugonordell 4y ago
Disclaimer: one of my take home assignments I designed is part of the list compiled by the author. --- Every interview process will inherently be a flawed one. Having hired several hundred engineers in my career, I believe that it is essential to try to create an interview environment where candidates feel they are given the best possible chance at showing their potential. There's plenty of bias in engineering already that puts women, people of color and minorities at a disadvantage - and this shows through how most companies approach engineering hiring. In one of my previous roles, we looked at who made it though to subsequent interview rounds and found that we favored candidates with very similar backgrounds to our own. More than we had thought, as it turns out, than we believed going into the analysis. Our solution to reducing this bias up front was to design take home assignments that tried to resemble the day to day job as much as possible, and with measurable criteria for what we wanted candidates to cover to determine their experience and skill. It took a tremendous amount of work to get this process right, with a number of iterations to figure out the right balance for how much time a candidate should spend on an assignment and how much of day to day work our engineers should allocate to evaluating candidates. Over the course of two years, we were able to achieve an almost equal split between women and men being hired into our engineering organization, less employee churn and a generally more vibrant workplace. Take home assessments aren't a silver bullet, and they don't address every piece of the hiring puzzle. I do, however, believe that they are much better positioned to help interviewees perform at their best in an environment they feel comfortable in. This is especially true for people who today are less represented in software engineering roles. At the end of the day, take home assignments are one part of a larger toolbox of tools companies must make use of to hire great people. Just like any other approach, take homes can be abused or misused, but I think they deserve more attention than most hiring managers give them. Use them, or don't. Just make sure you hold yourself accountable to your own bias and try to reduce it so you don't miss out on great talent that don't fit your predefined view of what a great candidate looks like.

223 points by alexyang21 4y ago

Studies [1] show that a work sample test is the best predictor of candidate performance on the job, which is why many software engineering teams use take-home tests as one step in their hiring process. But designing an effective test is difficult and time-consuming. For example, candidates are reluctant to complete tests that are too long or not engaging enough. But make them too short and teams won’t get the signal they need for a proper evaluation. To encourage more thoughtful test design (and hopefully save future candidates from the worst offenders), my team compiled the largest library of non-“whiteboard” take-home tests that real engineering teams have used. You’ll find the challenges that Stripe and Microsoft gave to their full-stack candidates, front-end tests from Tailwind and Rivian, and back-end ones from Basecamp and Revolut. Whether you’re looking to evaluate an Android, DevOps, or Data Science candidate, a bootcamp grad, or senior engineer, we found a few options for each. Having built 20+ tests ourselves, we also rated the design of each test. The criteria for a 5-star rating: 1. Tests for skills highly relevant to those required for the position 2. Includes a well-written description of the prompt and even motivation for using a take-home test 3. Sets clear expectations for candidates (e.g. time requirements, evaluation criteria, submission details) 4. Asks for a reasonable time commitment from candidates (<4 hours) A few notes: - We found most of these test prompts in public GitHub repos, usually owned by the hiring team but occasionally in the candidate-owned submission. We sifted through hundreds of tests and filtered out those overly focused on algorithms (aka LeetCode), leaving us with 142 tests in the library. - The larger and more recognizable companies didn’t always have the best tests. Some of the most interesting prompts we found were from smaller teams (e.g. YC startups). This shouldn’t be surprising. Startups need to design candidate-friendly hiring experiences to compete for talent against more established players. - There were common themes among the tests we found. For example, front-end candidates were often given a Figma design + content feed to implement, while back-end candidates had to implement an API given a set of requirements. Data scientists were usually given a data set to clean, analyze, and submit a Jupyter notebook with their findings. - We’ll continue to update this library and add descriptions of each test so it’s easier to compare. Have feedback, or another take-home test we should add? We’d love to hear from you! [1] The Validity and Utility of Selection Methods in Personnel Psychology (https://www.researchgate.net/publication/232564809_The_Validity_and_Utility_of_Selection_Methods_in_Personnel_Psychology https://www.researchgate.net/publication/232564809_The_Valid...)

noisenotsignal 4y ago
While this certainly sounds like reasonable criteria for rating a take home, does this truly account for general candidate reluctance to do take homes due to the time commitment? Even 2-4 hours is more time than conventional Leetcode-style initial assessments, and many may rather use that time to interview with 2-4 companies instead of just one. I recently interviewed with Ramp and enjoyed their model of practical problems, like a take home, but condensed into just one hour with less of the end-to-end expectations of a true take home.
- ryandrake 4y ago
  If you are interviewing for 10-20 jobs in a particular week, a 2-4 hour take home test is a full time job 40 hours worth of work. I would accept a 2-4 hour time commitment but only much later in the interview process, when much of the interviewing was done and the list was whittled down to 2-3 serious companies.
  - lost953 4y ago
    Seriously, if a company wants me to spend more than an hour of my time on something for them, they better be including a check as well.
    - dymk 4y ago
      Less competition for everybody else looking at the position, then.
- mellavora 4y ago
  > Even 2-4 hours is more time than conventional Leetcode-style initial assessments, Hard disagree on this one. Unless your day-to-day work includes many leetcode style problems, you need to put in significant time training on leetcode if you want to pass the interview. You should be able to complete a take-home based on your current skill set. Yes, it may take 4-6 hours (or 8-10), and yes, that is a big ask of a candidate, but leetcode can take 5 to 10x prep time and you still can muff the interview.

pmonza 4y ago

It's painful to imagine how many teams miss out on candidates because their take-home or technical interviews are not extracting the right signals. Having access to a library like this will save so much time on getting new ideas and iterating. At the very least, I hope it saves the effort of running into useless leetcode libraries.

SQueeeeeL 4y ago
Or it'll just cause the leetcode bar to be moved to fit the tests with "stronger signals". Goodhart's Law will basically apply to any exam under capitalism: "When a measure becomes a target, it ceases to be a good measure." If these jobs didn't offer more compensation and prestige than other positions, matching candidate competency to job requirements would be much more achievable, but as it is now, it's an adverseal system that demands gamification (because when any candidate plays the game, if you don't, you are actively harming your material conditions with a lower position and smaller salary)
- stephenhogsten 4y ago
  Everything will be gamed when there are stakes. Whatever is closest to the real skills you need will makes it harder to improve at the measure without also improving at the real job skills. At least that would be the dream.
convolvatron 4y ago
too many companies are using the take home as an initial filter. twice now I've spent a couple days really polishing the assignment only to discover later that there is absolutely no chance we will work together well. at least make me _want_ the job before you start placing demands on my time and those of who I might provide as references.

thinkmorebetter 4y ago

Very useful, thanks!

twawaaay 4y ago

As an interviewer, I believe take home tasks are unfair and also poor return on investment. Take home tasks incentivise candidates to burn as much time as possible on the task. Consequently, they are a test of who is willing to put a week of work even when the instructions say it should not take more than 3 hours to complete. I prefer pair programming with the candidate on the premise that if I ask you to spend 2 hours of your time it is only fair that I also spend this time with you. And also give you a chance to learn about me, your future boss. Not to mention I can learn more about you working with you for half an hour than I would ever learn by studying the code you might or might not have written yourself.

stephenhogsten 4y ago
These are great points. How do you feel about take-homes that have an enforced cap at 60-120 minutes to try to remove the competition on time investment? Do you learn more than it's possible to learn asynchronously? Most take-homes only have candidates write code, but it's possible to understand a candidate more deeply by asking questions about, for example, how they approached the decisions they made.
- kevmo314 4y ago
  How would such a cap be enforced? If it's due 120 minutes from the time it's given, it's no longer a take home interview, it's a remote interview.
  - smegsicle 4y ago
    and however it was enforced, it would increase the stress a great deal
    - stephenhogsten 4y ago
      True. More stress than an untimed take-home, but I would think less stress than a normal interview (for most engineers, at least).

higherintel_io 4y ago

Thank you for this! Great work!

teeceetime2 4y ago

Hardware Reverse Engineer here. Would love to see some more tests geared towards my field. (Either reverse or hardware engineering)

SketchySeaBeast 4y ago
I've never run into someone who that is their career - do you work for companies that are trying to take the competitions stuff apart, or legacy hardware, or is there some other stuff that's obvious that I'm missing?
- teeceetime2 4y ago
  Think more along the lines of "bad guys" vs "good guys" rather than corporate competition. Perfectly harmless systems are built everyday. But, the consumers of those systems may not always be harmless. Understanding - and eventually controlling - the inner workings of these custom systems [covertly in some circumstances] is often necessary to stop the bad guys.

lelandfe 4y ago

As there are no tests rated 1 star, I wonder if the scoring is too lax. I've taken some of these – shocked that Algolia's made it to a 3 star. Look at how much work this is: https://github.com/algolia/solutions-hiring-assignment https://github.com/algolia/solutions-hiring-assignment

alexyang21 4y ago
Yeah I agree that a test that's super long is awful for candidates. We did our best to be objective in grading, so we didn't want to dock more stars based on our subjective opinion on what's considered reasonable. FWIW my team believes that test designers should strive to scope tests to no more than 1-2 hours (ideally 1). The reason why there aren't any 1-star tests isn't because they don't exist, but because we didn't think anyone would want to see them. Cataloguing these was a ton of work (we sifted through hundreds of tests). Including the 1-star ones seemed like it would only shame companies who used them.
- lelandfe 4y ago
  I guess what I’m saying is that if that is a 3 star, I’m not sure how valuable the rating is.
jakearmitage 4y ago
That's an absurd amount of work to do for free. Plus, it is useless: You won't learn anything new, it is going to impact no one, and it is going to sit stale at the inbox of 2 senior developers for 2 weeks until it is seen, as a chore. This vicious cycle of wasting time must stop.
- yamtaddle 4y ago
  Holy shit, y'all weren't kidding. That Algolia task would be many thousands of dollars of work at agencies I've worked at, and I'd expect $1-2k even bidding as a freelancer (yes, just for a mockup of this size and complexity, not prod-ready). That's so far beyond reasonable, it can no longer even see its border. Also, WTF is this about?: > Important: Do not fork this repository to create your assignment. Doing so will wake a bot that prints out your code, immediately sends it to the shredder, and archives your application in our applicant tracking system. And anyway we’d rather give everyone an equal shot to show us what they can do. They want you to host it on GH Pages later on in the instructions, so what's the point of this restriction?

_odey 4y ago

Now take that list, throw it in the rubbish bin and wash your hands thoroughly. Take home challenges are simply an absurd waste of time and a huge disrespect towards your candidates personal time. You ask them to spend 4 or more hour of their free time on a useless, throwaway excersise that you "review" in 5 minutes and you conclude they're good enough because they laid out code the way you like it... You never see the really important things that you should be screening for: how they explain their approach, their though process, how well they communicate, the compromises they had to make to fit within the timeframe, and why they'd made those compromises, how they react and adapt to changing requirements and feedback... It's just flawed on so many levels... As a new Engineering Manager I've eliminated the take home challenge in my new company in favour of this process: - You have open source code or closed code that you can show? Walk me through it. Our best engineers passed the interview this way. - You don't have code to show, 1 hour live coding challenge, focus on communication, search want you want online, ask us any questions, and explain what you're doing. We had about 40 such interviewes this year, one person refused the live challenge, 11 passed this stage, and we ended up with 7 hires accepting the offer. If you lack inspiration, use this: https://github.com/guardian/coding-exercises https://github.com/guardian/coding-exercises

m0llusk 4y ago
As a software engineer with significant experience I think the option to explain some existing code is valuable. This option allows me to show my skill with something that is typically more interesting and demonstrative than a quick coding test. As a professional I also value my time and find explaining how I know I can help with what needs to be done takes far less time and effort than going through a coding test and then discussing that.
time_to_smile 4y ago
Another huge problem with take home challenges is that they are one time use. At least if you're grinding leet code questions you're spending time that is useful at multiple companies. I have a big folder on my computer of homeworks I've finished for various companies over the years. If I recall I passed all of them, it's sort of annoying that I can't simply hand over this list of other projects I've passed and say "here, all of these companies thought this was good". What I like about your proposal is that if many other companies did it, the result would be every few years I should take a month of evenings and build out a cool project for myself demoing my ability to write code. If life gets to busy, it's fine if the project ages a bit. This way you can scratch a personal itch and prep for interviews all the same time.

llimllib 4y ago

All of ours[1] are given three stars but I would love to know why! I feel like they meet your criteria, but who knows. Would love yours, or anybody else's feedback on our problems. We do care about them, and wrote about why we use homework here: https://adhoc.team/2018/02/26/why-we-use-homework-to-recruit-engineers/ https://adhoc.team/2018/02/26/why-we-use-homework-to-recruit... [1]: https://homework.adhoc.team/ https://homework.adhoc.team/

alexyang21 4y ago
Hey Bill! Rating aside, I'm a big fan of Ad Hoc's homework assignments. Here's what I love about them: 1. Your team has made them public for others to use. This is HUGE. 2. The library is organized and the prompts are generally clear and explicit about the requirements to complete. 3. You offer support for candidates with questions about the assignment. 4. Kathy Keating told me about the grading environment you all use with rubrics, blind grading, and rotating cohorts of graders. We actually created similar tooling for the teams we work with. I also want to explain why they're rated as 3 stars (and why I think this undersells how good they are). When we designed the rating criteria, it was really important to recognize tests that could extract hiring signal without requiring a ton of time from candidates, since that's one of the biggest issues that candidates face. So one of our criteria was "setting clear expectations for candidates (e.g. time expectations)" and another was that the time requested was "reasonable...(<4 hours)". So a test that stated upfront that it would take 8-10 hours would meet only one of those criteria. You can see the full rubric if you hover over the "5-star scale" text in the sub-header. Unfortunately, the Ad Hoc tests don't specify time anywhere (technically missing both of these criteria) while doing many other things well that aren't captured in our rubric (e.g. candidate chat tool, blind grading). I admit that our rubric isn't perfect and it feels like the Ad Hoc tests are being doubly penalized for something that's easy to fix. In fact, if you add this to the tests, I'd happily update these to 5-stars. Finally, if you're up for a chat sometime, I'd love to meet you. I really appreciate the work that your team has done to improve the hiring experience for candidates beyond those at Ad Hoc! You can reach me at alex@trytapioca.com

endisneigh 4y ago

I actually don’t mind take home tests, but I want the company to also burn resources as to disincentivize ghosting. Problem with take homes is that you can spend hours and then they can spend seconds or minutes reviewing. At least with traditional interviews the time spent is equal for both parties.

bogomipz 4y ago
Agreed. That's been exactly my experience on the few I did before I refused to them. The onus is all on you and your only interface is basically a flakey recruiter. There's no guarantee that any will every look at your submission.
- alexyang21 4y ago
  As a candidate, it's an awful feeling to spend time only to get ghosted. It's usually not malicious, just disorganization. From a hiring manager's perspective, it's too easy for things to slip through the cracks. When the volume of candidates increases, it's hard to keep track of all the zip file submissions + individual repos while making sure the eng team reviews them all. This is one of the problems my team is hoping to solve through software. So far, we're seeing teams working with us getting back to 100% of candidates (often with personalized feedback), with median times as fast as 1 day. I hope we can reduce the frequency of terrible take-home experiences.
  - bogomipz 4y ago
    I didn't say or even imply that it was malicious. Regardless, it's disrespectful and unprofessional. I fail to see how adding yet another SaaS tool into the mix of the dozens that companies already use is somehow going to fix what is essentially a cultural problem. Applicant Tracking Systems have already been a thing for a long time now.
    - alexyang21 4y ago
      From talking to many hiring managers who use take-homes and using 3 ATSs myself, most engineering take-home processes don't run through the ATS. I agree with you that it's disrespectful to candidates. My point is that it's not just a cultural problem - it's also a tooling problem.

not_the_fda 4y ago

Why would I do any of these? They are a huge time sink and I have plenty of job opportunities without jump through these hoops.

davidjfelix 4y ago
So there are two possibilities: - There is no reason to take them and this process will slowly fail over time as companies who adopt these policies slowly push away talent options. - You are not the target audience of a take home assignment hiring process. I think it's easy to believe the first since it affirms your world view. Speaking as somebody who has a deep pocket of job opportunities and has taken one of these take home assignments, sometimes those opportunities become stale or you're in a tough market or need a job immediately. A number of companies who have bad hiring practices are still fairly acceptable to work at, even if it's not ideal. TLDR - to pay the bills.
alfalfasprout 4y ago
Because they weed out people that just do a bunch of leetcode from people that actually know how to write quality software.
fernandotakai 4y ago
i honestly prefer a take-home exercise to live-coding. at least with a take-home exercise, i can take my time, do in a few hours, without the pressure of having someone looking at what i'm doing.
- deleted 4y ago
  [deleted]
ArrayBoundCheck 4y ago
While hiring I've done a 15min test before an interview (always looking through their portfolio first) that took some candidates 30-45mins. If they fail or took longer than 45mins it ends there. I then do a 30min interview that sometimes take 45min if they talk a lot or ask me questions about the company How does this sound? I have no idea if this is ideal but so far we didn't hire any more bad candidates than we did using a different process

bogomipz 4y ago

The entry for this company Haraj states: >"Just fyi, this submission make up to 50% from your overall hiring score. While your CV only make up to 5%. So do your best to create great submission!" >"UPDATE (2022-02-05): Due to many great candidates have applied to this vacancy, we decided to close this challenge on Friday (2022-02-11). So we will wait for the last submission until Thursday (2022-02-10) at 23:59 WIB." [1] This is like a cattle call audition with a hard deadline. Who would do this? Also there is absolutely no indication of how much time one should spend this. I'm guessing this is intentional. If you want to give me a deadline there better be pay involved. I hope people are smart enough to call BS on these companies. Hopefully a company making demands one's time without compensation is a huge red flag for people. [1] https://github.com/riandyrn/owldetect https://github.com/riandyrn/owldetect

Jensson 4y ago
People with a bad CV but good skills would do it. There are a lot of those people entering the market every year so it makes sense for some companies to look for gems among the people other companies considers unhireable.
- bogomipz 4y ago
  That doesn't seem to make sense. If someone had good skills why wouldn't they just embellish their resume a bit? Also there's still a preliminary phone screen interview before even asking a candidate to complete this.

rsstack 4y ago

It's my turn today to complain about website building instead of content: When macOS developers write in CSS "overflow: scroll;" they get the same behavior as they do with "overflow: auto;", but they don't realize that viewers on Linux and Windows now see ugly default operating scrollbar at all times, especially ugly when the dropdowns aren't supposed to have horizontal scrolling. Test web software on two browser and two operating systems :)

pmonza 4y ago
Oh dang, thanks for flagging this. It's fixed now :)
- rsstack 4y ago
  Beautiful!!

irf1 4y ago

Give candidates real work & pay them for their time. https://blog.algora.io/post/2022-04-pay-when-hiring-part-2/ https://blog.algora.io/post/2022-04-pay-when-hiring-part-2/

morelisp 4y ago
He calls it a "Ride Along": top candidates spend a day of coding with his team and receive $500. How do you figure out if they're top candidates? First we did screening & technical interviews to arrive at our 5-10 most promising candidates Right, so we're back at square one doing "technical interviews" again.
- irf1 4y ago
  That's just how Thomas at Cloudthread.io run his process, he used real paid work for his final hiring decision. It's not the only way. We personally (Algora.io founders) jumped straight into paid work without prior interviews. We shared a hiring bounty with a dozen applicants purely based on their resumes, never spoke with anyone on the phone, received 5 PRs, awarded & hired 3 of them.
  - morelisp 4y ago
    You awarded 5 of them, or 3 of them? If you're only paying the people you hire, that's not paying people for their time. I mean sorry, but if you did a good job, we're going to make you an offer such that you won't need to quibble about $500. The entire difficult part is what you're asking from people you don't hire!
    - irf1 4y ago
      Received 5 PRs, awarded the top 3. You can actually see them https://app.algora.io/algora/challenge/pulls/5 https://app.algora.io/algora/challenge/pulls/5 We clarified it in our bounty issue that only the best-winning submissions would get the bounty. Other folks have chosen to award all PR submissions regardless of hiring outcome. The format is up to the company hiring. You're making a great point! It's exactly about what we ask from people who don't get hired. I think real-world coding, getting reviewed by the CTO / hiring manager and getting compensated beats standardized coding tests & imaginary take-home assignments any time of the day.

alfalfasprout 4y ago

I think a lot of people are getting hung up on the fact that these challenges take a long time. Yep, and that's a problem... But the goal of having a candidate work on a real problem is a good once. IMO a far better approach is to do a live exercise with them. Not a leetcode problem... take an existing codebase and add a feature to it, fix an issue with it, etc. Heck, you don't need them to complete everything, but you just need to get signals on how they approach the task, how they pick up working in an unfamiliar codebase, attention to detail, communication, etc. Anything to get rid of leetcode interviews that are a collosal waste of everyone's time and lead to awful hires. Over the years there has been a democratization of study material for "coding interviews" which has led to a huge influx of wildly inexperienced/unqualified candidates. It also means if you want to hop to another company, you have to waste months grinding leetcode problems completely irrelevant to anything you'll ever do.

frumper 4y ago
Why not, you know, interview them? See if you like them, see if they are reasonably knowledgable about what you need them to do. Putting people on the spot doesn't mean they can or can't develop software or solutions. Sending homework is even worse though.
alexyang21 4y ago
I'm a big fan of live exercises too, mainly because it's a great way to see how candidates think and how they collaborate with others. There are only a couple tradeoffs with this format (every approach has them): live exercises can be more stressful and teams with many applicants won't have the bandwidth to offer this to everyone who could be qualified. My favorite is a combination: short (1 hr) take-home followed by live discussion/pairing with anyone who does a half-decent job. It reduces stress because candidates will already be familiar with the code (they wrote it!) while being efficient with time.

mr_gibbins 4y ago

Picked one at random, the Zero5 assessment, technologies: SQL. The first question was a scenario where I would build a backend endpoint using an API/Lambda and a web frontend. Nothing to do with SQL except for the 'database' icon in the architecture diagram. Rated 5 stars.

alexyang21 4y ago
Sorry, looks like we didn't tag this one sufficiently! We're updating this now. There was a TON of content to filter through - we're doing our best :) I also think there's a surprising number of edge cases that need to be considered for this challenge. Consider cars that were already parked at the start of the time range, ones that were entered during the range and never left, etc. So I think it does require familiarity with SQL. Did you disagree with the tagging, the stars, or both?

stakkur 4y ago

There’s no such thing as a ‘realistic engineering take-home test’. When will we admit tests are not only unrealistic, but an utterly useless predictor of…anything?

francisofascii 4y ago

I like the assignments where a starting project is provided, and you are given a few enhancements and bug fix tasks within the project. That is probably what your first few months will be like at the job anyway. You can match the coding standard of the existing project rather than agonizing about naming standards or other subjective decisions. I have an issue with the more open ended tasks where you create a project from scratch. I saw the directive "make the interface as polished as possible". That is how a 3 hour project turns into a week of work.

alexyang21 4y ago
I agree! It's great when companies take the time to do this. Though it's tough when it's been a few months/years and the code hasn't been maintained. You'll see some repos in the library that are up to 8 years old. One of the ideas my team brainstormed is to create and maintain a library like this that engineering teams can rely on.

magicloop 4y ago

My attitude is that take-home tests should be given to candidates if they cannot share work they've already done. This creates a win-win incentive because it would become standard in our industry to have a portfolio of GitHub contributions, a stack overflow profile, etc. At least those artefacts are helpful to a wider community as well as showing your abilities. Couple this with an open ended technical conversation with the engineer, progressively drilling down on items they know about to see the depth of their skillset, you will understand the key skills they can present for the job opportunity.

tharkun__ 4y ago
I have never in my life done something you can find on Github and I doubt I ever will. I have worked on plenty of stuff that I can share generic 'war stories' on, stuff you might even have 'used' without knowing but that isn't actually anything I could really "show" to you. Other than said war stories, like 'did you ever ride a train and it was late? Yeah I worked on xyz backend thing which ultimately is involved in getting that info to that info screen at the train station'. And no I don't think you can put any incentives anywhere to get that code onto a Github repo, nevermind a public one.

d23 4y ago

These are great in theory. In practice, I've regretted the handful of times I've done them. It seems like some companies use this as a way to "waste" less of their interviewers' time. One particular dysfunctional company that everyone here would recognized gave me the feedback: you're one of the candidates out of nearly 200 that did this [7-10 hour take home test] and passed! Then they ghosted me for a month before asking me to come on for a full day onsite.

alexyang21 4y ago
That's awful! I'm sorry you had to go through that. The intent isn't usually malicious, but some teams (esp larger ones) are really disorganized and don't value candidates' time enough. My team is doing our best to create the "perfect" take-home experience and reduce the frequency of these horror stories.
- fuzzythinker 4y ago
  Sorry to break this to you. If you believe there's "perfect" take-home, you will be wasting your time. The perfect take-home is no take-home. Take-homes may make some sense for engineer with little experience.
NeverFade 4y ago
As a technical hiring manager, I was involved in some teams that did use homework assignments. Their fatal issue on the hiring side is that they take too much time to properly score. If you ask a good candidate to work for 4+ hours, they'll produce code that will cost 2+ hours of a great engineer to thoroughly check. Lots of companies are very fond of giving candidates huge tasks and having them spend 4+ hours on these. But practically no company will have their best senior engineers (a very scarce resource) dedicate dozens of hours per month to check these tasks. Hence, these tasks effectively never receive the resources required to properly check them. Especially at the more senior level. If you're hiring a senior engineer, and giving them a 4 hour homework task, a staff-level engineer would have to spend 2-3 hours to properly check it. Forget about it, that's not happening. Instead, these teams do ask the senior engineer to take 4 hours to work on the task. However, once the hard part arrives, the team will never check it properly. They'll assign some junior engineer for an hour to check it. The team won't care, because it's the candidate's time being wasted, while theirs is efficiently preserved. Plus, our junior engineers are so amazing, they can certainly score the work of a senior in less than an hour.

rehanc 4y ago

Hugely helpful! Thank you for the work you did compiling these.

0x20cowboy 4y ago

Software engineering has been turned into Star Search: https://m.youtube.com/watch?v=FfRihaJxozo https://m.youtube.com/watch?v=FfRihaJxozo When is Musk finishing that rocket to Mars?

itronitron 4y ago

Did anyone actually read the article? Just kidding, there isn't actually anything to read...

rendall 4y ago

Once, when I started out, I did a take-home test and didn't get the job. The feedback was literally "In some of the files, you mixed tabs and spaces" - which was not true. The penny dropped a few years later: there never was a job. I did some kid's homework.

thisiswrongggg 4y ago

Unless I'm desperate I'll never do an unpaid take home test again in my life. For me it has been a waste of time and hugely demoralizing. Best/worst story and the one that broke camel's back: at one uber-copycat company I had my test failing on one reviewer. Then after a few months another reviewer interviewing me for another company told me that "this test would be fine by me. Had I reviewed it when I was back at the uber-copycat company I'd pass it". And my experience was more or less like that with all unpaid take home tests. So, bitter truth is I have more to my life than doing boring unpaid work with minimal chances of getting to an offer and TBH I doubt I'd enjoy working in a company that hires like that. So, my stance is give me as much whiteboard or pair programming or whatever _time_bounded_ test you want and let's go from there. PS: I looked one of the tests. It said "follow SOLID". I'm not gonna say what I think of SOLID here but it's good that they say what they adhere to beforehand. I mean if you're going to have people pouring hours and days in your tests at least be transparent of what exactly you expect.

throwaway5959 4y ago
What’s wrong with SOLID?

VoodooJuJu 4y ago

Take-home tests are incredibly disrespectful. Not only is it an implication my work and experience are illegitimate, but it's a request for me to do work for free. Not happening. My frame is always "I'm a professional meeting another professional to see if what I'm selling is fit for what they're seeking to buy, and at a fair price." You can imagine my shock when the interviewer's frame is something like "Do this work for free to prove to me you're worthy." Sorry to burst your bubble, but I have nothing to prove, my friend. I like to put things like this in mob terms because it distills things down to their simple essence. Mobsters are all about essence and simplicity. No Bullshit™. This humbles the nerd. Here: >You like steak? I deliver steak. I could deliver you steak. At a good price too. https://www.youtube.com/watch?v=ii5CiYxwuMo https://www.youtube.com/watch?v=ii5CiYxwuMo That's it. That's the essence of the ideal interview. Two professionals having a conversation. No Bullshit™. Like literally every other industry. I'm a respectable steak delivery man and you are a respectable potential client. That's the frame. None of this "prove to me you're worthy" nonsense. When I deliver, you pay. If I don't deliver, you don't pay. Simple as. >Before I hire you as a plumber, I just need you to install a section of piping free of charge, because I don't trust you or believe who you say you are. Sounds ridiculous, doesn't it? It is. You'll be installing your own pipe with this attitude. There's already a great deal of asymmetry in the hiring process. You're getting paid to interview me, but I'm not. Let's not make it any more asymmetric than it already is. I understand interviewing is difficult. A lot of this difficulty stems from the fact that many of us in this business are pretty nerdy, i.e. socially awkward, both interviewers and candidates alike. This makes it difficult to probe someone via conversation to see if they'd be a good fit. But difficulty with socializing is no excuse. If you're a socially awkward hiring manager, you either need a different role or need to improve your social skills. If you're a socially awkward candidate, same thing goes for you - improve your social skills. If a hiring manager tries to saddle you with unpaid work, you tell him "fuck you, pay me". Gabish? Have some self-respect people. The more candidates that set these managers straight, the more they'll be forced to start conducting normal human interviews.

speby 4y ago

I have a better idea. Don't do take-home assignments as a candidate and don't give them as part of your interview process as an employer. Problem solved.

porknubbins 4y ago

Just a nitpick but when I see engineering without software in front of it I think EE or mech. This is a useful resource I almost skipped over. I realize certain industry segments are probably so steeped in the jargon that its obvious to people on the inside.

porknubbins 4y ago

Omnipresent 4y ago

This is a phenomenal resource. I wish it also included the DevOps category -- k8s, aws, terraforms of the world.

alexyang21 4y ago
You'll see DevOps as one of the options in the 'Roles' filter with 13 examples in the library.

hugonordell 4y ago

Disclaimer: one of my take home assignments I designed is part of the list compiled by the author. --- Every interview process will inherently be a flawed one. Having hired several hundred engineers in my career, I believe that it is essential to try to create an interview environment where candidates feel they are given the best possible chance at showing their potential. There's plenty of bias in engineering already that puts women, people of color and minorities at a disadvantage - and this shows through how most companies approach engineering hiring. In one of my previous roles, we looked at who made it though to subsequent interview rounds and found that we favored candidates with very similar backgrounds to our own. More than we had thought, as it turns out, than we believed going into the analysis. Our solution to reducing this bias up front was to design take home assignments that tried to resemble the day to day job as much as possible, and with measurable criteria for what we wanted candidates to cover to determine their experience and skill. It took a tremendous amount of work to get this process right, with a number of iterations to figure out the right balance for how much time a candidate should spend on an assignment and how much of day to day work our engineers should allocate to evaluating candidates. Over the course of two years, we were able to achieve an almost equal split between women and men being hired into our engineering organization, less employee churn and a generally more vibrant workplace. Take home assessments aren't a silver bullet, and they don't address every piece of the hiring puzzle. I do, however, believe that they are much better positioned to help interviewees perform at their best in an environment they feel comfortable in. This is especially true for people who today are less represented in software engineering roles. At the end of the day, take home assignments are one part of a larger toolbox of tools companies must make use of to hire great people. Just like any other approach, take homes can be abused or misused, but I think they deserve more attention than most hiring managers give them. Use them, or don't. Just make sure you hold yourself accountable to your own bias and try to reduce it so you don't miss out on great talent that don't fit your predefined view of what a great candidate looks like.