15 ms·

A hiring test I'd like to run

172 points by edent 7y ago

bulutzuku 7y ago
This is an excellent idea!
commandlinefan 7y ago
> Are they able to read a manual? Uh, I can read a manual (in fact, I enjoy reading software documentation), but it's going to take me longer than an hour...
- edent 7y ago
  Let me amend that. Are you able to look up information in a manual - either using the index or hitting CTRL+F?
  - GreenJelloShot 7y ago
    It completely depends on the manual. If this is the first time I have ever seen it, and it's about something I have never used, it very well might take me a while to figure out how to look up information quickly and effectively. Does the manual having working code examples? Then is it probably a good manual. If it does not have working examples that you can copy & paste (and they actually work), then it probably is terrible.
  - perl4ever 7y ago
    There is enormous variation in the quality of different indexes (indices?) The whole concept of an index, at least since computers have been around, should be that it's better than a full text search, but that's not something you can automatically generate.
rsstack 7y ago
This is very similar to what I do for roles that are supposed to be independent and to work with customer software. I ask the interviewee to do something with a technology or software they don't know, but give them no other constraints. They can use whatever programming language, whatever OS, whatever IDE, whatever existing libraries online and even IRC/Slack if time permits - the goal is to see that they can solve a problem that is new to them without panicking. In real life they'll have a lifeline or two in the form of colleagues, but those colleagues might also not have the answers. Obviously a terrible solution for a standard product engineering position, it is only relevant for special roles.
ses1984 7y ago
Some of the tests in my current company work like this. We don't expect you to have everything memorized. We give you a problem and then say, "feel free to use any reference you want, help, man pages, google, stack overflow, whatever." Some people still stubbornly refuse to look for help. Some people act on the first google search result even though it's obviously wrong. Some people don't know how to read a man page. I find it very illuminating, but I don't really have enough data points to say how successful it is.
- reallydontask 7y ago
  > Some people still stubbornly refuse to look for help. The thing is that some people might feel that they will get assessed lower if they look for help rather than come up or even luck into a solution. Despite making it abundantly clear to candidates that they should take the test(s) as a pair programming sessions, we had candidates struggle with some basic stuff and not ask questions. They just kept trying things while getting more nervous. A lot of people would feel that it might be held against them. The thing is a lot of people under pressure will forget the basics
  - codyb 7y ago
    One thing I’ve had help melt the pressure on both sides of the table is a laugh. I’m fortunate enough to be able to sort of slice through tension and get a rise out of most people. Stressing exactly what you’re looking for, that you’re not perfect either, that there’s no one solution, and letting them know things can be collaborative has been helpful too. Let’s ping pong off each other and see where we get, I mean that’s what we’d be doing in the day to day right? To this end I’ve sat down and added new questions to the interview guides at my previous position because I was concerned that any person asking the same question over and over again would really start to be biased towards some sort of “ideal” solution. But there’s really nothing like a good laugh to put people at ease. After one interview I mentioned that I felt I did a good job of interviewing because I could usually help relax the tensions which generally gets the best out of people and someone said that they didn’t think they’d ever gotten a laugh out of someone during an interview. That’s a bit too serious for me.
throwawayhhakdl 7y ago
Our team uses hacker rank or something similar (I forget) to weed out bad profiles (most of them). We combed through their question base to remove things that felt too specific. But I’m happy with the result, because there’s nothing you can’t google, and googling is allowed. We’re a python and R shop. You can choose either test. As a non R user, I can get a decent score on the R test, because I can read docs. And that’s great. We have also dabbled in game based assessments, and while I have qualms with it, the same kinds of skill sets seem to be the primary differentiator of success. My 2c on two types of assessments people probably normally disregard
motohagiography 7y ago
This is very specifically a terrible idea. I passed an analogous interview to this once when I was much younger, and then subjected several applicants to it thinking it was a useful filter. It's not, it's abusive psychological torture that rewards sadistic amateur psychologists and interview administrators. The only way to succeed is to submit to the interviewer by asking for help. It filters for people who know to implicate others in failure while taking credit for small successes, and bullies people into submission. The interview I passed was a variation (I learned after) on the bridges of konigsberg, where instead of asking for help, I asked if it was impossible. The person administering it hired me on the spot, but even this was wrong because the place was full of people who thought the reason they were there was because they were intelligent - something they didn't have control over - and they acted As pettily as psychotically as you would expect. The problem on teams isn't individuals not asking for help, it's managers who don't have the confidence of their teams to ask. If you are reading this and I put you through that exercise 15+ years ago, I apologise.
- dlkf 7y ago
  From the article: > I'd tell them that at the start, obviously. This isn't The Secret Rules For Getting Hired. > "I know you've never used Blender*," I'd say, "And this job doesn't require it. But we want to see how quickly and accurately you can learn to use something unfamiliar while under pressure."
  - shantly 7y ago
    I don't think I've ever had to learn something very complex under the same short time constraints and high do-or-die pressure as an interview. Tiny things (think: new-to-me unixy commands), yeah, but nothing very complex. With as many unknowns as that introduces, should it ever come up in a real job situation with a very short time window available, my expert (I'm quite serious using that term here) advice would be that we should count on having it take quite a while, and immediately begin whatever mitigation is required to allow for that, rather than waiting to begin planning for and mitigating the harm that'll do and simply gambling that this complex thing won't surprise us and end up taking longer than we'd hoped (likelihood that it will: extremely high).
techstrategist 7y ago
I experienced an extremely simple (and IMO effective) version of this. For a dba-adjacent job, I sat down with a dba and was shown how to connect to a sql server instance and change some particular setting. At the end of our hour of discussing other technical matters and my resume, he asked me to demonstrate recall of this simple command.
ditonal 7y ago
I interviewed at Twilio. My resume and phone chats made it abundantly clear that my professional experience was 95% backend in languages like Java and Go. I looked at their Glassdoor where people wrote they were heavily biased to the algorithm type questions, for better or worse. In reality I was given only a single coding interview my whole onsite, and it was to build a JavaScript SPA, including routing/linking, without the use of any framework, just vanilla JS. I was allowed to Google whatever but 50 minutes is still a crazy time crunch to figure out a lot of the stuff that needed to be figured out. It felt like a massively unfair way to evaluate me. I have honestly never felt more screwed over and time wasted by an interview than at Twilio and I would highly recommend avoiding interviewing there. My overall point being, is there’s no silver bullet. It’s not necessarily about this magical process will work well and this one will not. It’s about the small details, especially things like making sure all of your interviewers actually read the candidates resume and that everyone knows what type of role they are interviewing for, and that the interview is designed to discover both the candidates strength and weakness. I’ve been on both sides of the process and I know too often shortcuts are taken and churning the pipeline forward takes priority over doing a diligent job and respecting the time of every candidate you engage with. The very fact that the foundation of tech recruiting is inexperienced recruiters, with minimal tech knowledge, who are incentivized by sales type numbers, and churned through themselves at a comical rate, speaks to how the industry views recruiting. College athletes get recruited, top law school graduates get recruited, engineers aren’t recruited but relentlessly spammed in hopes of finding more bodies to feed into a process straight from a Kafka short story, the problem starts there and no a-ha process improvement will fix that.
- numlock86 7y ago
  > was allowed to Google whatever but 50 minutes is still a crazy time crunch to figure out a lot of the stuff that needed to be figured out. The point of such interviews is to see how you handle a situation with limited time and - like in this example - mostly new tech and paradigms to you. Usually you are not expected to output anything.
  - jjjensen90 7y ago
bilekas 7y ago
I love this.. Definitely going to set something like this up. And anyone who does go for the help gets an immediate offer. I hate people too stubborn to admit they don't know.
Msurrow 7y ago
First test of any software dev: Do you know how to look for answers on stackoverflow? Second test of any software dev: Do you know how to ask a good question on SO?
- adwww 7y ago
  15 years of software engineering experience, and I don't know how to ask a question on SO without having it shut down as non-constructive, or that I should have used one of the 300 odd ultra niche sister sites. Most vendors and libraries have IRC / Slack channels which are much more welcoming.
- notkaiho 7y ago
  First test, part deux: Are you able to ascertain the validity of answers on StackOverflow?
  - Msurrow 7y ago
    Indeed. Sort the crap from the useful fast is a nice addition.
- war1025 7y ago
  I've never found anything of actual use on StackOverflow in my ~decade of doing this professionally. The only time StackOverflow provided useful answers to my questions was during my first year of programming courses in university.
  - Insanity 7y ago
    StackOverflow is fine when learning a new language. But indeed, hardly seems to help in real-world problems.
    - Msurrow 7y ago
      Thats why I put in the "know how to ask questions" part. Sure, a lot of the time you don't get an answer. And you end up figuring out stuff the hard way yourself. But then I always force myself to go back and answer my own question. It might help the next guy, but more importantly if I have to answer the question myself I need to put what I learned in writing.. and in an understandable way as well. Often the exercise of explaining it will make me understand even better.
notkaiho 7y ago
This reminds me of the story of the young intern that went around a few years ago. They sheepishly admitted to their boss at the end of their summer internship that they had just googled all the Excel things that were required, that they hadn't actually known any of it. "But your generation KNOWS to look for help, and knows where to find it! That's where you were doing it right," was the boss's response. And that's heartening.
- commandlinefan 7y ago
  Yeah, exactly: that's the job! The boss doesn't have time to read the docs, so you read the docs and get the task done.
- beckingz 7y ago
  Heard of a guy who was interviewing for a position and had an in person section where he had to use software he had never even heard of. Guy googled enough to muddle through and got hired. Management explicitly said that he was the only person to look up information about the software, and that was why they hired him.
- shantly 7y ago
  One of the biggest differences between school and business is that in business no one really gives a shit how you got something done, within reason. You just knew it? Cool. You had to do the whole thing with the manual open on the desk next to you? Is it done? Fine, don't care, that's great, not even really sure why you told me that. You enlisted someone else to help? Is it done? Good, want to be a team lead? Of course there are constraints imposed all over, but fundamentally how you work is way less important and much more open-ended than all of school.
- dillondoyle 7y ago
  This is such a valuable skill. For our work we've found this ability to google or find answers without incredibly specific step by step instructions to be lacking from many younger workers. I've wondered if others have found this for many smart people under 30 as well?
onion2k 7y ago
If the test doesn't reflect the job you'll be doing then it's not a good test of whether or not you'll be able to do the job. If you're applying for a role that expects you to know an application then looking in the help is cheating. If using the help is acceptable in the job then it's not. It's also worth noting that people don't like "proxy tests" for the skills they'll need to use on the job. If the test only tangentially assesses a skill then it's going to be influenced by other skills or flaws and won't give an interviewer a good idea of how good you are at something. This is why people complain about whiteboard exercises in developer hiring. Unless remembering and explaining an algorithm on a board is something you'll actually be doing in the role then it's not a good test - it sort of assesses how good you are at explaining an algorithm, but it really assesses memory, presentation skills, etc. Someone with brilliant whiteboard skills isn't necessarily a good developer, so hiring them on because you were impressed by how well they explained something in front of a board feels wrong, especially to people who aren't good at the tangential stuff. In the case of the suggested hiring test in the article, it doesn't test someone's job skills. It tests how well they look for help, but it also tests how well someone copes in unfamiliar applications, or with esoteric UIs (in the case of Blender), or if they've actually used the app that's being used as a proxy. None of those things are necessarily a good assessment of whether the person can do the role they're applying for, so the test fails.
- disintegore 7y ago
  If you are required to demonstrate that you can create pivot tables, and you not only know what they are and how they work but also figured out how to do it in a brand new software suite within 10 minutes under stressful conditions, you have succeeded. Committing obscure UX patterns to memory is far less important than your ability to understand concepts and find ways to apply them. I understand a manager may prefer to supply someone with Microsoft Office 2001 Alpha Deluxe Edition (& Knuckles) and see them hit the ground sprinting but the time spent looking for such a person might be greater than what it would take for a motivated person to reach that level.
- dangerface 7y ago
  > If you're applying for a role that expects you to know an application then looking in the help is cheating. If you discourage your employees from learning you are going to have a lot of dumb employees. Even the developers who made excel do not know how to do everything that excel can do.
bryanrasmussen 7y ago
Well this is exactly like a test I went through at one time. It even used the same reasoning pretty much. The job was fullstack node and react, I went in and they said we want you to look at this game written in Python which is really slow and tell us what is wrong with it (I hadn't done python in years and back then wasn't very good at it). We will sit with you. I said I was game, they said lots of people weren't - one guy evidently got offended and said he didn't do that language (I can definitely think of one friend of mine who is totally competent who would get offended at being asked to work in a language that did not have anything to do with the job he was applying for) The game was using a big dictionary in an array and you had to look up words from it and it was slow (about 7 minutes to do searches). So I thought to myself, damn I cannot remember the name of that algorithm for this (binary search), ok let's stall a bit maybe I will remember the algorithm name. So what's the first thing you would do? Uhm how about checking a profiler. So we checked and found some simple things we could optimize. And then I looked through the code and I saw there were parts I could cache etc. So after all those things were done the time to do a search was a little bit more than halved and I still couldn't remember "binary search", so I stared dumbfounded at the screen for I think about a 3-4 seconds and I started to say "uhm" and then the guy sitting with me I guess ran out of the time he wanted to spend and said "ok well there is an algorithm for this blah blah blah" showed how to do it in python. Thanks for your time. I actually didn't want the job but still felt somewhat let down that I couldn't remember.
- js2 7y ago
  If this was Python 2.1 or later, too bad you didn't check the stdlib: https://docs.python.org/2/library/bisect.html https://docs.python.org/2/library/bisect.html
- marvinjames 7y ago
  Of course there was pressure, but why not just say that you know an algorithm that can do this faster, but you can't remember the name of it, and then just provide the intuition behind it?
  - hellisothers 7y ago
itronitron 7y ago
rather than give them the application, just show them the file (location, name, and extension) and ask them to make some useful revision to it
gota 7y ago
I don't mean to be derisivs but at that point - testing for reasoning, understanding, personality - Why stick with a software tool? You'd be better off arranging to play a strategy, preferrably collaborative, boardgame. One day I'll use Pandemic as an interviewing tool. I wonder of it'd work as a pre-formal interview stage
- edent 7y ago
  I've been to group interviews where they pretty much do that. Randomly divide all the candidates into groups, give them a task, see how they work together. I can't say I enjoyed the experience - and I've no idea how good it is as a tool - but it was illuminating to see how people treated each other.
tantalor 7y ago
See also Kobayashi Maru "a test of one's character or a solution that involves redefining the problem and managing an insurmountable scenario gracefully"
lettergram 7y ago
I think the primary issue with interviews is that the interviewer doesn’t share expectations. I think it’s totally fine to evaluate how a person asks for help, as long as it’s clear they aren’t expected to solve the problem. Further, you’d need some standardized way to evaluate how well someone asks for questions. Otherwise you’ll have interviewer bias. Which is why designing interviews are very difficult.
deleted 7y ago
[deleted]
arkanciscan 7y ago
After a couple of years working as a web developer, I decided to go to College. I took a "Web Development" class. I failed the final. The assignment was to code a website by hand. I used BBEdit to pretty print my hand coded HTML and it inserted a <meta name="editor"> tag that I didn't catch. I dropped out and went back to getting paid. Been doing it for 20 years now, though I use Prettier to do my formatting these days.
- avgDev 7y ago
  OOf I had the same final. I had to create a website by hand in a basic text editor like notepad++. I passed but I honestly don't see any value in it.
scrooched_moose 7y ago
I'm a mechanical engineer by training and still occasionally do some entry level hiring in it. My favorite question in recent years has been posing an advanced project - not unreasonably difficult, but one I know they won't have covered in school. Make it clear I don't expect them to have the answer, but have them talk through their approach to this project with me. "How would you get started?" "What would you need to know?" "What resources would you use?", etc. For us it's been a great filter - both a gauge of their current knowledge and ability to tackle an unknown.
- cushychicken 7y ago
  I'm an electrical engineer, and I have a similar question that I ask my entry level candidates - "How would you measure the voltage coming out of a wall socket?" It's familiar enough to be accessible, but technically just far enough out of their reach that they don't know exactly how to do it. We talk about approaches, tools they could use, some potential problems or issues they'd need to overcome. If they get flustered about where to start, I laugh, and say "Don't worry - I don't know the right answer. I'm actually afraid of AC mains electricity." Never fails to calm them down, because, well, it's kind of funny to hear an electrical engineer admit that they're afraid of electricity! It's a question that I've gotten a lot of mileage on.
  - perl4ever 7y ago
    Is a religious certainty you need a true RMS meter a plus or minus?
  - _AzMoo 7y ago
    Couldn't you just jam a multimeter into it?
    - cushychicken 7y ago
      You could! My immediate followons to that would be: - How would you jam a multimeter into the socket safely? - What information will the multimeter not tell you about the AC voltage signal?
haolez 7y ago
Another take on what the author proposing is to simply not care about the output of the task, but focus on how that person approaches the problem (even if they don't manage to complete the task at all).
Davertron 7y ago
I hear a lot (and believe myself I suppose) that I would rather hire someone that knows how to learn than someone who currently knows thing x but can't pick up and learn things on their own. But I feel like it's VERY hard to figure this out about someone. It's pretty easy to test how well someone knows a particular technology or tool (given some variance for operating in a high-pressure scenario etc.) but it seems pretty hard to tell if someone can learn new things quickly, and learn the right things. I feel like this is what algorithm tests are trying to do, but I also feel like a lot of algorithms are based on having a key insight into a problem that is, at least in my experience, pretty hard to have when you're in a room with strangers on a timer and being judged, so if you don't know the particular algorithm to apply, good luck. And also it's pretty easy to go study algorithms and hope you remember how to implement DFS if that's what they ask you in the interview. And there is some non-zero value to just knowing about different data structures and algorithms for certain types of work for sure. I don't think many of us web developer types will be implementing DFS ourselves, but I run into issues pretty often where having knowledge about different data structures and algorithms comes in handy. So I can see the appeal to doing a test like this that has nothing to do with a particular skill, where you're trying to evaluate someone's judgement and how quickly they can absorb new ideas or figure something out on the fly etc. BUT, I just think there are too many factors that can really muddy this up, and I still don't feel like there's an objective way to measure "smart and learns quickly". Even "learns quickly" is somewhat vague; I personally wouldn't expect someone to pick up, say, functional programming in an hour just by saying "go read about it." Some things take longer to learn than others. Anyway, there's no silver bullet. I still feel like we end up making pretty subjective decisions about other people when interviewing them. I don't think we're even all that aware of them ourselves. I would say though that if you're interviewing somewhere, you're getting useful information about that place by how they interview you, and so I wouldn't feel too disheartened if you feel like the interview is unfair or doesn't work for you. That's probably a good sign that you wouldn't enjoy working there anyway.
- GreenJelloShot 7y ago
  I totally agree. The best solution I have come up with for trying to assess if someone can learn new things quickly is simply to ask a candidate to describe the last new thing that they learned. It doesn't have to be job-related at all. Just the last thing they learned. And then ask some follow up questions about how they learned, why they learned it, and how have they used the knowledge. If nothing else, it will give you a general idea of how often they learn new things.
FiddlerClamp 7y ago
I had that same automated Microsoft Office test at a temp agency in the 1990s. Funny thing, it had a bug. As long as you didn't let go of the mouse button, it would let you open and navigate all of the menus. So I simply browsed the menu UI until I found the correct choice. 100% score.
lbarrett 7y ago
This reminds me so much of Vernor Vinge's story "Fast Times at Fairmont High". It's set in the near future, and one plot point is that high school tests cover this skill--to read through the manuals quickly and get things done with new tools. I recommend it, and the related book "Rainbows End".
catern 7y ago
I don't know about this. I don't think I should spend most of my time learning completely new tools. If people spend most of their time learning tools, that encourages tools which are easy to learn but lack deep and powerful functionality. I think it's fine, and even good, for tools to require focused learning to understand, and in exchange have a worldview and tool use that is really powerful. Like Emacs or vim or Excel, for example.
tracker1 7y ago
I partially agree, and it depends on what you are interviewing for. In the end, getting a job done, on time, without error/bugs is generally most important to the people you are going to be working for. As for algorithm questions at some FAANG or wannabe startup, it can be rough. Personally I tend to blank on a lot of terminology, similarly on names a lot of the time. Some trivia games kill me, I can see the answer's face, the movies they've been in, but just blank on the name. All the same, in my life and career, I've managed to learn, adapt and create both simple and complex systems in new languages and platforms. On the other side of the table, I tend to ask maybe 3-5 trivia like questions only to gauge where a person is at, and let the conversational parts determine if the person would be a good fit. I think you need some of both. I feel drive accounts for more than even skill, knowledge and experience.
- perl4ever 7y ago
  Maybe you just can't figure a person out in an interview. I got my current job as a contractor/temp, and thinking back to a previous company, one of the people that was the best and stayed the longest was a contractor that "graduated" to permanent. The simplest way to prevent people from keeping up a facade is just time.
reallydontask 7y ago
> I'd tell them that at the start, obviously. This isn't The Secret Rules For Getting Hired. > "I know you've never used Blender," I'd say, "And this job doesn't require it. But we want to see how quickly and accurately you can learn to use something unfamiliar while under pressure." > Or whatever. The main problem is that the level of pressure that people feel under interview conditions will vary dramatically. Personally, I would find this a very bad test as I get horrendously stressed at interviews yet somehow manage to portrait a facade of calmness (to be fair I've been told once that I looked remarkably calm). To the usual refrain of: If you can't handle the pressure of an interview then maybe you are not a good fit for our company I normally reply: If day to day working at your company generates interview like levels of stress, then please do not offer me a job
- perl4ever 7y ago
  If someone says there was a lack of culture fit, it stings and certainly could be some type of illegal discrimination. But at the same time, it's necessarily true! Unless your culture is discriminating unfairly...
myself248 7y ago
I was pretty happy with the interviews I went through for my current position, although that might be selection bias because I passed them. They all shared some important attributes: 1: Fairly open-ended, there was no "bzzt-wrong" moment like the Office 97 case in TFA, so I didn't feel a lot of pressure. It was definitely enough rope to hang oneself, so the interviewers could form an actionable and objective judgment, but as a candidate, I didn't feel that I was on trial, more just chilling and talking tech with some other nerds. 2: The underlying skills (understanding one's audience, breaking a large problem into pieces, making reasonable assumptions) were obviously directly applicable to the position. I never felt that I was doing senseless work, even when the questions were clearly contrived, because the connections were self-evident. 3: Despite the above, the actual questions didn't require a lot of domain-specific knowledge. Even the one that sorta did, could be answered by assuming that it was similar to a more common system, and that's how I approached it. 4: In every case, there were "plan B" options and accommodations for having a gap in one's knowledge. "I don't know this off the top of my head, but here's where I'd go to find out and here's the guess I'm going to use in the meantime" was a valid answer. Perhaps the best answer in some cases. The result is that I work with a team of people who are super adaptable, can think on their feet, but have a keen interest in pushing towards correctness as soon as real information is available. Whenever I think about possibly going elsewhere, I remember that most HR departments don't select for any of those attributes, often quite the opposite. (And those places tend to be the clients whose problems we get called in to help solve, as a result.)
dcole2929 7y ago
The single best interview experience I've seen was given by Oracle when I worked there a few years ago. The onsite was a panel interview where candidates were given a laptop with a configured IDE and codebase with a specific set of bugs. Their task was to debug the various issues and complete as many as they could with the understanding that it was not expected for them to solve them all or even most of them. There were also tests to help them. And a different section of the interview asked them to right some tests and some code for again brain dead easy coding algorithms like reverse a string. I thought it was great because it closely resembled what being a dev is like and actually tested for the skills we cared about. No you didn't get to use your own custom vim configuration but I'm not sure how that would have really worked even if they wanted.
city41 7y ago
At my previous company, the technical interview was 2 hours long. The candidate was given a choice of about 10 very small, simple projects. Then we sat at a pairing station and paired on the problem. The candidate can use google, their favorite editor, write tests, anything they want. The goal is not to finish the project, but just code with them for 2 hours. It's the best technical assessment I've yet seen in an interview setting. One caveat is this company really believed in pair programming, and virtually all code was created in pairs. So that's part of why this interview approach worked.
- organsnyder 7y ago
  I did an interview once at a company that is religious about pair programming. I found the exercise unsettling: would they evaluate technical ability or culture fit more? If they were focusing on the former, I should take the wheel and try to demonstrate my chops (perhaps making me appear arrogant); if the latter, I should try to demonstrate how I can collaborate with the other developer to make both of us better (perhaps at the expense of me not demonstrating as many skills within that brief window of time). It turns out that they were really focusing on how I interacted with the fake "client" and whether I had the proper "consultant mindset". Didn't see that coming (perhaps that's my fault). Though I feel I ended up dodging a bullet in not getting an offer from them. My suggestion is to try to give candidates specific times to demonstrate various competencies (technical, colleague interaction, customer-facing), and tell them which ones are the focus of each exercise. Yes, that's not how the real world works (you need to utilize them all at the same time, of course), but an interview setting is hardly the real world. And you'll still see glimpses of their overall capabilities in each stage.
  - city41 7y ago
    That's a really good point. I honestly don't remember if we gave the candidate a heads up in that regard. In general interviewing is extremely imprecise. It's an unsolved problem unfortunately, but I feel most companies "fix" the problem by just over-interviewing and being ok with false negatives.
  - baumy 7y ago
learnstats2 7y ago
This made me laugh - I first applied for temp jobs around this time and there was a typing test. "Wow," said the person who had left me alone with a computer that felt old in the 1990s, "I've never seen anyone score 50WPM before." "Uhh... I could type a lot faster, but the letter T is sticky and I had to delete a lot of Ts" She looked mortally offended by this. I was given a job as a dishwasher which I was sacked from after one shift.
LanceH 7y ago
I've always wanted to run one where we bring them to a room to interview. Maybe take a break. Move their stuff to a slightly different room. Pick back up right where we left off. Maybe change out the furniture and people outside during the interview. Introduce them to numerous employees, who will respond to names swapped amongst each other. Another break - switch interviewers. Pick up right where the first one left off - insist that this employee has conducted the entire thing. If they drove there, it would be best to move their car. I'm not sure what position I would be hiring for, but this is the interview I would like to conduct. Other possible ideas would be a background check that is either insanely deep (so you're 3rd grade teacher was unavailable for a recommendation) or raises issues that they have no part of, "we see you've spent time in North Korea."
- notkaiho 7y ago
  ... But why?
  - jedieaston 7y ago
    To see how they respond under pressure.
    - LanceH 7y ago
      Today's changing markets demand employee's who can adapt quickly. Or static old companies seem to want employees who don't notice change around them.
    - notkaiho 7y ago
      I'd be very confused and ask questions as to the reasons they're fucking around.
neo2006 7y ago
I do hiring interview in my company and one of the test we like to do is to give the interviewee a problem out of his area of expertise and open a browser and an editor for him and tell him that he is allowed to search for things. Most of people I interviewed do not use Google help they are afraid you judge them negatively for it even if you say you will not.
ngneer 7y ago
Interviews should be personalized. That they are conducted en masse with little consideration for applicant background is an artifact of the difficulty of adequate personalization. Targeted ads are valued precisely because they are targeted, presumably increasing the chance of a match. Has anyone heard of any startup companies in the personalized interview space?
Aeolun 7y ago
I like how one of the comments is about someone getting a job by telling the interviewer they knew what a computer was.
scarejunba 7y ago
Haha, that's funny, and an awful idea. At a previous employer, as a hiring manager, I offered candidates an option of bringing in their fully-configured IDE in their own laptop with the language of their choice and I'd give them the problem to solve. Essentially, act in the environment that you've already mastered. Only one guy did (I had maybe 10 a month) and for some inexplicable reason he couldn't figure out how to create his `main` equivalent in his language in his IDE. He must have been incredibly nervous because we tried "perhaps you could sketch out your thoughts in broad strokes and then we can return to the IDE" etc.
Iv 7y ago
I just got interested in the way the school 42 works. Especially the month-long "test" that they have to accept tuition free students. The idea is to make you complete a lot of programming projects in languages and tools you normally don't know at all. When you start. The evaluation does not only take into account if you finished project but also if you helped others, knew how to look for help (basically: google or other students). Its goal is to select students who would work well in a peer-learning project-based-learning paradigm. This, in my opinion, is much closer to real-life conditions than any academic test.
Tade0 7y ago
There must be a serious dearth of candidates in my area, because the last time I was subjected to a time-constrained test was almost three years ago, and it was a Codility challenge, so I had some time to prepare for it. I quit my job recently, and in order to find another I went full-auto with the applications - I've sent close to thirty of them I think, so a long series of recruitment processes followed. They were all mostly the same - I either received some homework to do or had a 1-2h technical interview during which I was asked a set of questions which were easy for the most part. And that was it. My point being: I've discovered that I can't really relate to the stories other people posted in this thread. Does this mean I wasn't ambitious enough regarding my applications, or are the others subjected to a difficult job market? Maybe both? Or none? I really don't know.
stedman 7y ago
We do this at retriever.co for our virtual assistants, in two areas: 1. we have applicants do a data transform with an esoteric excel function that they've /almost surely/ never used, and prompt them to find videos / tutorials on how to use it if they don't know (which, we assume, they wont) 2. we have people answer fake customer support tickets that require them to do some on-the-job learning about the Internet Service Provider industry -- to a level of detail that requires that they read through and grok a page on wikipedia explaining up-time calculation. While neither of these pieces of knowledge are necessary for their day-to-day at Retriever, they allow us to stratify candidates' ability to learn new skills and grok new information. 100% would recommend!
- aniketpanjwani 7y ago
  I've been thinking through how best to screen personal assistants, and these are both very helpful ideas. I particularly like the first idea to screen applicants on esoteric utilities - thanks for your comment!
Iv 7y ago
I keep hearing and reading horror stories like this one on HN and other website. I never did interviews at big companies, nor did I do any in the US, in 15 years of freelancing, I only had once a technical interview, with a C++ guru, I failed most of the questions but still got hired because "you still did good, kept calm and asked good questions". In my last gig, it was for a work on a Java application. I was upfront that I only did a bit of Java in school, but that I was more of a C++ guy. The Java guru hirer was like "Yeah, if you know C++, you'll get Java quickly" they were more interested in my experience with video and image processing. I basically avoid to do HR-based interviews. Short-circuit them when you can.
deleted 7y ago
[deleted]
deleted 7y ago
[deleted]
sylvainkalache 7y ago
That's how Holberton School (we train people to become software engineer) select and train students. We are a college-alternative where students learn by working on projects, in group. The whole education is flipped compared to a regular one. We give students problem sets and they have to find the answer. We never give lectures (we have no teachers), just guidance so that they can get started on a topic, but never enough that they can actually achieve the tasks we give them. > Are they able to read a manual? Candidates (no prior experience in coding needed) need to answer questions on a specific command that can be found by reading the manpage. At first we provide the manpage, then we show them how to find a manpage, and finally, we don't provide any guidance, because at this point they should be able to do it on their own. In the curriculum, once students, we have a quizz to makre sure they can answer basic questions on the topic before jumping into the coding part. >Can they formulate a search query? We ask trivial questions that can be answered just by Googling. Might sound like a no-brainer to the developer community, but not everybody is good at this. >How do they assess whether the tutorial they found is suitable or reliable? That's the tricky part, but basically, because the school is not providing material that contains the answer, candidates and students have to navigate the ocean of information/tutorial that can be wrong/incomplete of correct. By pressing a button, students work (code at this point) is automatically and instantly corrected, so they can easily figure that what they built is correct or not. > What steps do they take to make sure they're finding - and learning - the right information? On this one, we as a school, are the one providing this guidance. Basically when students are doing their projects, we define learning goals. Getting coding part done is great, but we also want to make sure that students have been able to grasp concepts that are key to becoming a great software engineers. So students look at these "learning points" before, and by the end of the project, they must be able to discuss all the points.
andrewfong 7y ago
Honestly, I wish more companies would just tell you in advance of the interview what the specific prompt is going to be (and perhaps the evaluation criteria too). Sure, you could "cheat" by Googling the answer in advance or asking someone for help. But if the question's sufficiently open-ended, it's not that hard to suss out in follow up questions whether you have a deep first principles understanding of the subject area or you're just repeating something you read.
- bagacrap 7y ago
  I'm pretty sure if you give me a subject I know nothing about and a couple days to study I can teach myself about it, at least to the level of someone who has real experience in the subject which may be a couple of years out of use. So then how does a hiring manager know to filter me out?
  - andrewfong 7y ago
    Presumably, at some point, you'll be interviewed by someone with more up to date experience. And if you study enough to fool that person, that seems fine? It's a good sign you're capable of the job, prior experience be damned.
myegorov 7y ago
fwiw this describes my experience of interviewing at Databricks

172 points by edent 7y ago

This is an excellent idea!

> Are they able to read a manual? Uh, I can read a manual (in fact, I enjoy reading software documentation), but it's going to take me longer than an hour...

edent 7y ago
Let me amend that. Are you able to look up information in a manual - either using the index or hitting CTRL+F?
- GreenJelloShot 7y ago
  It completely depends on the manual. If this is the first time I have ever seen it, and it's about something I have never used, it very well might take me a while to figure out how to look up information quickly and effectively. Does the manual having working code examples? Then is it probably a good manual. If it does not have working examples that you can copy & paste (and they actually work), then it probably is terrible.
- perl4ever 7y ago
  There is enormous variation in the quality of different indexes (indices?) The whole concept of an index, at least since computers have been around, should be that it's better than a full text search, but that's not something you can automatically generate.

rsstack 7y ago

This is very similar to what I do for roles that are supposed to be independent and to work with customer software. I ask the interviewee to do something with a technology or software they don't know, but give them no other constraints. They can use whatever programming language, whatever OS, whatever IDE, whatever existing libraries online and even IRC/Slack if time permits - the goal is to see that they can solve a problem that is new to them without panicking. In real life they'll have a lifeline or two in the form of colleagues, but those colleagues might also not have the answers. Obviously a terrible solution for a standard product engineering position, it is only relevant for special roles.

ses1984 7y ago

Some of the tests in my current company work like this. We don't expect you to have everything memorized. We give you a problem and then say, "feel free to use any reference you want, help, man pages, google, stack overflow, whatever." Some people still stubbornly refuse to look for help. Some people act on the first google search result even though it's obviously wrong. Some people don't know how to read a man page. I find it very illuminating, but I don't really have enough data points to say how successful it is.

reallydontask 7y ago
> Some people still stubbornly refuse to look for help. The thing is that some people might feel that they will get assessed lower if they look for help rather than come up or even luck into a solution. Despite making it abundantly clear to candidates that they should take the test(s) as a pair programming sessions, we had candidates struggle with some basic stuff and not ask questions. They just kept trying things while getting more nervous. A lot of people would feel that it might be held against them. The thing is a lot of people under pressure will forget the basics
- codyb 7y ago
  One thing I’ve had help melt the pressure on both sides of the table is a laugh. I’m fortunate enough to be able to sort of slice through tension and get a rise out of most people. Stressing exactly what you’re looking for, that you’re not perfect either, that there’s no one solution, and letting them know things can be collaborative has been helpful too. Let’s ping pong off each other and see where we get, I mean that’s what we’d be doing in the day to day right? To this end I’ve sat down and added new questions to the interview guides at my previous position because I was concerned that any person asking the same question over and over again would really start to be biased towards some sort of “ideal” solution. But there’s really nothing like a good laugh to put people at ease. After one interview I mentioned that I felt I did a good job of interviewing because I could usually help relax the tensions which generally gets the best out of people and someone said that they didn’t think they’d ever gotten a laugh out of someone during an interview. That’s a bit too serious for me.

throwawayhhakdl 7y ago

Our team uses hacker rank or something similar (I forget) to weed out bad profiles (most of them). We combed through their question base to remove things that felt too specific. But I’m happy with the result, because there’s nothing you can’t google, and googling is allowed. We’re a python and R shop. You can choose either test. As a non R user, I can get a decent score on the R test, because I can read docs. And that’s great. We have also dabbled in game based assessments, and while I have qualms with it, the same kinds of skill sets seem to be the primary differentiator of success. My 2c on two types of assessments people probably normally disregard

motohagiography 7y ago

This is very specifically a terrible idea. I passed an analogous interview to this once when I was much younger, and then subjected several applicants to it thinking it was a useful filter. It's not, it's abusive psychological torture that rewards sadistic amateur psychologists and interview administrators. The only way to succeed is to submit to the interviewer by asking for help. It filters for people who know to implicate others in failure while taking credit for small successes, and bullies people into submission. The interview I passed was a variation (I learned after) on the bridges of konigsberg, where instead of asking for help, I asked if it was impossible. The person administering it hired me on the spot, but even this was wrong because the place was full of people who thought the reason they were there was because they were intelligent - something they didn't have control over - and they acted As pettily as psychotically as you would expect. The problem on teams isn't individuals not asking for help, it's managers who don't have the confidence of their teams to ask. If you are reading this and I put you through that exercise 15+ years ago, I apologise.

dlkf 7y ago
From the article: > I'd tell them that at the start, obviously. This isn't The Secret Rules For Getting Hired. > "I know you've never used Blender*," I'd say, "And this job doesn't require it. But we want to see how quickly and accurately you can learn to use something unfamiliar while under pressure."
- shantly 7y ago
  I don't think I've ever had to learn something very complex under the same short time constraints and high do-or-die pressure as an interview. Tiny things (think: new-to-me unixy commands), yeah, but nothing very complex. With as many unknowns as that introduces, should it ever come up in a real job situation with a very short time window available, my expert (I'm quite serious using that term here) advice would be that we should count on having it take quite a while, and immediately begin whatever mitigation is required to allow for that, rather than waiting to begin planning for and mitigating the harm that'll do and simply gambling that this complex thing won't surprise us and end up taking longer than we'd hoped (likelihood that it will: extremely high).

techstrategist 7y ago

I experienced an extremely simple (and IMO effective) version of this. For a dba-adjacent job, I sat down with a dba and was shown how to connect to a sql server instance and change some particular setting. At the end of our hour of discussing other technical matters and my resume, he asked me to demonstrate recall of this simple command.

ditonal 7y ago

I interviewed at Twilio. My resume and phone chats made it abundantly clear that my professional experience was 95% backend in languages like Java and Go. I looked at their Glassdoor where people wrote they were heavily biased to the algorithm type questions, for better or worse. In reality I was given only a single coding interview my whole onsite, and it was to build a JavaScript SPA, including routing/linking, without the use of any framework, just vanilla JS. I was allowed to Google whatever but 50 minutes is still a crazy time crunch to figure out a lot of the stuff that needed to be figured out. It felt like a massively unfair way to evaluate me. I have honestly never felt more screwed over and time wasted by an interview than at Twilio and I would highly recommend avoiding interviewing there. My overall point being, is there’s no silver bullet. It’s not necessarily about this magical process will work well and this one will not. It’s about the small details, especially things like making sure all of your interviewers actually read the candidates resume and that everyone knows what type of role they are interviewing for, and that the interview is designed to discover both the candidates strength and weakness. I’ve been on both sides of the process and I know too often shortcuts are taken and churning the pipeline forward takes priority over doing a diligent job and respecting the time of every candidate you engage with. The very fact that the foundation of tech recruiting is inexperienced recruiters, with minimal tech knowledge, who are incentivized by sales type numbers, and churned through themselves at a comical rate, speaks to how the industry views recruiting. College athletes get recruited, top law school graduates get recruited, engineers aren’t recruited but relentlessly spammed in hopes of finding more bodies to feed into a process straight from a Kafka short story, the problem starts there and no a-ha process improvement will fix that.

numlock86 7y ago
> was allowed to Google whatever but 50 minutes is still a crazy time crunch to figure out a lot of the stuff that needed to be figured out. The point of such interviews is to see how you handle a situation with limited time and - like in this example - mostly new tech and paradigms to you. Usually you are not expected to output anything.
- jjjensen90 7y ago

bilekas 7y ago

I love this.. Definitely going to set something like this up. And anyone who does go for the help gets an immediate offer. I hate people too stubborn to admit they don't know.

Msurrow 7y ago

First test of any software dev: Do you know how to look for answers on stackoverflow? Second test of any software dev: Do you know how to ask a good question on SO?

adwww 7y ago
15 years of software engineering experience, and I don't know how to ask a question on SO without having it shut down as non-constructive, or that I should have used one of the 300 odd ultra niche sister sites. Most vendors and libraries have IRC / Slack channels which are much more welcoming.
notkaiho 7y ago
First test, part deux: Are you able to ascertain the validity of answers on StackOverflow?
- Msurrow 7y ago
  Indeed. Sort the crap from the useful fast is a nice addition.
war1025 7y ago
I've never found anything of actual use on StackOverflow in my ~decade of doing this professionally. The only time StackOverflow provided useful answers to my questions was during my first year of programming courses in university.
- Insanity 7y ago
  StackOverflow is fine when learning a new language. But indeed, hardly seems to help in real-world problems.
  - Msurrow 7y ago
    Thats why I put in the "know how to ask questions" part. Sure, a lot of the time you don't get an answer. And you end up figuring out stuff the hard way yourself. But then I always force myself to go back and answer my own question. It might help the next guy, but more importantly if I have to answer the question myself I need to put what I learned in writing.. and in an understandable way as well. Often the exercise of explaining it will make me understand even better.

notkaiho 7y ago

This reminds me of the story of the young intern that went around a few years ago. They sheepishly admitted to their boss at the end of their summer internship that they had just googled all the Excel things that were required, that they hadn't actually known any of it. "But your generation KNOWS to look for help, and knows where to find it! That's where you were doing it right," was the boss's response. And that's heartening.

commandlinefan 7y ago
Yeah, exactly: that's the job! The boss doesn't have time to read the docs, so you read the docs and get the task done.
beckingz 7y ago
Heard of a guy who was interviewing for a position and had an in person section where he had to use software he had never even heard of. Guy googled enough to muddle through and got hired. Management explicitly said that he was the only person to look up information about the software, and that was why they hired him.
shantly 7y ago
One of the biggest differences between school and business is that in business no one really gives a shit how you got something done, within reason. You just knew it? Cool. You had to do the whole thing with the manual open on the desk next to you? Is it done? Fine, don't care, that's great, not even really sure why you told me that. You enlisted someone else to help? Is it done? Good, want to be a team lead? Of course there are constraints imposed all over, but fundamentally how you work is way less important and much more open-ended than all of school.
dillondoyle 7y ago
This is such a valuable skill. For our work we've found this ability to google or find answers without incredibly specific step by step instructions to be lacking from many younger workers. I've wondered if others have found this for many smart people under 30 as well?

onion2k 7y ago

If the test doesn't reflect the job you'll be doing then it's not a good test of whether or not you'll be able to do the job. If you're applying for a role that expects you to know an application then looking in the help is cheating. If using the help is acceptable in the job then it's not. It's also worth noting that people don't like "proxy tests" for the skills they'll need to use on the job. If the test only tangentially assesses a skill then it's going to be influenced by other skills or flaws and won't give an interviewer a good idea of how good you are at something. This is why people complain about whiteboard exercises in developer hiring. Unless remembering and explaining an algorithm on a board is something you'll actually be doing in the role then it's not a good test - it sort of assesses how good you are at explaining an algorithm, but it really assesses memory, presentation skills, etc. Someone with brilliant whiteboard skills isn't necessarily a good developer, so hiring them on because you were impressed by how well they explained something in front of a board feels wrong, especially to people who aren't good at the tangential stuff. In the case of the suggested hiring test in the article, it doesn't test someone's job skills. It tests how well they look for help, but it also tests how well someone copes in unfamiliar applications, or with esoteric UIs (in the case of Blender), or if they've actually used the app that's being used as a proxy. None of those things are necessarily a good assessment of whether the person can do the role they're applying for, so the test fails.

disintegore 7y ago
If you are required to demonstrate that you can create pivot tables, and you not only know what they are and how they work but also figured out how to do it in a brand new software suite within 10 minutes under stressful conditions, you have succeeded. Committing obscure UX patterns to memory is far less important than your ability to understand concepts and find ways to apply them. I understand a manager may prefer to supply someone with Microsoft Office 2001 Alpha Deluxe Edition (& Knuckles) and see them hit the ground sprinting but the time spent looking for such a person might be greater than what it would take for a motivated person to reach that level.
dangerface 7y ago
> If you're applying for a role that expects you to know an application then looking in the help is cheating. If you discourage your employees from learning you are going to have a lot of dumb employees. Even the developers who made excel do not know how to do everything that excel can do.

bryanrasmussen 7y ago

Well this is exactly like a test I went through at one time. It even used the same reasoning pretty much. The job was fullstack node and react, I went in and they said we want you to look at this game written in Python which is really slow and tell us what is wrong with it (I hadn't done python in years and back then wasn't very good at it). We will sit with you. I said I was game, they said lots of people weren't - one guy evidently got offended and said he didn't do that language (I can definitely think of one friend of mine who is totally competent who would get offended at being asked to work in a language that did not have anything to do with the job he was applying for) The game was using a big dictionary in an array and you had to look up words from it and it was slow (about 7 minutes to do searches). So I thought to myself, damn I cannot remember the name of that algorithm for this (binary search), ok let's stall a bit maybe I will remember the algorithm name. So what's the first thing you would do? Uhm how about checking a profiler. So we checked and found some simple things we could optimize. And then I looked through the code and I saw there were parts I could cache etc. So after all those things were done the time to do a search was a little bit more than halved and I still couldn't remember "binary search", so I stared dumbfounded at the screen for I think about a 3-4 seconds and I started to say "uhm" and then the guy sitting with me I guess ran out of the time he wanted to spend and said "ok well there is an algorithm for this blah blah blah" showed how to do it in python. Thanks for your time. I actually didn't want the job but still felt somewhat let down that I couldn't remember.

js2 7y ago
If this was Python 2.1 or later, too bad you didn't check the stdlib: https://docs.python.org/2/library/bisect.html https://docs.python.org/2/library/bisect.html
marvinjames 7y ago
Of course there was pressure, but why not just say that you know an algorithm that can do this faster, but you can't remember the name of it, and then just provide the intuition behind it?
- hellisothers 7y ago

itronitron 7y ago

rather than give them the application, just show them the file (location, name, and extension) and ask them to make some useful revision to it

gota 7y ago

I don't mean to be derisivs but at that point - testing for reasoning, understanding, personality - Why stick with a software tool? You'd be better off arranging to play a strategy, preferrably collaborative, boardgame. One day I'll use Pandemic as an interviewing tool. I wonder of it'd work as a pre-formal interview stage

edent 7y ago
I've been to group interviews where they pretty much do that. Randomly divide all the candidates into groups, give them a task, see how they work together. I can't say I enjoyed the experience - and I've no idea how good it is as a tool - but it was illuminating to see how people treated each other.

tantalor 7y ago

See also Kobayashi Maru "a test of one's character or a solution that involves redefining the problem and managing an insurmountable scenario gracefully"

lettergram 7y ago

I think the primary issue with interviews is that the interviewer doesn’t share expectations. I think it’s totally fine to evaluate how a person asks for help, as long as it’s clear they aren’t expected to solve the problem. Further, you’d need some standardized way to evaluate how well someone asks for questions. Otherwise you’ll have interviewer bias. Which is why designing interviews are very difficult.

deleted 7y ago

[deleted]

arkanciscan 7y ago

After a couple of years working as a web developer, I decided to go to College. I took a "Web Development" class. I failed the final. The assignment was to code a website by hand. I used BBEdit to pretty print my hand coded HTML and it inserted a <meta name="editor"> tag that I didn't catch. I dropped out and went back to getting paid. Been doing it for 20 years now, though I use Prettier to do my formatting these days.

avgDev 7y ago
OOf I had the same final. I had to create a website by hand in a basic text editor like notepad++. I passed but I honestly don't see any value in it.

scrooched_moose 7y ago

I'm a mechanical engineer by training and still occasionally do some entry level hiring in it. My favorite question in recent years has been posing an advanced project - not unreasonably difficult, but one I know they won't have covered in school. Make it clear I don't expect them to have the answer, but have them talk through their approach to this project with me. "How would you get started?" "What would you need to know?" "What resources would you use?", etc. For us it's been a great filter - both a gauge of their current knowledge and ability to tackle an unknown.

cushychicken 7y ago
I'm an electrical engineer, and I have a similar question that I ask my entry level candidates - "How would you measure the voltage coming out of a wall socket?" It's familiar enough to be accessible, but technically just far enough out of their reach that they don't know exactly how to do it. We talk about approaches, tools they could use, some potential problems or issues they'd need to overcome. If they get flustered about where to start, I laugh, and say "Don't worry - I don't know the right answer. I'm actually afraid of AC mains electricity." Never fails to calm them down, because, well, it's kind of funny to hear an electrical engineer admit that they're afraid of electricity! It's a question that I've gotten a lot of mileage on.
- perl4ever 7y ago
  Is a religious certainty you need a true RMS meter a plus or minus?
- _AzMoo 7y ago
  Couldn't you just jam a multimeter into it?
  - cushychicken 7y ago
    You could! My immediate followons to that would be: - How would you jam a multimeter into the socket safely? - What information will the multimeter not tell you about the AC voltage signal?

haolez 7y ago

Another take on what the author proposing is to simply not care about the output of the task, but focus on how that person approaches the problem (even if they don't manage to complete the task at all).

Davertron 7y ago

I hear a lot (and believe myself I suppose) that I would rather hire someone that knows how to learn than someone who currently knows thing x but can't pick up and learn things on their own. But I feel like it's VERY hard to figure this out about someone. It's pretty easy to test how well someone knows a particular technology or tool (given some variance for operating in a high-pressure scenario etc.) but it seems pretty hard to tell if someone can learn new things quickly, and learn the right things. I feel like this is what algorithm tests are trying to do, but I also feel like a lot of algorithms are based on having a key insight into a problem that is, at least in my experience, pretty hard to have when you're in a room with strangers on a timer and being judged, so if you don't know the particular algorithm to apply, good luck. And also it's pretty easy to go study algorithms and hope you remember how to implement DFS if that's what they ask you in the interview. And there is some non-zero value to just knowing about different data structures and algorithms for certain types of work for sure. I don't think many of us web developer types will be implementing DFS ourselves, but I run into issues pretty often where having knowledge about different data structures and algorithms comes in handy. So I can see the appeal to doing a test like this that has nothing to do with a particular skill, where you're trying to evaluate someone's judgement and how quickly they can absorb new ideas or figure something out on the fly etc. BUT, I just think there are too many factors that can really muddy this up, and I still don't feel like there's an objective way to measure "smart and learns quickly". Even "learns quickly" is somewhat vague; I personally wouldn't expect someone to pick up, say, functional programming in an hour just by saying "go read about it." Some things take longer to learn than others. Anyway, there's no silver bullet. I still feel like we end up making pretty subjective decisions about other people when interviewing them. I don't think we're even all that aware of them ourselves. I would say though that if you're interviewing somewhere, you're getting useful information about that place by how they interview you, and so I wouldn't feel too disheartened if you feel like the interview is unfair or doesn't work for you. That's probably a good sign that you wouldn't enjoy working there anyway.

GreenJelloShot 7y ago
I totally agree. The best solution I have come up with for trying to assess if someone can learn new things quickly is simply to ask a candidate to describe the last new thing that they learned. It doesn't have to be job-related at all. Just the last thing they learned. And then ask some follow up questions about how they learned, why they learned it, and how have they used the knowledge. If nothing else, it will give you a general idea of how often they learn new things.

FiddlerClamp 7y ago

I had that same automated Microsoft Office test at a temp agency in the 1990s. Funny thing, it had a bug. As long as you didn't let go of the mouse button, it would let you open and navigate all of the menus. So I simply browsed the menu UI until I found the correct choice. 100% score.

lbarrett 7y ago

This reminds me so much of Vernor Vinge's story "Fast Times at Fairmont High". It's set in the near future, and one plot point is that high school tests cover this skill--to read through the manuals quickly and get things done with new tools. I recommend it, and the related book "Rainbows End".

catern 7y ago

I don't know about this. I don't think I should spend most of my time learning completely new tools. If people spend most of their time learning tools, that encourages tools which are easy to learn but lack deep and powerful functionality. I think it's fine, and even good, for tools to require focused learning to understand, and in exchange have a worldview and tool use that is really powerful. Like Emacs or vim or Excel, for example.

tracker1 7y ago

I partially agree, and it depends on what you are interviewing for. In the end, getting a job done, on time, without error/bugs is generally most important to the people you are going to be working for. As for algorithm questions at some FAANG or wannabe startup, it can be rough. Personally I tend to blank on a lot of terminology, similarly on names a lot of the time. Some trivia games kill me, I can see the answer's face, the movies they've been in, but just blank on the name. All the same, in my life and career, I've managed to learn, adapt and create both simple and complex systems in new languages and platforms. On the other side of the table, I tend to ask maybe 3-5 trivia like questions only to gauge where a person is at, and let the conversational parts determine if the person would be a good fit. I think you need some of both. I feel drive accounts for more than even skill, knowledge and experience.

perl4ever 7y ago
Maybe you just can't figure a person out in an interview. I got my current job as a contractor/temp, and thinking back to a previous company, one of the people that was the best and stayed the longest was a contractor that "graduated" to permanent. The simplest way to prevent people from keeping up a facade is just time.

reallydontask 7y ago

> I'd tell them that at the start, obviously. This isn't The Secret Rules For Getting Hired. > "I know you've never used Blender," I'd say, "And this job doesn't require it. But we want to see how quickly and accurately you can learn to use something unfamiliar while under pressure." > Or whatever. The main problem is that the level of pressure that people feel under interview conditions will vary dramatically. Personally, I would find this a very bad test as I get horrendously stressed at interviews yet somehow manage to portrait a facade of calmness (to be fair I've been told once that I looked remarkably calm). To the usual refrain of: If you can't handle the pressure of an interview then maybe you are not a good fit for our company I normally reply: If day to day working at your company generates interview like levels of stress, then please do not offer me a job

perl4ever 7y ago
If someone says there was a lack of culture fit, it stings and certainly could be some type of illegal discrimination. But at the same time, it's necessarily true! Unless your culture is discriminating unfairly...

myself248 7y ago

I was pretty happy with the interviews I went through for my current position, although that might be selection bias because I passed them. They all shared some important attributes: 1: Fairly open-ended, there was no "bzzt-wrong" moment like the Office 97 case in TFA, so I didn't feel a lot of pressure. It was definitely enough rope to hang oneself, so the interviewers could form an actionable and objective judgment, but as a candidate, I didn't feel that I was on trial, more just chilling and talking tech with some other nerds. 2: The underlying skills (understanding one's audience, breaking a large problem into pieces, making reasonable assumptions) were obviously directly applicable to the position. I never felt that I was doing senseless work, even when the questions were clearly contrived, because the connections were self-evident. 3: Despite the above, the actual questions didn't require a lot of domain-specific knowledge. Even the one that sorta did, could be answered by assuming that it was similar to a more common system, and that's how I approached it. 4: In every case, there were "plan B" options and accommodations for having a gap in one's knowledge. "I don't know this off the top of my head, but here's where I'd go to find out and here's the guess I'm going to use in the meantime" was a valid answer. Perhaps the best answer in some cases. The result is that I work with a team of people who are super adaptable, can think on their feet, but have a keen interest in pushing towards correctness as soon as real information is available. Whenever I think about possibly going elsewhere, I remember that most HR departments don't select for any of those attributes, often quite the opposite. (And those places tend to be the clients whose problems we get called in to help solve, as a result.)

dcole2929 7y ago

The single best interview experience I've seen was given by Oracle when I worked there a few years ago. The onsite was a panel interview where candidates were given a laptop with a configured IDE and codebase with a specific set of bugs. Their task was to debug the various issues and complete as many as they could with the understanding that it was not expected for them to solve them all or even most of them. There were also tests to help them. And a different section of the interview asked them to right some tests and some code for again brain dead easy coding algorithms like reverse a string. I thought it was great because it closely resembled what being a dev is like and actually tested for the skills we cared about. No you didn't get to use your own custom vim configuration but I'm not sure how that would have really worked even if they wanted.

city41 7y ago

At my previous company, the technical interview was 2 hours long. The candidate was given a choice of about 10 very small, simple projects. Then we sat at a pairing station and paired on the problem. The candidate can use google, their favorite editor, write tests, anything they want. The goal is not to finish the project, but just code with them for 2 hours. It's the best technical assessment I've yet seen in an interview setting. One caveat is this company really believed in pair programming, and virtually all code was created in pairs. So that's part of why this interview approach worked.

organsnyder 7y ago
I did an interview once at a company that is religious about pair programming. I found the exercise unsettling: would they evaluate technical ability or culture fit more? If they were focusing on the former, I should take the wheel and try to demonstrate my chops (perhaps making me appear arrogant); if the latter, I should try to demonstrate how I can collaborate with the other developer to make both of us better (perhaps at the expense of me not demonstrating as many skills within that brief window of time). It turns out that they were really focusing on how I interacted with the fake "client" and whether I had the proper "consultant mindset". Didn't see that coming (perhaps that's my fault). Though I feel I ended up dodging a bullet in not getting an offer from them. My suggestion is to try to give candidates specific times to demonstrate various competencies (technical, colleague interaction, customer-facing), and tell them which ones are the focus of each exercise. Yes, that's not how the real world works (you need to utilize them all at the same time, of course), but an interview setting is hardly the real world. And you'll still see glimpses of their overall capabilities in each stage.
- city41 7y ago
  That's a really good point. I honestly don't remember if we gave the candidate a heads up in that regard. In general interviewing is extremely imprecise. It's an unsolved problem unfortunately, but I feel most companies "fix" the problem by just over-interviewing and being ok with false negatives.
- baumy 7y ago

learnstats2 7y ago

This made me laugh - I first applied for temp jobs around this time and there was a typing test. "Wow," said the person who had left me alone with a computer that felt old in the 1990s, "I've never seen anyone score 50WPM before." "Uhh... I could type a lot faster, but the letter T is sticky and I had to delete a lot of Ts" She looked mortally offended by this. I was given a job as a dishwasher which I was sacked from after one shift.

LanceH 7y ago

I've always wanted to run one where we bring them to a room to interview. Maybe take a break. Move their stuff to a slightly different room. Pick back up right where we left off. Maybe change out the furniture and people outside during the interview. Introduce them to numerous employees, who will respond to names swapped amongst each other. Another break - switch interviewers. Pick up right where the first one left off - insist that this employee has conducted the entire thing. If they drove there, it would be best to move their car. I'm not sure what position I would be hiring for, but this is the interview I would like to conduct. Other possible ideas would be a background check that is either insanely deep (so you're 3rd grade teacher was unavailable for a recommendation) or raises issues that they have no part of, "we see you've spent time in North Korea."

notkaiho 7y ago
... But why?
- jedieaston 7y ago
  To see how they respond under pressure.
  - LanceH 7y ago
    Today's changing markets demand employee's who can adapt quickly. Or static old companies seem to want employees who don't notice change around them.
  - notkaiho 7y ago
    I'd be very confused and ask questions as to the reasons they're fucking around.

neo2006 7y ago

I do hiring interview in my company and one of the test we like to do is to give the interviewee a problem out of his area of expertise and open a browser and an editor for him and tell him that he is allowed to search for things. Most of people I interviewed do not use Google help they are afraid you judge them negatively for it even if you say you will not.

ngneer 7y ago

Interviews should be personalized. That they are conducted en masse with little consideration for applicant background is an artifact of the difficulty of adequate personalization. Targeted ads are valued precisely because they are targeted, presumably increasing the chance of a match. Has anyone heard of any startup companies in the personalized interview space?

Aeolun 7y ago

I like how one of the comments is about someone getting a job by telling the interviewer they knew what a computer was.

scarejunba 7y ago

Haha, that's funny, and an awful idea. At a previous employer, as a hiring manager, I offered candidates an option of bringing in their fully-configured IDE in their own laptop with the language of their choice and I'd give them the problem to solve. Essentially, act in the environment that you've already mastered. Only one guy did (I had maybe 10 a month) and for some inexplicable reason he couldn't figure out how to create his `main` equivalent in his language in his IDE. He must have been incredibly nervous because we tried "perhaps you could sketch out your thoughts in broad strokes and then we can return to the IDE" etc.

Iv 7y ago

I just got interested in the way the school 42 works. Especially the month-long "test" that they have to accept tuition free students. The idea is to make you complete a lot of programming projects in languages and tools you normally don't know at all. When you start. The evaluation does not only take into account if you finished project but also if you helped others, knew how to look for help (basically: google or other students). Its goal is to select students who would work well in a peer-learning project-based-learning paradigm. This, in my opinion, is much closer to real-life conditions than any academic test.

Tade0 7y ago

There must be a serious dearth of candidates in my area, because the last time I was subjected to a time-constrained test was almost three years ago, and it was a Codility challenge, so I had some time to prepare for it. I quit my job recently, and in order to find another I went full-auto with the applications - I've sent close to thirty of them I think, so a long series of recruitment processes followed. They were all mostly the same - I either received some homework to do or had a 1-2h technical interview during which I was asked a set of questions which were easy for the most part. And that was it. My point being: I've discovered that I can't really relate to the stories other people posted in this thread. Does this mean I wasn't ambitious enough regarding my applications, or are the others subjected to a difficult job market? Maybe both? Or none? I really don't know.

stedman 7y ago

We do this at retriever.co for our virtual assistants, in two areas: 1. we have applicants do a data transform with an esoteric excel function that they've /almost surely/ never used, and prompt them to find videos / tutorials on how to use it if they don't know (which, we assume, they wont) 2. we have people answer fake customer support tickets that require them to do some on-the-job learning about the Internet Service Provider industry -- to a level of detail that requires that they read through and grok a page on wikipedia explaining up-time calculation. While neither of these pieces of knowledge are necessary for their day-to-day at Retriever, they allow us to stratify candidates' ability to learn new skills and grok new information. 100% would recommend!

aniketpanjwani 7y ago
I've been thinking through how best to screen personal assistants, and these are both very helpful ideas. I particularly like the first idea to screen applicants on esoteric utilities - thanks for your comment!

Iv 7y ago

I keep hearing and reading horror stories like this one on HN and other website. I never did interviews at big companies, nor did I do any in the US, in 15 years of freelancing, I only had once a technical interview, with a C++ guru, I failed most of the questions but still got hired because "you still did good, kept calm and asked good questions". In my last gig, it was for a work on a Java application. I was upfront that I only did a bit of Java in school, but that I was more of a C++ guy. The Java guru hirer was like "Yeah, if you know C++, you'll get Java quickly" they were more interested in my experience with video and image processing. I basically avoid to do HR-based interviews. Short-circuit them when you can.

deleted 7y ago

[deleted]

deleted 7y ago

[deleted]

sylvainkalache 7y ago

That's how Holberton School (we train people to become software engineer) select and train students. We are a college-alternative where students learn by working on projects, in group. The whole education is flipped compared to a regular one. We give students problem sets and they have to find the answer. We never give lectures (we have no teachers), just guidance so that they can get started on a topic, but never enough that they can actually achieve the tasks we give them. > Are they able to read a manual? Candidates (no prior experience in coding needed) need to answer questions on a specific command that can be found by reading the manpage. At first we provide the manpage, then we show them how to find a manpage, and finally, we don't provide any guidance, because at this point they should be able to do it on their own. In the curriculum, once students, we have a quizz to makre sure they can answer basic questions on the topic before jumping into the coding part. >Can they formulate a search query? We ask trivial questions that can be answered just by Googling. Might sound like a no-brainer to the developer community, but not everybody is good at this. >How do they assess whether the tutorial they found is suitable or reliable? That's the tricky part, but basically, because the school is not providing material that contains the answer, candidates and students have to navigate the ocean of information/tutorial that can be wrong/incomplete of correct. By pressing a button, students work (code at this point) is automatically and instantly corrected, so they can easily figure that what they built is correct or not. > What steps do they take to make sure they're finding - and learning - the right information? On this one, we as a school, are the one providing this guidance. Basically when students are doing their projects, we define learning goals. Getting coding part done is great, but we also want to make sure that students have been able to grasp concepts that are key to becoming a great software engineers. So students look at these "learning points" before, and by the end of the project, they must be able to discuss all the points.

andrewfong 7y ago

Honestly, I wish more companies would just tell you in advance of the interview what the specific prompt is going to be (and perhaps the evaluation criteria too). Sure, you could "cheat" by Googling the answer in advance or asking someone for help. But if the question's sufficiently open-ended, it's not that hard to suss out in follow up questions whether you have a deep first principles understanding of the subject area or you're just repeating something you read.

bagacrap 7y ago
I'm pretty sure if you give me a subject I know nothing about and a couple days to study I can teach myself about it, at least to the level of someone who has real experience in the subject which may be a couple of years out of use. So then how does a hiring manager know to filter me out?
- andrewfong 7y ago
  Presumably, at some point, you'll be interviewed by someone with more up to date experience. And if you study enough to fool that person, that seems fine? It's a good sign you're capable of the job, prior experience be damned.

myegorov 7y ago

fwiw this describes my experience of interviewing at Databricks