7 ms·
This is definitely true. The filters that companies and recruiters use mostly trend in a meaningful direction. There are a higher percentage of good programmers
by ammon 11y ago
This is definitely true. The filters that companies and recruiters use mostly trend in a meaningful direction. There are a higher percentage of good programmers among MIT grads than SUNY Potsdam grads. But this fact does nothing to help the good programmers who look bad on paper who everyone is ignoring. This is precisely the problem that we're trying to fix with Triplebyte. We want to make actual technical evaluation (not credential evaluation) cheap and accurate, so we can just interview everyone.
- sillysaurus3 11y agoThe filters that companies and recruiters use mostly trend in a meaningful direction. There are a higher percentage of good programmers among MIT grads than SUNY Potsdam grads. Imagine how mistaken that would sound if you substitute "startup founders" for "programmers." The reason you feel company filters are anything except random noise is personal bias. It's disheartening that your article notices that company preferences are essentially random noise, and then you don't really internalize that fact or take it to its logical conclusion: Interviews where programmers do anything except work-hire tests tuned to the signal that the companies want are doomed to fail. And it's almost impossible to do a work-hire test in an interview. A real work-hire test. The type of test where you actually do some work and then show that yours is effective. We're so backwards that people probably read this and think I'm saying whiteboard work or something stupid like that. No. Real, actual work. The problem you're working on can be fake, but the problem needs to be literally the same type of thing that the company does on a day-to-day basis. Want a 100% success rate? Do that, and then don't pay attention to anything but the results. Including whether the candidate has an MIT degree, or any degree at all. If you actually set up a work-hire test, a real one, and then stop asking about all the other stuff that doesn't matter, then you will get a 100% rate. Again, you have to tune the test to what the company is looking for, but that's not hard. "Here is an iOS app. Find and fix these bugs, then add a new feature."
- dmoy 11y agoHow do you do a work-hire test for someone who already has a job, and is not allowed to work on another company's stuff?
- vonmoltke 11y ago> The problem you're working on can be fake I'm assuming if it was a fake problem, it wouldn't be something the other company could actually use.
- ryandrake 11y agoIt's not about what they could use. It's about what they own and where it's going. If your employment agreement states that any code you write belongs to your employer and that you're not allowed to share company code with others, then a work-sample test essentially asks candidates to violate their current employment agreement.
- vonmoltke 11y agoTaking that interpretation would mean any interview where you write code would violate your employment agreement.
- sillysaurus3 11y agoEmployment agreements like that are certainly frustrating. I've had experience with a few. One thing to realize is that your employer can't really retaliate. The hypothetical iOS app I mentioned is set up for one purpose: to let candidates show they can do work the company is looking for. Not only will the code be thrown away, but it wouldn't make any sense to use it.
- s73v3r 11y agoI would say that if the new company is not in the same section of the industry, than enforcing that agreement is entirely unreasonable.
- citizens 11y agoIf all of an applicant's code is owned by their current employer would "whiteboard" style code also be prohibited in an interview? I guess I'm wondering where the line is drawn.
- ammon 11y agoThe cost of a real work test is much higher, both to the company, and to the applicant. Some companies take that approach (weebly for one), but it means that some of the best programmer will not apply, and it only makes worse the problem of having to aggressively pre-filter (based on credentials, or something else). You are 100% correct that most research into interviewing has show it to be far less predictive than companies believe. However, I'll add that no one is doing the false negative studies that would really be required to show this. For example, Google has shown that among people who pass their interview GPA is not correlated with success. This does not at all mean that GPA is not correlated with programming ability. In fact, it almost certainly is. People with low GPA who pass google interviews have a lot of other things going for them. To really analyze this, you'd have to give a job to a random sample of applicants. No one has done this. My goal is for Triplebye to get large enough that I can.
- Alex3917 11y ago> No one has done this. Not for coding specifically, but there have been some studies correlating GPA with career success, and the correlation has been found to be very low. But the reason there are very few studies that look at this is that GPA A) isn't epistemologically meaningful B) isn't a measure in the mathematical sense. "A grade can be regarded only as an inadequate report of an inaccurate judgment by a biased and variable judge of the extent to which a student has attained an undefined level of mastery of an unknown proportion of an indefinite amount of material." -Paul Dressel
- sillysaurus3 11y agoThe best programmers do apply. Whatever you think a work-hire test is, that wasn't it. A work-hire test attracts the best programmers. It doesn't repel them. It's a chance to demonstrate their skill. It's also far better: anything is better than the fake contests they're currently forced to endure. Which would you prefer? Spend a couple hours remotely fixing some bugs and adding a feature on a fake iOS app, or spend all day playing "dodge the bias" for a hit rate of ~60% at the cost of a vacation day? Why do you think the programmer from your article didn't get a job when you "sat back to watch him get a job"? You didn't tune your test to what the company was looking for. Any YC founders who are reading this: Refuse to talk to Triplebyte until they set up a work hire test for your company. Force them to do it, and force them to work with you to tune the test to the work you need. You will get a 100% hit rate. And you'll notice something else: Your retention rate will go way, way up. Want to not deal with firing people? Get them to show you they can do the work. Nothing else matters. The test needs to be set up so that the candidate can demonstrate they can do exactly the same things all other employees do on a day-to-day basis, or it's not a work-hire test.
- _delirium 11y ago> Imagine how mistaken that would sound if you substitute "startup founders" for "programmers." Isn't that pretty common among incubators and VCs? It's not the only factor, but for first-time founders, being from Stanford is a large plus when it comes to getting in the door. If you already have a track record then of course they just go by that. But when it's your first company, I don't think the "startup ecosystem" is any less reliant on these kinds of signalling credentials to filter the large volume of people who want funding.
- waterlesscloud 11y agoAnd the vast, vast majority of startups fail. So.
- rowborg 11y agoFor purposes of discussion, let's take it as a given that a work-hire tests are the best way to screen candidates. The effort required to administer an on-site work-hire test is non-zero, therefore I cannot administer such a test to every applicant. I therefore need a way to determine who to bring on-site, in order to administer such a test. I cannot phone screen every single applicant due to the cost involved. That process could also be a work-hire test of some sort (e.g. a remote coding project), but regardless of what has been said on this thread, many good applicants will drop off at this step. I know this empirically, because I've experienced it, many, many times. Additionally, the people who tend to drop off because of this extra effort tend to be senior applicants, who often have multiple offers from multiple companies due to the competiveness of the current hiring market. It also disproportionately drives away passive candidates, who are often the best candidates, because the best candidates are often not looking for work (since they are good, people who have worked with them previously want to work with them again, and they get poached). So I need some method of sourcing and filtering candidates down that is non-intrusive to both our development team AND the applicant. This is the reality. This simply has to happen in a startup's hiring pipeline. Currently, most companies do this by looking at resumes. That is obviously sub-optimal. Any suggestions for alternatives? EDIT: spelling.
- sillysaurus 11y agoI do. I was going to bail from this thread, but it sounds like you see the great potential that this idea can have, if it can work. The truth is that it works. Okay then, one last try: Set up a system that can spin up a droplet for a remote candidate. Any time a candidate expresses interest in your company, spin up an instance and email them a link to it. What does the link do? That depends on your company. Are you making an iOS app? Then the link takes them to where they can download source code for a fake, hypothetical iOS app. It says "X, Y, and Z bugs exist. Find them and fix them. Then add a feature: here is a clear description of what to add." When the candidate is done doing this, they zip up their code and send it back to you. If it sounds way more effective to look at that than to look at resumes, it is. If it sounds like it will repel candidates, well... Two things. First, if you're chasing a specific developer, then that isn't really the normal hiring process. You want them already. This pipeline is for everyone else. It makes no sense to subject them to a work hire test when you're actively seeking them out. Here's the other point. The type of candidates you will find with this method will shock you. They will be so skilled that it won't matter whether they're called a senior or fresh out of college. You'll know immediately that you want them. Everything I've described up to this point is a remote process. There is no on-site work hire test. By the time they come on site, you're mainly checking they can show up, and telling them about your company. You're no longer trying to filter them based on ability; they already demonstrated it. Let's say your company's website is the primary focus, not an iOS app. Ok. The link will take the candidate to a hypothetical, fake website built with a similar framework. Again, it will have multiple bugs and a missing feature. Tell them what the bugs are, and tell them what the feature needs to do. Then have them send you their code when they're done. I feel like at this point no one will even try to do this. You can think of so many reasons not to try: it takes too much work, it will scare too many people off, it will... Etc. These reasons turn out to be largely fake or mistaken. Try it. Invest the resources to build this pipeline, tell HN when it's ready, and you win. If this sounds prohibitive or unlikely, remember how counter-intuitive the most effective techniques in life are. Penicillin was discovered by accident. It sounds pretty unlikely that it would work. Same deal here. I've explained this as clearly as I can. It's up to everyone else to either try it or to watch others win after they try it. Because the filter I've explained is the only way to let talent find you. The type of people you'll discover will range from passive people who found the process amusing, to well-off senior developers who are demonstrating why you should pay them X equity or Y salary, to high school dropouts who turn out to be one of the most valuable people that join your team. I'm not even going to touch the topic of what tech companies currently do. It doesn't matter. I've described what works, and if whoever reads this suppresses their instincts and builds this, they will discover it's practically the key to winning.
- vellum 11y agoThis also depends on how clear the company is about the hiring process. A lot of companies have started using the coding projects before the resume screen, and then also throwing in an all-day whiteboard session. So when someone's been burned before, they're going to do a 180 when a company replies, "Do this project."
- 20years 11y agoYes to this 100x. I actually did this last year with outstanding results. We were looking to bring on paid interns. We narrowed all the apps down to 5 and paid each $50 to complete real stuff that we use in our environment. 2 out of the 5 completed the work within an hour, 1 took 3 hours and the other 2 returned uncompleted work. We hired the 3 that completed the work and the one that took 3 hours ended up being the best out of them. This person also had very little prior experience but what I liked was that she stuck with the task and not only found the bug but improved some other code within the app. All 3 ended up being very valuable and 2 of them stayed on well beyond their internship. Oh, and these projects were done from home before we ever brought them in for an interview and we hired them before ever meeting them.
- sytelus 11y agoWork-hire test works for positions that requires failry narrow skill sets. You need iOS 2D game developer or nodejs to MongoDB plumber or single page JS app developer? Sure it would work beautifully. However most companies are not looking for such very narrow skill sets. Companies look for people who are generalists at heart and can be specialized in given area with little ramp up on demand . One week you might write build script and other week debugging Rails and yet another week fixing javascript in UX. When doing these tasks, it's not expected that you already know all these tools and platforms. You will have some ramp up time of 1 day to a week. But the key is that you should be able to move around stack, across projects and get shit done given reasonable ramp up times. Also a lot of experienced programmers are generalists as well. You might have been C++ system guy in your past life, worked on JavaScript couple of years back and now working on Hadoop backend. You may not remember all the nitty gritty details of past platforms and tools but you can ramp up on demand. So this 2 hour work-hire test that narrowly focuses on specific tools and platform won't work for vast majority of programmers who haven't branded themselves in to one narrow area. Also remember that a lot of programmers at places like Google, FB and Microsoft work in highly proprietory environment with their own internal build system, platforms and tools. They are likely not familiar with everything that's latest and in fashion. However they are very smart people, can change gear easily and, most importantly, ramp up on demand very quickly. Yet another big issue is that work-hire test usually fail to measure hardcore computer science problem solving abilities. Sure, you are good at fixing some iOS bugs but can you figure out how to do driving directions on maps or may be scale email system to 100 million users? Now I do agree that lot of jobs don't require hard core CS but many do. And for a startup, it's usually impossible to tell if your future 6 month down the line will hing on having someone with hard core CS skills.
- hibikir 11y agoIf you want generalists, you won't really be able to test them properly by asking questions at random: All you are hoping for is that they share the same background as you do. The moment it's you asking the question, and they can't really check sources, you have already lost. As far as hardcore computer science abilities, a test still won't help, because what you are really testing is how far people are from college: Generalist cs is touched in college, but there are entire sections of computer science that someone with 10 years of experience might not have had to touch. I've had to do pretty specific CS stuff for work, but to get any of that done, I started with weeks of research, and then add my own spin to our specific situation top. You can't test the ability to do that by just checking if someone has memorized Raft, or black red trees, or any other random problem. I don't have a lot of CS algorithms memorized: that's why there's the internet, and why I have Knuth in my shelf. You want to check for hard cs, ask the candidate to choose his favorite CS topic, and ask for am impromptu presentation on that. anything else is just going to fail at finding candidates that have different backgrounds than you, and that's PRECISELY the people you want to hire.
- LordHumungous 11y ago>The filters that companies and recruiters use mostly trend in a meaningful direction. There are a higher percentage of good programmers among MIT grads than SUNY Potsdam grads. Uh are you sure about that? Google's experience seems to be that academic achievement is not a good predictor. And they know a thing or two about hiring. http://qz.com/180247/why-google-doesnt-care-about-hiring-top-college-graduates/ http://qz.com/180247/why-google-doesnt-care-about-hiring-top...