9 ms·

Hiring and the market for lemons (2016)

128 points by ff_ 8y ago

andrewstuart 8y ago
When the topic of "great" developers comes up, I always challenge anyone to define precisely, in quantifiable terms, what a "great developer" is exactly. How do you know someone is a "great developer" if you can't, in tangible terms, say exactly what a "great developer" is? This is even more of an issue when recruiting because recruiting is an assessment process to determine if someone meets some requirement. But just like software testing, if you can't quantify the requirement, then you can't quantify a test to determine if someone meets that requirement, and therefore the interview/assessment process is simply voodoo recruiting. Voodoo recruiting is when you have the appearance of scientific process in recruiting. - a process, - a set of questions that appear good, - a coding test that seems technical enough that it must be telling you something meaningful about the candidate, - the opinions of many people on the candidate. But in fact all of this is voodoo because you still can't directly define the requirement and thus you never known if you ever found what you are looking for, which is not actually known. The final decision in recruiting a "great developer", as always, comes down to some subjective opinion based on a range of criteria that can't be proven to have had any meaning in the first place.
- quotemstr 8y ago
  Was Heaviside a great engineer? Greatness does exist. I'm really very sick of the recent trend toward denying the basic idea of competence.
  - ItsMe000001 8y ago
    You create a strawman. Nobody denies what you claim. The issue is not at all the existence of "great people" (for anything). The problem is how do you find them. You have a thousand people in front of you. Now please make a prediction that actually works reliably for big numbers of people. Your argument suffers from the same problem as any of the others that always come up after something happened, "why didn't they...". You look backwards. You make a "prediction"about something in the past and of course you get 100% accuracy and think "that's easy". Okay, now please make tat prediction for the future. You have people in front of you whom you know zilch about. Now make - en mass - statements about their future work that when looking back from the future will still look good. About your > Was Heaviside a great engineer? Now please make that backward-looking prediction about his greatness for the same person before it turned out that he was good. And not just for one person either, do it for thousands of not-famous people with some degree of consistency. And then report about all your predictions, not just the few that worked, and don't forget to mention if you did some extreme filtering beforehand in order to ensure you get a lot of "hits", leaving a large part of potential candidates without any chance to begin with. That kind of thing works for some (like so many "recipes for success") but not for society.
    - sokoloff 8y ago
      > You have a thousand people in front of you. Now please make a prediction that actually works reliably for big numbers of people. The simplest answer is to predict that 0 of the 1000 are great developers. That will be reliable for most cohorts of 1000 people and most active industry applicants for developer positions. You can only reliably find greatness by filtering through many college applicants, training and mentoring them, keeping, motivating, and incenting the best, or by using networking and references to find the standout performers. It’s so rare for these industry experienced people to need to apply for a job cold that that is a tremendously adverse pool from which to select in my experience If greatness is the goal. If good enough is the goal, there are plenty more of those devs available.
      - pkaye 8y ago
        So your plan would be to hire all the college applicants that applied (or a random sample) and then mentor and train them to see who works out the best?
        sokoloff 8y ago
        No, you hire the ones who seem interested, skilled “enough”, and intelligent (possibly in reverse order of importance). You can very safely make a cut against some very likely poor candidates. But yeah, you have a light filter and then let them grow and evolve. You can’t possibly learn in 4 hours anywhere near what you can in 2 years. In college recruiting, you’re looking for “could be solid” and you get the raw material for that a lot. Some happen to turn out great. Finding greatness in industry is driven mostly by asking some of your best employees to search their networks for their best ex-colleagues and otherwise by dumb luck (someone tried a startup that went bust and no one in their network had an opening, someone is moving to a new city with no network, etc)
        pkaye 8y ago
        Those three traits are exactly what I use for college recruiting. I guess some people look at great developers on an absolute scale but I consider it on a relative scale within the role under consideration. I consider those three traits make a junior candidate great. Actually those same traits are important for all roles. It is just that for junior roles I don't expect much beyond that.
- dfabulich 8y ago
  I strongly agree. This problem applies both to the interview process and to evaluating the performance of engineers after hiring them. A lot of companies try to use supposedly "data-oriented" approaches to hiring. They assume that a "successful" interview process hires candidates who get good performance reviews a year later. But those reviews are just as subjective as the hiring process. If your metric for "success" is whether you still like someone a year from now, you might as well just hire whomever you like today!
  - andrewstuart 8y ago
    Yes and no. I still think you can have an effective recruiting process which attempts to identify someone's attributes across a range of criteria - I attempted to identify some of those characteristics here: http://www.supercoders.com.au/blog/50characteristicsofagreatsoftwaredeveloper.shtml http://www.supercoders.com.au/blog/50characteristicsofagreat... You would fail entirely in your recruiting effort if you just hired "whomever you like today" - you must make some attempt to measure a persons capabilities. What I am attempting to dispel is the idea that it is possible to measure someone as a "great developer" because it is a matter of opinion - person A may think someone is a "great developer", person B may not. I also think that pretty much every recruiting process at every company is "Voodoo Recruiting", or "Recruiting Theatre" - the appearance of rigor and scientific approach, but really just a show that allows the assessor to feel like some scientific, justifiable process has been executed that has resulted in a "correct" hiring "go/no go" decision, but in fact does nothing more than make people feel comfortable with making a subjective decision in the end. Whether or not someone is a great developer also changes according to time and context. What I mean by this is that some developer X who got amazing results in some job that they had, might be judged as being a shit developer when put into a job where they weren't inspired by the project, were working with unfamiliar technologies, where they are in a life phase dealing with depression, or where their colleagues or boss treated them without respect. Suddenly "great developer" in some previous context is now seen as a shit developer in some other context... again, it is simply not possible to evaluate a given person in a job interview and determine if they are "a great developer".
    - 8y ago
- existencebox 8y ago
  I'll chime in with the sister posts expressing vehement disagreement. I can tell you what "great" means to me, in 2(.5) very clear words. "You (fucking) Ship." To me a great dev has a track record of delivering the end result. A greater dev delivers the result faster, with less debt attached. One can fairly say that this is still subjective, but my criteria are _quite_ meaningful and even somewhat empirical/measurable, although it's admittedly hard to do in a short interview format. I certainly know a "Great dev" when I've met/worked with them, because they ship faster than I do, with less friction or need to revisit post-factum. Edit: Since half the responses seem to elide a very key part of my statement, let me echo: _debt matters_. Having someone who ships but burns your team down, not good. Someone who ships but leaves you with years of bugs or un-maintainable code, not good. As I'm sure we all know, you get optimizations against KPIS you measure, and if you don't measure "delivery" holistically, you'll absolutely get people who excel at one component while salting the fields.
  - grantwu 8y ago
    Oh I guess we don't need maintenance then
    - existencebox 8y ago
      I believe you're putting words in my mouth. A great dev can even be great within fixing problems/performing ops/maintenance, by my same qualification. _They ship fixes._ Perfection has very little to do with greatness. Aspiring for it might, but that's a second discussion about unrealistic goals and setting oneself up for disappointment.
      - Analemma_ 8y ago
        > I believe you're putting words in my mouth. You're the one who said you could describe "great" in two words that didn't include maintenance or quality. If it turns out those two words aren't enough, or need lots of asterisks and clarifying words, that's on you.
        8y ago
- smallnamespace 8y ago
  > The final decision in recruiting a "great developer", as always, comes down to some subjective opinion based on a range of criteria that can't be proven to have had any meaning in the first place. Careful, this road can lead to nihilism. Subjectivity is an important part of reality. Subjective things are also messy. We can acknowledge that these aspects are subjective and wrestle with them the best we can, or we can deny they exist at all, and end up pushing those problems somewhere else, where they might fester and become worse. > I always challenge anyone to define precisely, in quantifiable terms, what a "great developer" is exactly. Replace "great developer" with words like "democracy", "truth", "love", "good music", "healthy diet", "pornography". We can't reliably quantify or strictly define any of those things. Sometimes "I know it when I see it" and sometimes not, when cognitive biases and instincts lead us astray. But to deny there is such a thing as a good developer, or a "great" developer, because we don't have perfect metrics, seems like going too far. I agree that having a "pseudoscientific" process can be bad. When you are dealing with something that is inherently subjective, you should resist both urges: one is false precision, a system that claims to cleanly settle once and for all what is inherently messy and difficult; and the other is the denial that that 'something' exists at all.
  - threatofrain 8y ago
    I feel like empiricists routinely and easily perform the partially opinionated act of operationalization, and what's being critiqued here is the <lack> of opinion in operationalization, except for the mocking of scientific appearance. Likewise, many discussions on consciousness are worthy of critique for lacking operationalization. We will always question whether the operationalization is a good proxy, but there's a difference when firms lack even the opinion to do so.
  - ci5er 8y ago
    You're just saying that abstract nouns have no explicit referent like a brick or apple. I take exception to democracy as a noun or attribute like 'truth', however. Truth is an abstract attribute. Sometimes a noun. Democracy is a process: two wolves and a sheep voting on what's up for dinner.
- nostrademons 8y ago
  This is the case for basically all superlatives that apply to human emotions, though. What's a "great company to work for"? It's the company you want to keep working for. That's not going to be the same for every person - I have had great experiences at companies that are commonly known as "great companies to work for", but know coworkers that hated them. What's a "great investment"? It's something you can sell for a lot more than you paid for it. Why can you sell it for more than you paid for it? Because the person you're selling it to also believes it's a "great investment", and so you have a circular definition. What does it mean to "Make America Great Again"? Half the country believes we're doing it right now, and the other half believes we're in a bizarro throwback to the 1950s that isn't going to end well for anyone. So what does it mean to recruit a "great developer"? Well, it's when you hire the person that you wanted to hire. Yes, it's subjective - it's always subjective. That doesn't mean it has no meaning, though, because people ascribe meaning to subjective qualities all the time.
  - sheepmullet 8y ago
    > This is the case for basically all superlatives that apply to human emotions, though. Exactly! What makes a great wife? Or a great friend?
  - joe_the_user 8y ago
    The distinction I'd elucidate from the gp's is that "great" is a problem comes because it's both a superlative and because it can't be quantified. You can ask for a developer who produce a large amount of average code. You can ask for a developer that produces very few terrible errors. You can ask for a develop who produces decent code and happens know domain X very well. You can ask for a developer who can mentor younger developers. You can ask for a developer who might needs virtually no supervision to succeed. And so-forth. They could all be different people. Perhaps you can sift until you somehow find someone who's all that but they'd be fool to join you because you're looking for perfection rather than looking for fulfill your actual needs. Superlatives are a terrible way to make any kind of serious decision. Salespeople love specifically because they tend to be unmeasurable and irrelevant when they are measurable. Thus the more superlatives are entering a decision process, the more you can tell you're sold shine rather than substance.
- dsacco 8y ago
  Are you saying there's no such thing as developers significantly better than the norm, or rather that it's infeasible to identify them? I don't think your point about precisely quantifying the definition of a "great developer" is a good one. I can capture an observation of superior capability in several dimensions without having to rigorously quantify the relative differences in capability or precisely why one is more capable than the other. If one developer accomplishes in a few days what takes another two weeks with code that is at least as maintainable and performant, that developer is better. If that developer is better when put next to most of the other developers you have around you, then they're a "great developer." Your focus on quantifiable definition seems to imply that significant differences in human capital can only be ascertained mathematically, but that doesn't reflect quite a lot that we're already familiar with in everyday life. If I'm placed in front of two walls which are both much taller than I am, I can see which of the two is taller than the other if the difference is significant as long as the tops aren't out of sight. I don't need to quantify this; I could just say, "One looks to be a lot taller than the other, but I'm not sure by how much exactly, or what their respective heights are." I'll happily agree that a lot of tech interviewing (and interviewing in general) is dreadfully broken. I'll also happily agree that figuring out which skills to prioritize and how to judge the level of those skills in candidates is very difficult work. But I strongly disagree that the fundamental premise of that work is intrinsically "voodoo." If your bar is a mathematical formalization then of course you're going to be disappointed, but that also holds true for the majority of our professional and personal activity.
  - andrewstuart 8y ago
    In a sentence, I am saying that pretty much every recruiting process at every company, when it assesses developers, results in a meaningless decision, despite the universal certainty at every company that their recruiting process absolutely definitely results in making accurate decisions about people. Did the interview process find a great developer? Maybe. Did the recruiting process reject great developers? How would you ever know? I'm pretty much certain that most companies reject many more developers who would be really productive/effective, than they choose to hire.
    - 8y ago
- SmellTheGlove 8y ago
  Not only that, but is anyone really "great" across multiple/all domains? I've worked across a pretty diverse set of businesses and functions in management, and I can say pretty confidently that "greatness" is very dependent upon fit for the problem space, and I do mean that to work both ways: Fit is incredibly dependent upon the individual giving a shit about that problem space. No doubt someone could be very good working on something they don't care that much about, but we're talking great here, right? That means talking/writing in broad strokes about great developers approaches impossible. I also don't think you need "great" developers in most roles. Sometimes average is exactly what you need, and hiring above that just leads to having a great person leave because they're not particularly challenged or enthusiastic about the role. And to wrap all of that up, I don't even know that I can objectively define what "great" really means anyway, so maybe I have no place in this discussion :)
  - perl4ever 8y ago
    >Fit is incredibly dependent upon the individual giving a shit about that problem space. Amen. That should be obvious, but I don't think I've ever had any recruiter or phone screen where they acknowledged that in any way. I responded to a recruiter recently by asking what the company actually did, and they reacted like I was just hamfistedly demanding the company name or prematurely trying to research for the interview, but I really just wanted to know - what would I actually be contributing to the world in this job, never mind the technical challenges, the buzzwords, and the paycheck. It's like everyone is assumed to be at best apathetic to the larger context of their job, apart from delusions of grandeur about "changing the world".
- SatvikBeri 8y ago
  I often tell people that before they think about their hiring process, they should think about their performance reviews. You're doing those with multiple orders of magnitude more data, yet most managers will say their performance reviews have a lot of uncertainty. I spent a lot of time at my last job developing ways to measure performance across my team. One big surprise was that I had to significantly change the way we assigned work in order to make it at all measurable. Rather than giving people well-defined tasks, I switched to assigning projects that had some amount of (expected) business value. This meant the projects had to include things like deployment, talking to marketing, and whatever needed to happen to make it useful. Generally each one was 3-6 weeks of work. This provided a good first-order measure of effectiveness. Importantly, it helped people change their habits to contribute more business value. I found massive variation in the junior engineers, easily with 5-10x differences between the most and least effective. The differences in senior engineers were smaller, but still ~2x. Of course, there are other major factors you have to take into account, like code quality and mentoring. I developed similarly detailed systems for measuring those. Once again, you have to change the way you do things if you want to make them measurable, often accepting local inefficiencies for global insight. With all that data in hand, I correlated that to our hiring ratings. The results? Our two best engineers were both borderline in the hiring process, and would have been rejected except for a strong vote of confidence by our senior engineer, who, when we checked, turned out to have a basically perfect track record on hiring judgments-much better than anyone else in the company, me included.
  - majormajor 8y ago
    IME it's hard to precisely rank average devs who are in the middle. There's a lot of clumping. It's a lot less hard to identify overperformers or extreme underperformers. You mention the big possible range, but I've never had to dive that deeply into data to spot them - I've used the data to check my instincts, but have generally been right. And if you don't trust your instincts, just ask your team! They know whose code reviews they dread, who never offers helpful suggestions, who seems to do just the minimum. It's sometimes tricky (people often don't want to say bad things about their peers), but generally people already know. I know what I'm looking for in my performance reviews, and so I know what I'm looking for in my candidates. A competent coding baseline, and skills in the non-coding areas (knowing what not to build, anticipating edge cases, ability to distill business logic, ability to explain complex things clearly...). But this list is quite different from the "standard" tech interview loop: algorithms after algorithms with a sprinkling of "design a URL shortener."
- spitfire 8y ago
  I've posted this a few times before (taken shamelessly from tokenadult), but I'll post it again: There are three main criteria for hiring (in knowledge work environments): 1 General mental ability (Are they generally smart) 2 Work sample test (of real, representative work, no hazing!) 3 Integrity (Because 1 and 2 don't matter if you hire a psychopath or narcissist(cough) ). For 1 and 3 there are well known tests and you can find psychologists to administer them. Apply these tests evenly to everyone, even the people you're sure you want. The other approach you can take is the IBM/60's corporate approach. Which is apply the above approach to the "office girls" EI: general population and train the smart ones for the job. This can work surprisingly well if you have good people in leadership. http://emilkirkegaard.dk/en/wp-content/uploads/Schmidt-and-Hunter-1998-Validity-and-Utility-Psychological-Bulletin.pdf http://emilkirkegaard.dk/en/wp-content/uploads/Schmidt-and-H...
seattle_spring 8y ago
Want to know a great strategy for hiring lemons? Ask only Leetcode-style questions. Don't ask anything relevant to the job. Don't ask about past experiences, architecture, tech-specific questions, or anything like that. Only. Ask. Leetcode. Questions. If it's not a stupid algorithm disguised as something cute like a dog jumping over a river, then don't ask it. You will be in the exact situation this article describes: Passing up on good engineers, and not being able to tell if the people you do hire can actually perform on the job.
moistoreos 8y ago
Going to have to say that it also depends on the personality as well. The case for my company is that we base hiring on two facets of the interviewee: technical knowledge related to the stack(s) we use and their personality. The question we ask internally is: can they do the job but also fit into our culture? Culture is enormously important to my company. Without getting into the details we had personality conflicts on a fairly small team and this led to unprofessional bouts. He was a great developer, a fantastic developer. But he was constantly negative and unhappy for months which finally led to him seeking other opportunities. Which put a "great" developer on the market. The TL;DR is, this model of seeking the "great" developers and successfully keeping great developers goes both ways. Sure, it's great to have an awesome developer on your team.... as long as they're not a dickass.
- jt2190 8y ago
  +1 Whether your personality negatively/positively impacts your effectiveness does matter. If people were robots or Spock or pure math then this wouldn't be an issue. I will say that the reverse is true as well. The culture of a team can either increase or decrease the effectiveness of a developer.
- p0nce 8y ago
  Your culture is so great that someone that has left your team is called a "dickass" on the Internet.
deviationblue 8y ago
Everyone wants a Ferrari, but most work is ordinary enough that all they really need is a Kia. It's very unlikely that you're the next Google etc., better to hire according to your needs and stature. Maybe you know what you're doing, and if you need a Ferrari to get it done, get a Ferrari. But let's be honest, most jobs out there are "sell this to people, faster". How much intellect are we supposed to spend on that? I digress. All you really need for that shit is a Kia. Also, it's really hard to hide both competence and incompetence. I wouldn't worry about passing resume screens or coding interviews if you can actually make something. If you have the skills to make things, you can even start your own business. Most people are average skilled, though. To be honest, that's all you really need. Let's not pretend otherwise.
- deleted 8y ago
  [deleted]
- perl4ever 8y ago
  I was really enthusiastic about a recent interview, because the process started with a take home coding assignment that I thought I did well on and got me invited for a 4+ hour interview. But then the in person interview session ended with asking me to write a simple program in an unfamiliar IDE and language while they talked at me - they called it "pair programming". I am utterly unable to work anything out if I can't have intermittent peace and quiet. So, the verdict was accurate (in a tautological sort of way) that I wasn't a good fit, but clearly they have preconceptions that are preventing them from hiring useful people. One thing to keep in mind though, is that whenever you're dealing with a lot of similar trials of something, avoiding all the big mistakes is more important than getting all the big successes. I think this applies to hiring people, choosing investments, relationships, probably other things. So having irrational reasons for rejecting people is less harmful than having irrational reasons for accepting people. Which is probably an important reason why free markets don't automatically eliminate the prejudices we consider unacceptable.
  - stevenwoo 8y ago
    Did they tell you it was going to be pair programming or was it a surprise? I find things like this are better to practice beforehand or if one knows beforehand one would prefer not to do it, to just reject it before it gets that far along in the process.
rurabe 8y ago
I think it's possible for both of these to be true if you back off of the extreme positions. 1) Great developers are seldom looking for new jobs, but 2) There are a ton of inefficiencies in hiring that making finding great developers possible if you think outside the box.
Tehdasi 8y ago
> needing a weird combination of skills, can be solved by hiring people with half or a third of the expertise you need and training people. I'll add that companies don't mention their internal tools/libraries/current code base (which is most likely of pretty crap quality and poorly documented) which is much more a hurdle than whatever language/toolset than they put in a job ad as required experience.
- FLUX-YOU 8y ago
  This is a frustrating part. Even if you ask about it in an interview (there's almost always internal tools/libs/code), there's not really any compensation if they lie about their code quality and then put you on that stuff full-time. Rewriting that stuff is not great for a career. It's always been the stuff I've chosen to leave out because I don't want to signal to anyone that I'm willing to full-time rewrite your stuff.
  - jaggederest 8y ago
    As one of my better former managers put it, "Nobody gets a promotion for reducing the size of a tire fire by 50%"
jamestimmins 8y ago
Does anyone know what he's referring to when he talks about Matasano's unique hiring strategy?
- rahimnathwani 8y ago
  Search matasano crypto challenges
- antoncohen 8y ago
  https://sockpuppet.org/blog/2015/03/06/the-hiring-post/ https://sockpuppet.org/blog/2015/03/06/the-hiring-post/
dvt 8y ago
I hold Joel Spolsky in very high regard, but I just think he's fundamentally wrong here. I've seen (first hand) many times when great developers stagnate at various companies and they leave. This can happen for a few reasons, but we're deluding ourselves if we think every single job is "solving the big one" -- no, sometimes we need people to write the plumbing, the boring stuff, the unit tests, and so on. Great developers, more often than not, get bored. If they have any sort of intellectual curiosity, they will leave. In most cases, corporate programming is just not very interesting. So first, we have the curious geniuses. I know many of these people that hop around from job to job and they say "meh Google sucked" or "Facebook was boring so I'm at this new startup now." Second, we have the people like me. I hope my hubris isn't showing, but I think I'm a pretty decent developer. I wrote a few books, had some contributions to a few big-boy OSS projects, am in the top 1.5% of Stack Overflow (even though I haven't really been contributing for like 4 years now), etc. I also built things -- many, many things -- some have a few hundred stars on Github, many have none. My goal in life is to "do my own thing" and, eventually, get that FU money. That is never going to happen working a regular 9-5 job and, in my early 30s now, I've completely accepted that. So I work for a few years (2-3) and then take a sabbatical where I dedicate my entire time on a project or two, trying to get it off the ground. When it inevitably fails, I go back to a "regular" job. The gap, contrary to popular belief, doesn't hurt you. When you explain and show what you built, people are more likely to be blown away rather than derisive. Finally, we have people like Max Howell who famously got rejected by Google even though literally everyone at Google uses his software. Max Howell is most definitely a great developer, and yet Google tripped all over their own corporate shoelaces. Let me put it this way: if I did a startup, I would 100% want Max Howell on my founding team. I mean just take a look at not only the code quality, but also how PRs/reviews are handled in homebrew.
- monocasa 8y ago
  WRT to the Max Howell thing, if you can't swap the left and right branches of a tree, I have a hard time believing that you're a great developer.
  - tabeth 8y ago
    What do you believe is so special about swapping branches of a tree? Couldn't you just as easily say "if you can't make and maintain something more than a few thousand people use reliably, then you're not a great developer?"
naveen99 8y ago
That’s why the best time to hire is out of grad school, college or high school.
tabeth 8y ago
I've been thinking about this for a while and have an interesting idea -- feedback welcome! Imagine a fully open company -- code, tools, library, marketing data, strategy, etc. Literally anything about the company you can think of is accessible on the open and ready to interact with, except for the underlying database data. Following along? Cool. Now, imagine all of this stuff being on some open repository (not necessarily Git, but for the sake of example let's suppose so). As employees create their backlog of items they add four properties: - A 1-100 score on how much difficulty they would have completing the item, 100 being extreme, 1 being none - A 1-100 score on how much difficulty they think an entry employee would have with the item. An entry employee would be defined clearly as the lowest level for their ladder, e.g. Software Developer I. - A time estimate on how much time they would have completing the item. - A time estimate they estimate on how long they think an entry level employee would have. So, knowing on this. Would the perfect hiring process simply not be taking the items, picking things that should take a reasonable amount of time (less than 8 hours) that are in the highest reported entry difficulty possible and then scaling that by the average self difficulty? With this system you could even anonymize the dummy interview pull requests and have employees rate the quality of it. Differences in quality and difficulty could be a positive signal (e.g. high quality for low difficulty item is good, obviously, but high quality for low difficulty item that was estimated to take less time than the average estimate, by employees, is even better). --- So the purpose of this is less to identify who is objectively great, but rather to figure out how good the candidate is relative to people in your own organization. Personally I think finding an objective measure is an intractable problem.
- BWStearns 8y ago
  Domain knowledge is the uncontrollable here. A junior is still probably a junior 3 months in but they are a lot faster for their domain knowledge (codebase, business rules etc). Day 1 (I.e. Your proposed interview) there would be little observable difference between various experience levels/quality of engineer since the overhead of domain knowledge will dwarf anything they already have in their head. That is, unless the problem you pick is so isolated that there is no domain knowledge required. In that case you have expensively reproduced fizzbuzz without the mass memorization of code golf style answers.
itsdrewmiller 8y ago
This is just not persuasive - here's a key point from it: "If we put aside Joel's argument and look at the job market, there's incomplete information, but both current and prospective employers have incomplete information, and whose information is better varies widely. It's actually quite common for prospective employers to have better information than current employers!" Ok, sure, maybe there are employers who don't know what they've got. But just based on the raw amount of information on either side, all else being equal current employers should have a much better idea of actual performance.
deleted 8y ago
[deleted]
johngalt 8y ago
Dead on. There is also an environmental component in play. You'll find a lot more noise in the hiring pool when the tech field is booming.
chrisbennet 8y ago
"Joel's claim is basically that "great" developers won't have that many jobs compared to "bad" developers because companies will try to keep "great" developers. " This is a not what Joel said. Joel said that great developers are not on the market very often. They don't need to go on the open market to find their next job.
mattsfrey 8y ago
The problem with this is that so many companies (especially startups with young ceos) have _no idea_ how valuable an employee is. They negotiate hard and underpay often because that's just a general prescription in business. From my personal experience "great" developers change jobs all the time.
hpcjoe 8y ago
What I've tried to do over my career has been to hire people who showed ability to learn and grow, and then offer them the ability to step up to challenges. What I've gotten has been a rather bad mixture of incompetence, cluelessness, etc. They were eager to pass the interviews and get the job. They were not eager to do the work under the timelines we needed. This isn't just devs. This was sales, ops, and finance. I don't believe in any hard and fast rules about good vs non-good. I do believe that past performance is no indicator of future ability. That at least has been my experience. Even more relevant, pedigree is of no value in estimating potential success or failure of an employee at their mission. The best dev I'd ever worked with had a GED (high school equivalency diploma), and never went to college. The worst had a top-of-heap pedigree. Hiring is hard. You have to be an optimist. In the case of a small company, a bad hire can be a fatal mistake.
- PostOnce 8y ago
  If everyone you hire in every department is "not eager to do the work under the timelines you need", then perhaps your timelines are not realistic and you either need to adjust them, hire more staff, or both? A commander can blame his soldiers for losing him the war, but is it more their fault or his?
- shanghaiaway 8y ago
  What you have is clearly a management problem - you are the problem.
nstj 8y ago
The article paraphrases Spolsky incorrectly: great developers don’t look for jobs more than 4x in their career not because they stay in their jobs longer (ie: career length/4) but because they are headhunted for most of their jobs so don’t have to look themselves
PaulHoule 8y ago
The "Bob" scenario happens all too often. Sometimes the "slow and sure" developer gets a lot of grief from pathological management that doesn't realize that developers who seem faster as short tasks might go in circles for years on bigger projects.
mixmastamyk 8y ago
I do freelancing. Three times in the last few months I’ve had to sit in front of coderpad to attack a problem I’m not familiar with and come up with a solution—with one or two folks watching, the clock ticking, my family’s livelihood on the line, next to ~two hundred grand in a suitcase. I write well tested code every day but can’t perform under these conditions. It’s basically the scene from Swordfish, and just as ridiculous. Two of the three groups seemed to realize it’s a poor test, didn’t have high expectations but went through with it anyway. One seemed to think I was incompetent because I didn’t have his trick memorized about keyword args making closures possible in a loop in Python. I don’t write clever code on purpose, and certainly not enough to have it memorized to use under the gun. The 'shortage' is a myth and I blame time-wasting hiring practices such as these. Am thankful a smart guy like Luu is highlighting the bullshit.
rdlecler1 8y ago
This is sort of like the venture capital market. Entrepreneurs will sort rank their investor list based on perceived value of that investor and will work their way down. Most VCs simply never see the great companies, which is why top firms consistently outperform while the overall asset class has fared poorly. In short, deal flow is everything.
taurath 8y ago
I think there's a really key point with nontraditional developers. I'm also terrible at interviews because of both anxiety and lack of formal education (I can usually get optimal answers, but come across as "not confident" in them), but consistently get really high ratings and reviews from wherever I work.
thoughtexprmnt 8y ago
Most teams should not waste their time searching for a “great” developer (whatever that is, exactly). Instead they would be better off just hiring the first candidate that comes along in whom they have reasonable level of confidence will soon be a productive contributor to the team's goals and makeup, and then move on with business. Everyone involved in the recruiting, screening, and interview process should not lose sight of this as the objective, and resist any temptation to get sidetracked by silliness like “pedigree” and “trendiness”.

128 points by ff_ 8y ago

When the topic of "great" developers comes up, I always challenge anyone to define precisely, in quantifiable terms, what a "great developer" is exactly. How do you know someone is a "great developer" if you can't, in tangible terms, say exactly what a "great developer" is? This is even more of an issue when recruiting because recruiting is an assessment process to determine if someone meets some requirement. But just like software testing, if you can't quantify the requirement, then you can't quantify a test to determine if someone meets that requirement, and therefore the interview/assessment process is simply voodoo recruiting. Voodoo recruiting is when you have the appearance of scientific process in recruiting. - a process, - a set of questions that appear good, - a coding test that seems technical enough that it must be telling you something meaningful about the candidate, - the opinions of many people on the candidate. But in fact all of this is voodoo because you still can't directly define the requirement and thus you never known if you ever found what you are looking for, which is not actually known. The final decision in recruiting a "great developer", as always, comes down to some subjective opinion based on a range of criteria that can't be proven to have had any meaning in the first place.

quotemstr 8y ago
Was Heaviside a great engineer? Greatness does exist. I'm really very sick of the recent trend toward denying the basic idea of competence.
- ItsMe000001 8y ago
  You create a strawman. Nobody denies what you claim. The issue is not at all the existence of "great people" (for anything). The problem is how do you find them. You have a thousand people in front of you. Now please make a prediction that actually works reliably for big numbers of people. Your argument suffers from the same problem as any of the others that always come up after something happened, "why didn't they...". You look backwards. You make a "prediction"about something in the past and of course you get 100% accuracy and think "that's easy". Okay, now please make tat prediction for the future. You have people in front of you whom you know zilch about. Now make - en mass - statements about their future work that when looking back from the future will still look good. About your > Was Heaviside a great engineer? Now please make that backward-looking prediction about his greatness for the same person before it turned out that he was good. And not just for one person either, do it for thousands of not-famous people with some degree of consistency. And then report about all your predictions, not just the few that worked, and don't forget to mention if you did some extreme filtering beforehand in order to ensure you get a lot of "hits", leaving a large part of potential candidates without any chance to begin with. That kind of thing works for some (like so many "recipes for success") but not for society.
  - sokoloff 8y ago
    > You have a thousand people in front of you. Now please make a prediction that actually works reliably for big numbers of people. The simplest answer is to predict that 0 of the 1000 are great developers. That will be reliable for most cohorts of 1000 people and most active industry applicants for developer positions. You can only reliably find greatness by filtering through many college applicants, training and mentoring them, keeping, motivating, and incenting the best, or by using networking and references to find the standout performers. It’s so rare for these industry experienced people to need to apply for a job cold that that is a tremendously adverse pool from which to select in my experience If greatness is the goal. If good enough is the goal, there are plenty more of those devs available.
    - pkaye 8y ago
      So your plan would be to hire all the college applicants that applied (or a random sample) and then mentor and train them to see who works out the best?
      - sokoloff 8y ago
        No, you hire the ones who seem interested, skilled “enough”, and intelligent (possibly in reverse order of importance). You can very safely make a cut against some very likely poor candidates. But yeah, you have a light filter and then let them grow and evolve. You can’t possibly learn in 4 hours anywhere near what you can in 2 years. In college recruiting, you’re looking for “could be solid” and you get the raw material for that a lot. Some happen to turn out great. Finding greatness in industry is driven mostly by asking some of your best employees to search their networks for their best ex-colleagues and otherwise by dumb luck (someone tried a startup that went bust and no one in their network had an opening, someone is moving to a new city with no network, etc)
        pkaye 8y ago
        Those three traits are exactly what I use for college recruiting. I guess some people look at great developers on an absolute scale but I consider it on a relative scale within the role under consideration. I consider those three traits make a junior candidate great. Actually those same traits are important for all roles. It is just that for junior roles I don't expect much beyond that.
dfabulich 8y ago
I strongly agree. This problem applies both to the interview process and to evaluating the performance of engineers after hiring them. A lot of companies try to use supposedly "data-oriented" approaches to hiring. They assume that a "successful" interview process hires candidates who get good performance reviews a year later. But those reviews are just as subjective as the hiring process. If your metric for "success" is whether you still like someone a year from now, you might as well just hire whomever you like today!
- andrewstuart 8y ago
  Yes and no. I still think you can have an effective recruiting process which attempts to identify someone's attributes across a range of criteria - I attempted to identify some of those characteristics here: http://www.supercoders.com.au/blog/50characteristicsofagreatsoftwaredeveloper.shtml http://www.supercoders.com.au/blog/50characteristicsofagreat... You would fail entirely in your recruiting effort if you just hired "whomever you like today" - you must make some attempt to measure a persons capabilities. What I am attempting to dispel is the idea that it is possible to measure someone as a "great developer" because it is a matter of opinion - person A may think someone is a "great developer", person B may not. I also think that pretty much every recruiting process at every company is "Voodoo Recruiting", or "Recruiting Theatre" - the appearance of rigor and scientific approach, but really just a show that allows the assessor to feel like some scientific, justifiable process has been executed that has resulted in a "correct" hiring "go/no go" decision, but in fact does nothing more than make people feel comfortable with making a subjective decision in the end. Whether or not someone is a great developer also changes according to time and context. What I mean by this is that some developer X who got amazing results in some job that they had, might be judged as being a shit developer when put into a job where they weren't inspired by the project, were working with unfamiliar technologies, where they are in a life phase dealing with depression, or where their colleagues or boss treated them without respect. Suddenly "great developer" in some previous context is now seen as a shit developer in some other context... again, it is simply not possible to evaluate a given person in a job interview and determine if they are "a great developer".
  - 8y ago
existencebox 8y ago
I'll chime in with the sister posts expressing vehement disagreement. I can tell you what "great" means to me, in 2(.5) very clear words. "You (fucking) Ship." To me a great dev has a track record of delivering the end result. A greater dev delivers the result faster, with less debt attached. One can fairly say that this is still subjective, but my criteria are _quite_ meaningful and even somewhat empirical/measurable, although it's admittedly hard to do in a short interview format. I certainly know a "Great dev" when I've met/worked with them, because they ship faster than I do, with less friction or need to revisit post-factum. Edit: Since half the responses seem to elide a very key part of my statement, let me echo: _debt matters_. Having someone who ships but burns your team down, not good. Someone who ships but leaves you with years of bugs or un-maintainable code, not good. As I'm sure we all know, you get optimizations against KPIS you measure, and if you don't measure "delivery" holistically, you'll absolutely get people who excel at one component while salting the fields.
- grantwu 8y ago
  Oh I guess we don't need maintenance then
  - existencebox 8y ago
    I believe you're putting words in my mouth. A great dev can even be great within fixing problems/performing ops/maintenance, by my same qualification. _They ship fixes._ Perfection has very little to do with greatness. Aspiring for it might, but that's a second discussion about unrealistic goals and setting oneself up for disappointment.
    - Analemma_ 8y ago
      > I believe you're putting words in my mouth. You're the one who said you could describe "great" in two words that didn't include maintenance or quality. If it turns out those two words aren't enough, or need lots of asterisks and clarifying words, that's on you.
      - 8y ago
smallnamespace 8y ago
> The final decision in recruiting a "great developer", as always, comes down to some subjective opinion based on a range of criteria that can't be proven to have had any meaning in the first place. Careful, this road can lead to nihilism. Subjectivity is an important part of reality. Subjective things are also messy. We can acknowledge that these aspects are subjective and wrestle with them the best we can, or we can deny they exist at all, and end up pushing those problems somewhere else, where they might fester and become worse. > I always challenge anyone to define precisely, in quantifiable terms, what a "great developer" is exactly. Replace "great developer" with words like "democracy", "truth", "love", "good music", "healthy diet", "pornography". We can't reliably quantify or strictly define any of those things. Sometimes "I know it when I see it" and sometimes not, when cognitive biases and instincts lead us astray. But to deny there is such a thing as a good developer, or a "great" developer, because we don't have perfect metrics, seems like going too far. I agree that having a "pseudoscientific" process can be bad. When you are dealing with something that is inherently subjective, you should resist both urges: one is false precision, a system that claims to cleanly settle once and for all what is inherently messy and difficult; and the other is the denial that that 'something' exists at all.
- threatofrain 8y ago
  I feel like empiricists routinely and easily perform the partially opinionated act of operationalization, and what's being critiqued here is the <lack> of opinion in operationalization, except for the mocking of scientific appearance. Likewise, many discussions on consciousness are worthy of critique for lacking operationalization. We will always question whether the operationalization is a good proxy, but there's a difference when firms lack even the opinion to do so.
- ci5er 8y ago
  You're just saying that abstract nouns have no explicit referent like a brick or apple. I take exception to democracy as a noun or attribute like 'truth', however. Truth is an abstract attribute. Sometimes a noun. Democracy is a process: two wolves and a sheep voting on what's up for dinner.
nostrademons 8y ago
This is the case for basically all superlatives that apply to human emotions, though. What's a "great company to work for"? It's the company you want to keep working for. That's not going to be the same for every person - I have had great experiences at companies that are commonly known as "great companies to work for", but know coworkers that hated them. What's a "great investment"? It's something you can sell for a lot more than you paid for it. Why can you sell it for more than you paid for it? Because the person you're selling it to also believes it's a "great investment", and so you have a circular definition. What does it mean to "Make America Great Again"? Half the country believes we're doing it right now, and the other half believes we're in a bizarro throwback to the 1950s that isn't going to end well for anyone. So what does it mean to recruit a "great developer"? Well, it's when you hire the person that you wanted to hire. Yes, it's subjective - it's always subjective. That doesn't mean it has no meaning, though, because people ascribe meaning to subjective qualities all the time.
- sheepmullet 8y ago
  > This is the case for basically all superlatives that apply to human emotions, though. Exactly! What makes a great wife? Or a great friend?
- joe_the_user 8y ago
  The distinction I'd elucidate from the gp's is that "great" is a problem comes because it's both a superlative and because it can't be quantified. You can ask for a developer who produce a large amount of average code. You can ask for a developer that produces very few terrible errors. You can ask for a develop who produces decent code and happens know domain X very well. You can ask for a developer who can mentor younger developers. You can ask for a developer who might needs virtually no supervision to succeed. And so-forth. They could all be different people. Perhaps you can sift until you somehow find someone who's all that but they'd be fool to join you because you're looking for perfection rather than looking for fulfill your actual needs. Superlatives are a terrible way to make any kind of serious decision. Salespeople love specifically because they tend to be unmeasurable and irrelevant when they are measurable. Thus the more superlatives are entering a decision process, the more you can tell you're sold shine rather than substance.
dsacco 8y ago
Are you saying there's no such thing as developers significantly better than the norm, or rather that it's infeasible to identify them? I don't think your point about precisely quantifying the definition of a "great developer" is a good one. I can capture an observation of superior capability in several dimensions without having to rigorously quantify the relative differences in capability or precisely why one is more capable than the other. If one developer accomplishes in a few days what takes another two weeks with code that is at least as maintainable and performant, that developer is better. If that developer is better when put next to most of the other developers you have around you, then they're a "great developer." Your focus on quantifiable definition seems to imply that significant differences in human capital can only be ascertained mathematically, but that doesn't reflect quite a lot that we're already familiar with in everyday life. If I'm placed in front of two walls which are both much taller than I am, I can see which of the two is taller than the other if the difference is significant as long as the tops aren't out of sight. I don't need to quantify this; I could just say, "One looks to be a lot taller than the other, but I'm not sure by how much exactly, or what their respective heights are." I'll happily agree that a lot of tech interviewing (and interviewing in general) is dreadfully broken. I'll also happily agree that figuring out which skills to prioritize and how to judge the level of those skills in candidates is very difficult work. But I strongly disagree that the fundamental premise of that work is intrinsically "voodoo." If your bar is a mathematical formalization then of course you're going to be disappointed, but that also holds true for the majority of our professional and personal activity.
- andrewstuart 8y ago
  In a sentence, I am saying that pretty much every recruiting process at every company, when it assesses developers, results in a meaningless decision, despite the universal certainty at every company that their recruiting process absolutely definitely results in making accurate decisions about people. Did the interview process find a great developer? Maybe. Did the recruiting process reject great developers? How would you ever know? I'm pretty much certain that most companies reject many more developers who would be really productive/effective, than they choose to hire.
  - 8y ago
SmellTheGlove 8y ago
Not only that, but is anyone really "great" across multiple/all domains? I've worked across a pretty diverse set of businesses and functions in management, and I can say pretty confidently that "greatness" is very dependent upon fit for the problem space, and I do mean that to work both ways: Fit is incredibly dependent upon the individual giving a shit about that problem space. No doubt someone could be very good working on something they don't care that much about, but we're talking great here, right? That means talking/writing in broad strokes about great developers approaches impossible. I also don't think you need "great" developers in most roles. Sometimes average is exactly what you need, and hiring above that just leads to having a great person leave because they're not particularly challenged or enthusiastic about the role. And to wrap all of that up, I don't even know that I can objectively define what "great" really means anyway, so maybe I have no place in this discussion :)
- perl4ever 8y ago
  >Fit is incredibly dependent upon the individual giving a shit about that problem space. Amen. That should be obvious, but I don't think I've ever had any recruiter or phone screen where they acknowledged that in any way. I responded to a recruiter recently by asking what the company actually did, and they reacted like I was just hamfistedly demanding the company name or prematurely trying to research for the interview, but I really just wanted to know - what would I actually be contributing to the world in this job, never mind the technical challenges, the buzzwords, and the paycheck. It's like everyone is assumed to be at best apathetic to the larger context of their job, apart from delusions of grandeur about "changing the world".
SatvikBeri 8y ago
I often tell people that before they think about their hiring process, they should think about their performance reviews. You're doing those with multiple orders of magnitude more data, yet most managers will say their performance reviews have a lot of uncertainty. I spent a lot of time at my last job developing ways to measure performance across my team. One big surprise was that I had to significantly change the way we assigned work in order to make it at all measurable. Rather than giving people well-defined tasks, I switched to assigning projects that had some amount of (expected) business value. This meant the projects had to include things like deployment, talking to marketing, and whatever needed to happen to make it useful. Generally each one was 3-6 weeks of work. This provided a good first-order measure of effectiveness. Importantly, it helped people change their habits to contribute more business value. I found massive variation in the junior engineers, easily with 5-10x differences between the most and least effective. The differences in senior engineers were smaller, but still ~2x. Of course, there are other major factors you have to take into account, like code quality and mentoring. I developed similarly detailed systems for measuring those. Once again, you have to change the way you do things if you want to make them measurable, often accepting local inefficiencies for global insight. With all that data in hand, I correlated that to our hiring ratings. The results? Our two best engineers were both borderline in the hiring process, and would have been rejected except for a strong vote of confidence by our senior engineer, who, when we checked, turned out to have a basically perfect track record on hiring judgments-much better than anyone else in the company, me included.
- majormajor 8y ago
  IME it's hard to precisely rank average devs who are in the middle. There's a lot of clumping. It's a lot less hard to identify overperformers or extreme underperformers. You mention the big possible range, but I've never had to dive that deeply into data to spot them - I've used the data to check my instincts, but have generally been right. And if you don't trust your instincts, just ask your team! They know whose code reviews they dread, who never offers helpful suggestions, who seems to do just the minimum. It's sometimes tricky (people often don't want to say bad things about their peers), but generally people already know. I know what I'm looking for in my performance reviews, and so I know what I'm looking for in my candidates. A competent coding baseline, and skills in the non-coding areas (knowing what not to build, anticipating edge cases, ability to distill business logic, ability to explain complex things clearly...). But this list is quite different from the "standard" tech interview loop: algorithms after algorithms with a sprinkling of "design a URL shortener."
spitfire 8y ago
I've posted this a few times before (taken shamelessly from tokenadult), but I'll post it again: There are three main criteria for hiring (in knowledge work environments): 1 General mental ability (Are they generally smart) 2 Work sample test (of real, representative work, no hazing!) 3 Integrity (Because 1 and 2 don't matter if you hire a psychopath or narcissist(cough) ). For 1 and 3 there are well known tests and you can find psychologists to administer them. Apply these tests evenly to everyone, even the people you're sure you want. The other approach you can take is the IBM/60's corporate approach. Which is apply the above approach to the "office girls" EI: general population and train the smart ones for the job. This can work surprisingly well if you have good people in leadership. http://emilkirkegaard.dk/en/wp-content/uploads/Schmidt-and-Hunter-1998-Validity-and-Utility-Psychological-Bulletin.pdf http://emilkirkegaard.dk/en/wp-content/uploads/Schmidt-and-H...

seattle_spring 8y ago

Want to know a great strategy for hiring lemons? Ask only Leetcode-style questions. Don't ask anything relevant to the job. Don't ask about past experiences, architecture, tech-specific questions, or anything like that. Only. Ask. Leetcode. Questions. If it's not a stupid algorithm disguised as something cute like a dog jumping over a river, then don't ask it. You will be in the exact situation this article describes: Passing up on good engineers, and not being able to tell if the people you do hire can actually perform on the job.

moistoreos 8y ago

Going to have to say that it also depends on the personality as well. The case for my company is that we base hiring on two facets of the interviewee: technical knowledge related to the stack(s) we use and their personality. The question we ask internally is: can they do the job but also fit into our culture? Culture is enormously important to my company. Without getting into the details we had personality conflicts on a fairly small team and this led to unprofessional bouts. He was a great developer, a fantastic developer. But he was constantly negative and unhappy for months which finally led to him seeking other opportunities. Which put a "great" developer on the market. The TL;DR is, this model of seeking the "great" developers and successfully keeping great developers goes both ways. Sure, it's great to have an awesome developer on your team.... as long as they're not a dickass.

jt2190 8y ago
+1 Whether your personality negatively/positively impacts your effectiveness does matter. If people were robots or Spock or pure math then this wouldn't be an issue. I will say that the reverse is true as well. The culture of a team can either increase or decrease the effectiveness of a developer.
p0nce 8y ago
Your culture is so great that someone that has left your team is called a "dickass" on the Internet.

deviationblue 8y ago

Everyone wants a Ferrari, but most work is ordinary enough that all they really need is a Kia. It's very unlikely that you're the next Google etc., better to hire according to your needs and stature. Maybe you know what you're doing, and if you need a Ferrari to get it done, get a Ferrari. But let's be honest, most jobs out there are "sell this to people, faster". How much intellect are we supposed to spend on that? I digress. All you really need for that shit is a Kia. Also, it's really hard to hide both competence and incompetence. I wouldn't worry about passing resume screens or coding interviews if you can actually make something. If you have the skills to make things, you can even start your own business. Most people are average skilled, though. To be honest, that's all you really need. Let's not pretend otherwise.

deleted 8y ago
[deleted]
perl4ever 8y ago
I was really enthusiastic about a recent interview, because the process started with a take home coding assignment that I thought I did well on and got me invited for a 4+ hour interview. But then the in person interview session ended with asking me to write a simple program in an unfamiliar IDE and language while they talked at me - they called it "pair programming". I am utterly unable to work anything out if I can't have intermittent peace and quiet. So, the verdict was accurate (in a tautological sort of way) that I wasn't a good fit, but clearly they have preconceptions that are preventing them from hiring useful people. One thing to keep in mind though, is that whenever you're dealing with a lot of similar trials of something, avoiding all the big mistakes is more important than getting all the big successes. I think this applies to hiring people, choosing investments, relationships, probably other things. So having irrational reasons for rejecting people is less harmful than having irrational reasons for accepting people. Which is probably an important reason why free markets don't automatically eliminate the prejudices we consider unacceptable.
- stevenwoo 8y ago
  Did they tell you it was going to be pair programming or was it a surprise? I find things like this are better to practice beforehand or if one knows beforehand one would prefer not to do it, to just reject it before it gets that far along in the process.

rurabe 8y ago

I think it's possible for both of these to be true if you back off of the extreme positions. 1) Great developers are seldom looking for new jobs, but 2) There are a ton of inefficiencies in hiring that making finding great developers possible if you think outside the box.

Tehdasi 8y ago

> needing a weird combination of skills, can be solved by hiring people with half or a third of the expertise you need and training people. I'll add that companies don't mention their internal tools/libraries/current code base (which is most likely of pretty crap quality and poorly documented) which is much more a hurdle than whatever language/toolset than they put in a job ad as required experience.

FLUX-YOU 8y ago
This is a frustrating part. Even if you ask about it in an interview (there's almost always internal tools/libs/code), there's not really any compensation if they lie about their code quality and then put you on that stuff full-time. Rewriting that stuff is not great for a career. It's always been the stuff I've chosen to leave out because I don't want to signal to anyone that I'm willing to full-time rewrite your stuff.
- jaggederest 8y ago
  As one of my better former managers put it, "Nobody gets a promotion for reducing the size of a tire fire by 50%"

jamestimmins 8y ago

Does anyone know what he's referring to when he talks about Matasano's unique hiring strategy?

rahimnathwani 8y ago
Search matasano crypto challenges
antoncohen 8y ago
https://sockpuppet.org/blog/2015/03/06/the-hiring-post/ https://sockpuppet.org/blog/2015/03/06/the-hiring-post/

dvt 8y ago

I hold Joel Spolsky in very high regard, but I just think he's fundamentally wrong here. I've seen (first hand) many times when great developers stagnate at various companies and they leave. This can happen for a few reasons, but we're deluding ourselves if we think every single job is "solving the big one" -- no, sometimes we need people to write the plumbing, the boring stuff, the unit tests, and so on. Great developers, more often than not, get bored. If they have any sort of intellectual curiosity, they will leave. In most cases, corporate programming is just not very interesting. So first, we have the curious geniuses. I know many of these people that hop around from job to job and they say "meh Google sucked" or "Facebook was boring so I'm at this new startup now." Second, we have the people like me. I hope my hubris isn't showing, but I think I'm a pretty decent developer. I wrote a few books, had some contributions to a few big-boy OSS projects, am in the top 1.5% of Stack Overflow (even though I haven't really been contributing for like 4 years now), etc. I also built things -- many, many things -- some have a few hundred stars on Github, many have none. My goal in life is to "do my own thing" and, eventually, get that FU money. That is never going to happen working a regular 9-5 job and, in my early 30s now, I've completely accepted that. So I work for a few years (2-3) and then take a sabbatical where I dedicate my entire time on a project or two, trying to get it off the ground. When it inevitably fails, I go back to a "regular" job. The gap, contrary to popular belief, doesn't hurt you. When you explain and show what you built, people are more likely to be blown away rather than derisive. Finally, we have people like Max Howell who famously got rejected by Google even though literally everyone at Google uses his software. Max Howell is most definitely a great developer, and yet Google tripped all over their own corporate shoelaces. Let me put it this way: if I did a startup, I would 100% want Max Howell on my founding team. I mean just take a look at not only the code quality, but also how PRs/reviews are handled in homebrew.

monocasa 8y ago
WRT to the Max Howell thing, if you can't swap the left and right branches of a tree, I have a hard time believing that you're a great developer.
- tabeth 8y ago
  What do you believe is so special about swapping branches of a tree? Couldn't you just as easily say "if you can't make and maintain something more than a few thousand people use reliably, then you're not a great developer?"

naveen99 8y ago

That’s why the best time to hire is out of grad school, college or high school.

tabeth 8y ago

I've been thinking about this for a while and have an interesting idea -- feedback welcome! Imagine a fully open company -- code, tools, library, marketing data, strategy, etc. Literally anything about the company you can think of is accessible on the open and ready to interact with, except for the underlying database data. Following along? Cool. Now, imagine all of this stuff being on some open repository (not necessarily Git, but for the sake of example let's suppose so). As employees create their backlog of items they add four properties: - A 1-100 score on how much difficulty they would have completing the item, 100 being extreme, 1 being none - A 1-100 score on how much difficulty they think an entry employee would have with the item. An entry employee would be defined clearly as the lowest level for their ladder, e.g. Software Developer I. - A time estimate on how much time they would have completing the item. - A time estimate they estimate on how long they think an entry level employee would have. So, knowing on this. Would the perfect hiring process simply not be taking the items, picking things that should take a reasonable amount of time (less than 8 hours) that are in the highest reported entry difficulty possible and then scaling that by the average self difficulty? With this system you could even anonymize the dummy interview pull requests and have employees rate the quality of it. Differences in quality and difficulty could be a positive signal (e.g. high quality for low difficulty item is good, obviously, but high quality for low difficulty item that was estimated to take less time than the average estimate, by employees, is even better). --- So the purpose of this is less to identify who is objectively great, but rather to figure out how good the candidate is relative to people in your own organization. Personally I think finding an objective measure is an intractable problem.

BWStearns 8y ago
Domain knowledge is the uncontrollable here. A junior is still probably a junior 3 months in but they are a lot faster for their domain knowledge (codebase, business rules etc). Day 1 (I.e. Your proposed interview) there would be little observable difference between various experience levels/quality of engineer since the overhead of domain knowledge will dwarf anything they already have in their head. That is, unless the problem you pick is so isolated that there is no domain knowledge required. In that case you have expensively reproduced fizzbuzz without the mass memorization of code golf style answers.

itsdrewmiller 8y ago

This is just not persuasive - here's a key point from it: "If we put aside Joel's argument and look at the job market, there's incomplete information, but both current and prospective employers have incomplete information, and whose information is better varies widely. It's actually quite common for prospective employers to have better information than current employers!" Ok, sure, maybe there are employers who don't know what they've got. But just based on the raw amount of information on either side, all else being equal current employers should have a much better idea of actual performance.

deleted 8y ago

[deleted]

johngalt 8y ago

Dead on. There is also an environmental component in play. You'll find a lot more noise in the hiring pool when the tech field is booming.

chrisbennet 8y ago

"Joel's claim is basically that "great" developers won't have that many jobs compared to "bad" developers because companies will try to keep "great" developers. " This is a not what Joel said. Joel said that great developers are not on the market very often. They don't need to go on the open market to find their next job.

mattsfrey 8y ago

The problem with this is that so many companies (especially startups with young ceos) have _no idea_ how valuable an employee is. They negotiate hard and underpay often because that's just a general prescription in business. From my personal experience "great" developers change jobs all the time.

hpcjoe 8y ago

What I've tried to do over my career has been to hire people who showed ability to learn and grow, and then offer them the ability to step up to challenges. What I've gotten has been a rather bad mixture of incompetence, cluelessness, etc. They were eager to pass the interviews and get the job. They were not eager to do the work under the timelines we needed. This isn't just devs. This was sales, ops, and finance. I don't believe in any hard and fast rules about good vs non-good. I do believe that past performance is no indicator of future ability. That at least has been my experience. Even more relevant, pedigree is of no value in estimating potential success or failure of an employee at their mission. The best dev I'd ever worked with had a GED (high school equivalency diploma), and never went to college. The worst had a top-of-heap pedigree. Hiring is hard. You have to be an optimist. In the case of a small company, a bad hire can be a fatal mistake.

PostOnce 8y ago
If everyone you hire in every department is "not eager to do the work under the timelines you need", then perhaps your timelines are not realistic and you either need to adjust them, hire more staff, or both? A commander can blame his soldiers for losing him the war, but is it more their fault or his?
shanghaiaway 8y ago
What you have is clearly a management problem - you are the problem.

nstj 8y ago

The article paraphrases Spolsky incorrectly: great developers don’t look for jobs more than 4x in their career not because they stay in their jobs longer (ie: career length/4) but because they are headhunted for most of their jobs so don’t have to look themselves

PaulHoule 8y ago

The "Bob" scenario happens all too often. Sometimes the "slow and sure" developer gets a lot of grief from pathological management that doesn't realize that developers who seem faster as short tasks might go in circles for years on bigger projects.

mixmastamyk 8y ago

I do freelancing. Three times in the last few months I’ve had to sit in front of coderpad to attack a problem I’m not familiar with and come up with a solution—with one or two folks watching, the clock ticking, my family’s livelihood on the line, next to ~two hundred grand in a suitcase. I write well tested code every day but can’t perform under these conditions. It’s basically the scene from Swordfish, and just as ridiculous. Two of the three groups seemed to realize it’s a poor test, didn’t have high expectations but went through with it anyway. One seemed to think I was incompetent because I didn’t have his trick memorized about keyword args making closures possible in a loop in Python. I don’t write clever code on purpose, and certainly not enough to have it memorized to use under the gun. The 'shortage' is a myth and I blame time-wasting hiring practices such as these. Am thankful a smart guy like Luu is highlighting the bullshit.

rdlecler1 8y ago

This is sort of like the venture capital market. Entrepreneurs will sort rank their investor list based on perceived value of that investor and will work their way down. Most VCs simply never see the great companies, which is why top firms consistently outperform while the overall asset class has fared poorly. In short, deal flow is everything.

taurath 8y ago

I think there's a really key point with nontraditional developers. I'm also terrible at interviews because of both anxiety and lack of formal education (I can usually get optimal answers, but come across as "not confident" in them), but consistently get really high ratings and reviews from wherever I work.

thoughtexprmnt 8y ago

Most teams should not waste their time searching for a “great” developer (whatever that is, exactly). Instead they would be better off just hiring the first candidate that comes along in whom they have reasonable level of confidence will soon be a productive contributor to the team's goals and makeup, and then move on with business. Everyone involved in the recruiting, screening, and interview process should not lose sight of this as the objective, and resist any temptation to get sidetracked by silliness like “pedigree” and “trendiness”.