7 ms·
When the topic of "great" developers comes up, I always challenge anyone to define precisely, in quantifiable terms, what a "great developer" is exactly. How d
by andrewstuart 9y ago
When the topic of "great" developers comes up, I always challenge anyone to define precisely, in quantifiable terms, what a "great developer" is exactly.
How do you know someone is a "great developer" if you can't, in tangible terms, say exactly what a "great developer" is?
This is even more of an issue when recruiting because recruiting is an assessment process to determine if someone meets some requirement. But just like software testing, if you can't quantify the requirement, then you can't quantify a test to determine if someone meets that requirement, and therefore the interview/assessment process is simply voodoo recruiting.
Voodoo recruiting is when you have the appearance of scientific process in recruiting. - a process, - a set of questions that appear good, - a coding test that seems technical enough that it must be telling you something meaningful about the candidate, - the opinions of many people on the candidate. But in fact all of this is voodoo because you still can't directly define the requirement and thus you never known if you ever found what you are looking for, which is not actually known.
The final decision in recruiting a "great developer", as always, comes down to some subjective opinion based on a range of criteria that can't be proven to have had any meaning in the first place.
- quotemstr 9y agoWas Heaviside a great engineer? Greatness does exist. I'm really very sick of the recent trend toward denying the basic idea of competence.
- ItsMe000001 9y agoYou create a strawman. Nobody denies what you claim. The issue is not at all the existence of "great people" (for anything). The problem is how do you find them. You have a thousand people in front of you. Now please make a prediction that actually works reliably for big numbers of people. Your argument suffers from the same problem as any of the others that always come up after something happened, "why didn't they...". You look backwards. You make a "prediction"about something in the past and of course you get 100% accuracy and think "that's easy". Okay, now please make tat prediction for the future. You have people in front of you whom you know zilch about. Now make - en mass - statements about their future work that when looking back from the future will still look good. About your > Was Heaviside a great engineer? Now please make that backward-looking prediction about his greatness for the same person before it turned out that he was good. And not just for one person either, do it for thousands of not-famous people with some degree of consistency. And then report about all your predictions, not just the few that worked, and don't forget to mention if you did some extreme filtering beforehand in order to ensure you get a lot of "hits", leaving a large part of potential candidates without any chance to begin with. That kind of thing works for some (like so many "recipes for success") but not for society.
- sokoloff 9y ago> You have a thousand people in front of you. Now please make a prediction that actually works reliably for big numbers of people. The simplest answer is to predict that 0 of the 1000 are great developers. That will be reliable for most cohorts of 1000 people and most active industry applicants for developer positions. You can only reliably find greatness by filtering through many college applicants, training and mentoring them, keeping, motivating, and incenting the best, or by using networking and references to find the standout performers. It’s so rare for these industry experienced people to need to apply for a job cold that that is a tremendously adverse pool from which to select in my experience If greatness is the goal. If good enough is the goal, there are plenty more of those devs available.
- pkaye 9y agoSo your plan would be to hire all the college applicants that applied (or a random sample) and then mentor and train them to see who works out the best?
- sokoloff 9y agoNo, you hire the ones who seem interested, skilled “enough”, and intelligent (possibly in reverse order of importance). You can very safely make a cut against some very likely poor candidates. But yeah, you have a light filter and then let them grow and evolve. You can’t possibly learn in 4 hours anywhere near what you can in 2 years. In college recruiting, you’re looking for “could be solid” and you get the raw material for that a lot. Some happen to turn out great. Finding greatness in industry is driven mostly by asking some of your best employees to search their networks for their best ex-colleagues and otherwise by dumb luck (someone tried a startup that went bust and no one in their network had an opening, someone is moving to a new city with no network, etc)
- pkaye 9y agoThose three traits are exactly what I use for college recruiting. I guess some people look at great developers on an absolute scale but I consider it on a relative scale within the role under consideration. I consider those three traits make a junior candidate great. Actually those same traits are important for all roles. It is just that for junior roles I don't expect much beyond that.
- dfabulich 9y agoI strongly agree. This problem applies both to the interview process and to evaluating the performance of engineers after hiring them. A lot of companies try to use supposedly "data-oriented" approaches to hiring. They assume that a "successful" interview process hires candidates who get good performance reviews a year later. But those reviews are just as subjective as the hiring process. If your metric for "success" is whether you still like someone a year from now, you might as well just hire whomever you like today!
- andrewstuart 9y agoYes and no. I still think you can have an effective recruiting process which attempts to identify someone's attributes across a range of criteria - I attempted to identify some of those characteristics here: http://www.supercoders.com.au/blog/50characteristicsofagreatsoftwaredeveloper.shtml http://www.supercoders.com.au/blog/50characteristicsofagreat... You would fail entirely in your recruiting effort if you just hired "whomever you like today" - you must make some attempt to measure a persons capabilities. What I am attempting to dispel is the idea that it is possible to measure someone as a "great developer" because it is a matter of opinion - person A may think someone is a "great developer", person B may not. I also think that pretty much every recruiting process at every company is "Voodoo Recruiting", or "Recruiting Theatre" - the appearance of rigor and scientific approach, but really just a show that allows the assessor to feel like some scientific, justifiable process has been executed that has resulted in a "correct" hiring "go/no go" decision, but in fact does nothing more than make people feel comfortable with making a subjective decision in the end. Whether or not someone is a great developer also changes according to time and context. What I mean by this is that some developer X who got amazing results in some job that they had, might be judged as being a shit developer when put into a job where they weren't inspired by the project, were working with unfamiliar technologies, where they are in a life phase dealing with depression, or where their colleagues or boss treated them without respect. Suddenly "great developer" in some previous context is now seen as a shit developer in some other context... again, it is simply not possible to evaluate a given person in a job interview and determine if they are "a great developer".
- 9y ago
- existencebox 9y agoI'll chime in with the sister posts expressing vehement disagreement. I can tell you what "great" means to me, in 2(.5) very clear words. "You (fucking) Ship." To me a great dev has a track record of delivering the end result. A greater dev delivers the result faster, with less debt attached. One can fairly say that this is still subjective, but my criteria are _quite_ meaningful and even somewhat empirical/measurable, although it's admittedly hard to do in a short interview format. I certainly know a "Great dev" when I've met/worked with them, because they ship faster than I do, with less friction or need to revisit post-factum. Edit: Since half the responses seem to elide a very key part of my statement, let me echo: _debt matters_. Having someone who ships but burns your team down, not good. Someone who ships but leaves you with years of bugs or un-maintainable code, not good. As I'm sure we all know, you get optimizations against KPIS you measure, and if you don't measure "delivery" holistically, you'll absolutely get people who excel at one component while salting the fields.
- grantwu 9y agoOh I guess we don't need maintenance then
- existencebox 9y agoI believe you're putting words in my mouth. A great dev can even be great within fixing problems/performing ops/maintenance, by my same qualification. _They ship fixes._ Perfection has very little to do with greatness. Aspiring for it might, but that's a second discussion about unrealistic goals and setting oneself up for disappointment.
- Analemma_ 9y ago> I believe you're putting words in my mouth. You're the one who said you could describe "great" in two words that didn't include maintenance or quality. If it turns out those two words aren't enough, or need lots of asterisks and clarifying words, that's on you.
- 9y ago
- smallnamespace 9y ago> The final decision in recruiting a "great developer", as always, comes down to some subjective opinion based on a range of criteria that can't be proven to have had any meaning in the first place. Careful, this road can lead to nihilism. Subjectivity is an important part of reality. Subjective things are also messy. We can acknowledge that these aspects are subjective and wrestle with them the best we can, or we can deny they exist at all, and end up pushing those problems somewhere else, where they might fester and become worse. > I always challenge anyone to define precisely, in quantifiable terms, what a "great developer" is exactly. Replace "great developer" with words like "democracy", "truth", "love", "good music", "healthy diet", "pornography". We can't reliably quantify or strictly define any of those things. Sometimes "I know it when I see it" and sometimes not, when cognitive biases and instincts lead us astray. But to deny there is such a thing as a good developer, or a "great" developer, because we don't have perfect metrics, seems like going too far. I agree that having a "pseudoscientific" process can be bad. When you are dealing with something that is inherently subjective, you should resist both urges: one is false precision, a system that claims to cleanly settle once and for all what is inherently messy and difficult; and the other is the denial that that 'something' exists at all.
- threatofrain 9y agoI feel like empiricists routinely and easily perform the partially opinionated act of operationalization, and what's being critiqued here is the <lack> of opinion in operationalization, except for the mocking of scientific appearance. Likewise, many discussions on consciousness are worthy of critique for lacking operationalization. We will always question whether the operationalization is a good proxy, but there's a difference when firms lack even the opinion to do so.
- ci5er 9y agoYou're just saying that abstract nouns have no explicit referent like a brick or apple. I take exception to democracy as a noun or attribute like 'truth', however. Truth is an abstract attribute. Sometimes a noun. Democracy is a process: two wolves and a sheep voting on what's up for dinner.
- nostrademons 9y agoThis is the case for basically all superlatives that apply to human emotions, though. What's a "great company to work for"? It's the company you want to keep working for. That's not going to be the same for every person - I have had great experiences at companies that are commonly known as "great companies to work for", but know coworkers that hated them. What's a "great investment"? It's something you can sell for a lot more than you paid for it. Why can you sell it for more than you paid for it? Because the person you're selling it to also believes it's a "great investment", and so you have a circular definition. What does it mean to "Make America Great Again"? Half the country believes we're doing it right now, and the other half believes we're in a bizarro throwback to the 1950s that isn't going to end well for anyone. So what does it mean to recruit a "great developer"? Well, it's when you hire the person that you wanted to hire. Yes, it's subjective - it's always subjective. That doesn't mean it has no meaning, though, because people ascribe meaning to subjective qualities all the time.
- sheepmullet 9y ago> This is the case for basically all superlatives that apply to human emotions, though. Exactly! What makes a great wife? Or a great friend?
- joe_the_user 9y agoThe distinction I'd elucidate from the gp's is that "great" is a problem comes because it's both a superlative and because it can't be quantified. You can ask for a developer who produce a large amount of average code. You can ask for a developer that produces very few terrible errors. You can ask for a develop who produces decent code and happens know domain X very well. You can ask for a developer who can mentor younger developers. You can ask for a developer who might needs virtually no supervision to succeed. And so-forth. They could all be different people. Perhaps you can sift until you somehow find someone who's all that but they'd be fool to join you because you're looking for perfection rather than looking for fulfill your actual needs. Superlatives are a terrible way to make any kind of serious decision. Salespeople love specifically because they tend to be unmeasurable and irrelevant when they are measurable. Thus the more superlatives are entering a decision process, the more you can tell you're sold shine rather than substance.
- dsacco 9y agoAre you saying there's no such thing as developers significantly better than the norm, or rather that it's infeasible to identify them? I don't think your point about precisely quantifying the definition of a "great developer" is a good one. I can capture an observation of superior capability in several dimensions without having to rigorously quantify the relative differences in capability or precisely why one is more capable than the other. If one developer accomplishes in a few days what takes another two weeks with code that is at least as maintainable and performant, that developer is better. If that developer is better when put next to most of the other developers you have around you, then they're a "great developer." Your focus on quantifiable definition seems to imply that significant differences in human capital can only be ascertained mathematically, but that doesn't reflect quite a lot that we're already familiar with in everyday life. If I'm placed in front of two walls which are both much taller than I am, I can see which of the two is taller than the other if the difference is significant as long as the tops aren't out of sight. I don't need to quantify this; I could just say, "One looks to be a lot taller than the other, but I'm not sure by how much exactly, or what their respective heights are." I'll happily agree that a lot of tech interviewing (and interviewing in general) is dreadfully broken. I'll also happily agree that figuring out which skills to prioritize and how to judge the level of those skills in candidates is very difficult work. But I strongly disagree that the fundamental premise of that work is intrinsically "voodoo." If your bar is a mathematical formalization then of course you're going to be disappointed, but that also holds true for the majority of our professional and personal activity.
- andrewstuart 9y agoIn a sentence, I am saying that pretty much every recruiting process at every company, when it assesses developers, results in a meaningless decision, despite the universal certainty at every company that their recruiting process absolutely definitely results in making accurate decisions about people. Did the interview process find a great developer? Maybe. Did the recruiting process reject great developers? How would you ever know? I'm pretty much certain that most companies reject many more developers who would be really productive/effective, than they choose to hire.
- 9y ago
- SmellTheGlove 9y agoNot only that, but is anyone really "great" across multiple/all domains? I've worked across a pretty diverse set of businesses and functions in management, and I can say pretty confidently that "greatness" is very dependent upon fit for the problem space, and I do mean that to work both ways: Fit is incredibly dependent upon the individual giving a shit about that problem space. No doubt someone could be very good working on something they don't care that much about, but we're talking great here, right? That means talking/writing in broad strokes about great developers approaches impossible. I also don't think you need "great" developers in most roles. Sometimes average is exactly what you need, and hiring above that just leads to having a great person leave because they're not particularly challenged or enthusiastic about the role. And to wrap all of that up, I don't even know that I can objectively define what "great" really means anyway, so maybe I have no place in this discussion :)
- perl4ever 9y ago>Fit is incredibly dependent upon the individual giving a shit about that problem space. Amen. That should be obvious, but I don't think I've ever had any recruiter or phone screen where they acknowledged that in any way. I responded to a recruiter recently by asking what the company actually did, and they reacted like I was just hamfistedly demanding the company name or prematurely trying to research for the interview, but I really just wanted to know - what would I actually be contributing to the world in this job, never mind the technical challenges, the buzzwords, and the paycheck. It's like everyone is assumed to be at best apathetic to the larger context of their job, apart from delusions of grandeur about "changing the world".
- SatvikBeri 9y agoI often tell people that before they think about their hiring process, they should think about their performance reviews. You're doing those with multiple orders of magnitude more data, yet most managers will say their performance reviews have a lot of uncertainty. I spent a lot of time at my last job developing ways to measure performance across my team. One big surprise was that I had to significantly change the way we assigned work in order to make it at all measurable. Rather than giving people well-defined tasks, I switched to assigning projects that had some amount of (expected) business value. This meant the projects had to include things like deployment, talking to marketing, and whatever needed to happen to make it useful. Generally each one was 3-6 weeks of work. This provided a good first-order measure of effectiveness. Importantly, it helped people change their habits to contribute more business value. I found massive variation in the junior engineers, easily with 5-10x differences between the most and least effective. The differences in senior engineers were smaller, but still ~2x. Of course, there are other major factors you have to take into account, like code quality and mentoring. I developed similarly detailed systems for measuring those. Once again, you have to change the way you do things if you want to make them measurable, often accepting local inefficiencies for global insight. With all that data in hand, I correlated that to our hiring ratings. The results? Our two best engineers were both borderline in the hiring process, and would have been rejected except for a strong vote of confidence by our senior engineer, who, when we checked, turned out to have a basically perfect track record on hiring judgments-much better than anyone else in the company, me included.
- majormajor 9y agoIME it's hard to precisely rank average devs who are in the middle. There's a lot of clumping. It's a lot less hard to identify overperformers or extreme underperformers. You mention the big possible range, but I've never had to dive that deeply into data to spot them - I've used the data to check my instincts, but have generally been right. And if you don't trust your instincts, just ask your team! They know whose code reviews they dread, who never offers helpful suggestions, who seems to do just the minimum. It's sometimes tricky (people often don't want to say bad things about their peers), but generally people already know. I know what I'm looking for in my performance reviews, and so I know what I'm looking for in my candidates. A competent coding baseline, and skills in the non-coding areas (knowing what not to build, anticipating edge cases, ability to distill business logic, ability to explain complex things clearly...). But this list is quite different from the "standard" tech interview loop: algorithms after algorithms with a sprinkling of "design a URL shortener."
- spitfire 9y agoI've posted this a few times before (taken shamelessly from tokenadult), but I'll post it again: There are three main criteria for hiring (in knowledge work environments): 1 General mental ability (Are they generally smart) 2 Work sample test (of real, representative work, no hazing!) 3 Integrity (Because 1 and 2 don't matter if you hire a psychopath or narcissist(cough) ). For 1 and 3 there are well known tests and you can find psychologists to administer them. Apply these tests evenly to everyone, even the people you're sure you want. The other approach you can take is the IBM/60's corporate approach. Which is apply the above approach to the "office girls" EI: general population and train the smart ones for the job. This can work surprisingly well if you have good people in leadership. http://emilkirkegaard.dk/en/wp-content/uploads/Schmidt-and-Hunter-1998-Validity-and-Utility-Psychological-Bulletin.pdf http://emilkirkegaard.dk/en/wp-content/uploads/Schmidt-and-H...