5 ms·

As a former director of engineering who has hired many great developers and many terrible ones, I can safely say that grinding leetcode isn't a valid signal for

As a former director of engineering who has hired many great developers and many terrible ones, I can safely say that grinding leetcode isn't a valid signal for smart, just for conformity, ego, and time. Some of the worst developers I know could crush leetcode all day. Yes, it's an easy signal, it makes an unpredictable and messy process feel neat and binary. However, it is a false signal, and no amount of personal comfort changes that it has about as much effect on future employee productivity as hiring by height or favorite color. Unfortunately, software productivity is never going to be measurable. Therefore we will never ever have a reliable metric to even prove to ourselves that we are more or less productive than our coworkers, much less worth hiring. This fact should sober our prideful arrogance about hiring and introduce some humility. Our best practices are a farce with absolutely zero data to back up their validity. Ask yourselves where are the studies that prove candidates who crush leetcode perform better than those who couldn't? Where is the double blind research? We need a heaping dose of recognition that most of our best practices are there to validate the egos of the interviewers and give a neat binary decision without any mess. We have a toxic fraternity hazing ritual to make candidates bend the knee, with exactly zero evidence it works. We have the gall to call ourselves scientists, and we don't even measure a single thing.

michaelrpeskin 5y ago
Yes! See my earlier comment about how we hire. Want to add agreement that grinding leetcode is a terrible metric. The ideal candidate probably already has a full time job and if you add in a family and other responsibilities, they don’t have time to focus on leetcode. Heck, they probably are tired of thinking at work all day and don’t have the mental capacity to task switch into leetcode after work (I know I don’t!) I think leetcode is an antisignal
Mezzie 5y ago
> Unfortunately, software productivity is never going to be measurable. Therefore we will never ever have a reliable metric to even prove to ourselves that we are more or less productive than our coworkers, much less worth hiring. This fact should sober our prideful arrogance about hiring and introduce some humility. Our best practices are a farce with absolutely zero data to back up their validity. Ask yourselves where are the studies that prove candidates who crush leetcode perform better than those who couldn't? Where is the double blind research? I think there is something similar to art in GOOD software development, as somebody who does both. The primary reason I don't like coding for a living is the same reason I don't like writing fiction for a living: I really dislike being creative on demand and on a schedule. I find it stressful. I can't make myself get a story idea any more than I can make myself find an elegant solution to a coding issue. Sitting and staring at the screen more isn't going to solve the issue for me. Software development is to math/CS what painting is to color theory. There is science, but there's also creation. That's why it's so hard to measure and hire for. Imagine trying to judge 1000 newly graduated oil painters who had all taken the exact same classes. How would you decide who was 'best'?
moneywoes 5y ago
I agree wholeheartedly. However, what is the alternative? There are way too many developers out there and too few jobs
- JackMorgan 5y ago
  I think the opposite is true, we're doing the same thing doctors and lawyers do: artificially restrict the supply, driving up demand and pay. Not top down and centralized, but individually and decentralized. There's plenty of jobs, almost every company reports a 10% shortage, but the developers won't let in anyone.
Jensson 5y ago
> Our best practices are a farce with absolutely zero data to back up their validity. Ask yourselves where are the studies that prove candidates who crush leetcode perform better than those who couldn't? Where is the double blind research? Google has data on this, when I worked there anyone working there could view it. Performance on their technical interviews is the strongest signal they have that someone will perform well at the job. It is stronger than every other signal. They have tried a bunch of different things, Google HR hates these technical interviews since it is hard for them to game so they work hard to remove it, but they can't since they can't come up with anything with a stronger signal. Performance on a single technical interview isn't a strong signal, but the combined performance on 5 interviews is a strong signal. If you only run a single of these instead of many with many different people then you will hire a lot of bad people anyway. Also if you pay less than Google and similar then you will get their rejects, there is probably something wrong with the people who are good at algorithms but get rejected by FAANG, not sure why you would want to hire those.
- JackMorgan 5y ago
  I'm very interested in what else they tried. Also very interested to see the research, is there somewhere it is published? I do believe that performance in a technical interview corresponds closely with overall performance, but I'm very curious how they measure job performance, considering tech job performance is mostly a subjective popularity contest. I'm not convinced that is a valuable signal either. This is what I was getting at, we can't measure productivity _of employees_ with any rigor, so how can we know if our interviewing process works. Since job performance is so subjective, it's not very surprising to me that people hired with a different process aren't as well liked. The one thing I know about people is they like people like them. People who can and will grind leetcode are a very similar group indeed, so a person who didn't do that likely sticks out like a sore thumb. Perhaps all this is accounted for, in which case I'd love to see it.
  - Jensson 5y ago
    They base it on performance reviews by the managers. The mangers aren't the person rating the candidate interviews in the first place or deciding if they get hired, so it isn't just managers confirming their own hiring ratings. It isn't perfect but that is about as much as you can do, it is really hard to create a better model for evaluating hiring processes. I do believe you can do better than Google. But I don't believe you can do much better than Google at Google's scale with that many people if the goal is to hire people who can solve hard technical problems. In a small company you can have the founder or equivalent do the hiring, like early Google did, Larry Page reviewed every single candidate until they had thousands of employees. This is what they came up with after that. Also now instead of having Larry review everyone, they have the vice presidents review every hire in their org as an extra last step to ensure the hiring process doesn't turn bad or get gamed. Note: If you don't pay like Google, or your goal isn't to solve hard technical problems like Google has great need for, then you probably shouldn't try to hire like Google. Also if you just run a single algorithm question that is easy to look up online, then you are not copying their process, since Google forbids such questions and require multiple questions. Also if you just run an automatic algorithm challenge online then you miss all the soft skills part that comes up when you talk to a person when solving the problem, so not their process either. Also a person who isn't at least decent at algorithms cannot administer an algorithm interview properly. They need to come up with an original problem, and they need to identify when the candidate has solved the problem even if the candidate did it in a way you didn't expect. Most engineers at Google can do that since they hired for that from the start. Most engineers at most companies can't. Also if you have managers make the hiring decision instead of an independent committee then you aren't copying their process. And if your executives are not constantly checking that the process is in line with expectations and doesn't bring in a lot of bad candidates then you aren't doing their process. Etc etc. They do so many things for this to work, if you do all of that and wants the same thing as Google then it will work for you as well. But almost no companies wants to do all that work to get this working. They just say "Google does algorithm interviews, so go and ask the candidate an algorithm question and that is our process". Also some caveats about this process and why you hear so much bad things about it: - Managers at Google don't like it. They would prefer to hire the candidates they want, but as is it can be hard to even fill headcount you have been allocated if you aren't an attractive team, they would love to lower the bar to fill their seats. And the few people unattractive teams manage to hire often leave the team as soon as they can for better teams. I did that, I was first on a dying project and then I did well and joined a cool project working on cutting edge machine learning infrastructure. So this process isn't pushed by managers. It is the executives who want a well run organisation who wants this process. Which is why the executive review is mandatory, this process is first and foremost for their sake. And from heir perspective, hiring really smart people, put them in a poor project to test their ability and then get the best of those into the teams that matter is how you get great engineers to those important teams. There is no reason to lower the bar for poor maintenance projects even if those managers would love to do it. For Google the most important thing is that their search and advertisement algorithms become as great as possible, and that their infrastructure costs gets optimized down as much as possible. They need the best engineers in the world to work on those things because that is where the money is for Google, quantity doesn't matter there, the rest are just hobby projects but those hobby projects can work as a way to find talent to those important projects. You can't hire experts on search and ad algorithms, because Google is the world leader in both. Google has to cultivate such talent themselves, so their process focuses on finding people who maybe could be cultivated to do those things, no matter what technical role they hire for any person could potentially pivot to the teams that generate revenue. (Nice side effect is that every single technical role is filled by smart and ambitious people. When you talk with some guy maintaining a project you need to depend on, they are really quick to respond and answers your question or fixes your issue within minutes or hours. I never had problem with others not wanting to help or dragging things out. I'm sure it happens sometimes and maybe more in some areas than others, but I've never seen such a well run technical org anywhere else) - Engineers at Google don't like it. It forces them to spend a ton of time interviewing, also they would prefer if they could recommend their friends to get hired, but they can't since they have to go through the process as well. The exception is the meritocracy camp, which is pretty strong at Google but not very outspoken. - Engineers interviewing at Google of course don't like it, since it is a lot of work. / Sorry for writing a lot. I tend to use forums like this to structure my thoughts. This helps me better understand the pros and cons of things I know as well. Google's process is good at what it does, but not good at other things. Understanding what it is good at and what its weaknesses are is important. But now I think I have a better idea how to argue in this subject in the future. Although, once I feel I understand most things in a subject I stop arguing since I no longer learn anything. I like learning about just about every general concept, like hiring, business, programming, math, physics etc, discussions like this is what I do for fun.

michaelrpeskin 5y ago

Yes! See my earlier comment about how we hire. Want to add agreement that grinding leetcode is a terrible metric. The ideal candidate probably already has a full time job and if you add in a family and other responsibilities, they don’t have time to focus on leetcode. Heck, they probably are tired of thinking at work all day and don’t have the mental capacity to task switch into leetcode after work (I know I don’t!) I think leetcode is an antisignal

Mezzie 5y ago

> Unfortunately, software productivity is never going to be measurable. Therefore we will never ever have a reliable metric to even prove to ourselves that we are more or less productive than our coworkers, much less worth hiring. This fact should sober our prideful arrogance about hiring and introduce some humility. Our best practices are a farce with absolutely zero data to back up their validity. Ask yourselves where are the studies that prove candidates who crush leetcode perform better than those who couldn't? Where is the double blind research? I think there is something similar to art in GOOD software development, as somebody who does both. The primary reason I don't like coding for a living is the same reason I don't like writing fiction for a living: I really dislike being creative on demand and on a schedule. I find it stressful. I can't make myself get a story idea any more than I can make myself find an elegant solution to a coding issue. Sitting and staring at the screen more isn't going to solve the issue for me. Software development is to math/CS what painting is to color theory. There is science, but there's also creation. That's why it's so hard to measure and hire for. Imagine trying to judge 1000 newly graduated oil painters who had all taken the exact same classes. How would you decide who was 'best'?

moneywoes 5y ago

I agree wholeheartedly. However, what is the alternative? There are way too many developers out there and too few jobs

JackMorgan 5y ago
I think the opposite is true, we're doing the same thing doctors and lawyers do: artificially restrict the supply, driving up demand and pay. Not top down and centralized, but individually and decentralized. There's plenty of jobs, almost every company reports a 10% shortage, but the developers won't let in anyone.

Jensson 5y ago

> Our best practices are a farce with absolutely zero data to back up their validity. Ask yourselves where are the studies that prove candidates who crush leetcode perform better than those who couldn't? Where is the double blind research? Google has data on this, when I worked there anyone working there could view it. Performance on their technical interviews is the strongest signal they have that someone will perform well at the job. It is stronger than every other signal. They have tried a bunch of different things, Google HR hates these technical interviews since it is hard for them to game so they work hard to remove it, but they can't since they can't come up with anything with a stronger signal. Performance on a single technical interview isn't a strong signal, but the combined performance on 5 interviews is a strong signal. If you only run a single of these instead of many with many different people then you will hire a lot of bad people anyway. Also if you pay less than Google and similar then you will get their rejects, there is probably something wrong with the people who are good at algorithms but get rejected by FAANG, not sure why you would want to hire those.

JackMorgan 5y ago
I'm very interested in what else they tried. Also very interested to see the research, is there somewhere it is published? I do believe that performance in a technical interview corresponds closely with overall performance, but I'm very curious how they measure job performance, considering tech job performance is mostly a subjective popularity contest. I'm not convinced that is a valuable signal either. This is what I was getting at, we can't measure productivity _of employees_ with any rigor, so how can we know if our interviewing process works. Since job performance is so subjective, it's not very surprising to me that people hired with a different process aren't as well liked. The one thing I know about people is they like people like them. People who can and will grind leetcode are a very similar group indeed, so a person who didn't do that likely sticks out like a sore thumb. Perhaps all this is accounted for, in which case I'd love to see it.
- Jensson 5y ago
  They base it on performance reviews by the managers. The mangers aren't the person rating the candidate interviews in the first place or deciding if they get hired, so it isn't just managers confirming their own hiring ratings. It isn't perfect but that is about as much as you can do, it is really hard to create a better model for evaluating hiring processes. I do believe you can do better than Google. But I don't believe you can do much better than Google at Google's scale with that many people if the goal is to hire people who can solve hard technical problems. In a small company you can have the founder or equivalent do the hiring, like early Google did, Larry Page reviewed every single candidate until they had thousands of employees. This is what they came up with after that. Also now instead of having Larry review everyone, they have the vice presidents review every hire in their org as an extra last step to ensure the hiring process doesn't turn bad or get gamed. Note: If you don't pay like Google, or your goal isn't to solve hard technical problems like Google has great need for, then you probably shouldn't try to hire like Google. Also if you just run a single algorithm question that is easy to look up online, then you are not copying their process, since Google forbids such questions and require multiple questions. Also if you just run an automatic algorithm challenge online then you miss all the soft skills part that comes up when you talk to a person when solving the problem, so not their process either. Also a person who isn't at least decent at algorithms cannot administer an algorithm interview properly. They need to come up with an original problem, and they need to identify when the candidate has solved the problem even if the candidate did it in a way you didn't expect. Most engineers at Google can do that since they hired for that from the start. Most engineers at most companies can't. Also if you have managers make the hiring decision instead of an independent committee then you aren't copying their process. And if your executives are not constantly checking that the process is in line with expectations and doesn't bring in a lot of bad candidates then you aren't doing their process. Etc etc. They do so many things for this to work, if you do all of that and wants the same thing as Google then it will work for you as well. But almost no companies wants to do all that work to get this working. They just say "Google does algorithm interviews, so go and ask the candidate an algorithm question and that is our process". Also some caveats about this process and why you hear so much bad things about it: - Managers at Google don't like it. They would prefer to hire the candidates they want, but as is it can be hard to even fill headcount you have been allocated if you aren't an attractive team, they would love to lower the bar to fill their seats. And the few people unattractive teams manage to hire often leave the team as soon as they can for better teams. I did that, I was first on a dying project and then I did well and joined a cool project working on cutting edge machine learning infrastructure. So this process isn't pushed by managers. It is the executives who want a well run organisation who wants this process. Which is why the executive review is mandatory, this process is first and foremost for their sake. And from heir perspective, hiring really smart people, put them in a poor project to test their ability and then get the best of those into the teams that matter is how you get great engineers to those important teams. There is no reason to lower the bar for poor maintenance projects even if those managers would love to do it. For Google the most important thing is that their search and advertisement algorithms become as great as possible, and that their infrastructure costs gets optimized down as much as possible. They need the best engineers in the world to work on those things because that is where the money is for Google, quantity doesn't matter there, the rest are just hobby projects but those hobby projects can work as a way to find talent to those important projects. You can't hire experts on search and ad algorithms, because Google is the world leader in both. Google has to cultivate such talent themselves, so their process focuses on finding people who maybe could be cultivated to do those things, no matter what technical role they hire for any person could potentially pivot to the teams that generate revenue. (Nice side effect is that every single technical role is filled by smart and ambitious people. When you talk with some guy maintaining a project you need to depend on, they are really quick to respond and answers your question or fixes your issue within minutes or hours. I never had problem with others not wanting to help or dragging things out. I'm sure it happens sometimes and maybe more in some areas than others, but I've never seen such a well run technical org anywhere else) - Engineers at Google don't like it. It forces them to spend a ton of time interviewing, also they would prefer if they could recommend their friends to get hired, but they can't since they have to go through the process as well. The exception is the meritocracy camp, which is pretty strong at Google but not very outspoken. - Engineers interviewing at Google of course don't like it, since it is a lot of work. / Sorry for writing a lot. I tend to use forums like this to structure my thoughts. This helps me better understand the pros and cons of things I know as well. Google's process is good at what it does, but not good at other things. Understanding what it is good at and what its weaknesses are is important. But now I think I have a better idea how to argue in this subject in the future. Although, once I feel I understand most things in a subject I stop arguing since I no longer learn anything. I like learning about just about every general concept, like hiring, business, programming, math, physics etc, discussions like this is what I do for fun.