6 ms·

Frankly, I'm surprised that instead of recruiters or agents, there aren't firms dedicated to employee assessment. The value add to employers would be a standar

Frankly, I'm surprised that instead of recruiters or agents, there aren't firms dedicated to employee assessment. The value add to employers would be a standard assessment tool, reduced screening time for employees that can be eliminated quickly based on lack of skill, and using an assessment that is potentially better correlated with good employees than things like degree and school, or previous employer (decreasing both false positives from people who can't hack it despite their credentials, and decreasing false negatives by letting people with the skills demonstrate them). For prospective employees, you get a single assessment that may open up a lot of jobs. Since the company doing the assessment isn't the one hiring you, they can tell you what you need to work on, even provide training opportunities. Imagine someone telling you, "You aren't quite what we need, take these onlike courses and pass them and we will almost definitely be able to find someone to hire you." And by acting as an intermediary between prospective employees and employers, the time cost of submitting resumes to employers that aren't well matched is decreased. Further you might be partially eligible for a job, but lack specific skills, and they can ask if you are willing to take a temporary pay cut in return for on the job training as you work. One of the things personally holding me back, is that you really need to be in school to get an internship. I already have a degree, just not in CS, and I'd love to be able to learn on the job at the intern level, instead of feeling stuck unless I go back for a master's degree. I think there is a real opportunity, not just for such a company, but for a company that creates a standard or standards for assessment and licenses it/them to different companies. Further, such assessments would be useful to assess not just individuals, but educational programs. Compare performance for people who graduate from different programs, including online training.

imaginenore 12y ago
> I'm surprised that instead of recruiters or agents, there aren't firms dedicated to employee assessment. It's a very hard problem that hasn't been solved yet. What are you going to test? Logical problems? Programming puzzles? Knowledge of algorithms? Knowledge of technical details? GPA? None of them strictly correlate with how productive / useful a programmer is.
- lsiebert 12y ago
  how productive/useful a programmer is is, frankly, a slippery concept. Especially since things like attractiveness, gender, race and so on can be shown to affect such assessments unconsciously. Seriously, we live in a world where height is correlated with salary. http://www.apa.org/monitor/julaug04/standing.aspx http://www.apa.org/monitor/julaug04/standing.aspx My degree is in psychology, not computer science, so let me see if I can give you a general answer with a specific example of how you might work to figure out this. Short Answer: Research. Long Answer: You'd start with a hypothesis, and you'd test it and then improve your testing based on the results. You try to nail down what you can, and eventually you have an internally consistent system, which is about the best you can do. For example, you might say, "Knowledge of Big O should be positively correlated with job performance." First you come up with an assessment of knowledge of Big O. A test with variable questions that assesses such knowledge. You probably test the assessment by comparing how well a person does if they take the test twice (using Pearson correlation, or maybe Spearman), to see if you have test-retest reliability, and dividing the questions in half randomly, and seeing if it's internally consistent. So you know your test appears to be reliable... you can't necessarily say that it's measuring Big O perfectly, but you can argue that it's measuring Big O knowledge to some degree. You'd start out with a big company, or multiple companies, and assess the programmers there, then use something like relative income (controlling for company, gender, the afore mentioned height, whatever) to see if you are right. Or use another metric, git commits, lines of code. You are dealing with a system where nothing is perfectly correlated with being a great programmer, where how highly you assess a person may be based on characteristics that don't earn your company any money but which you are unconsciously biased towards, but you can still differentiate based on something. Let's say that yes, it seems like better programmers know big O better. Then you might assess big O skills of new hires, and track them to see how successful they are. Maybe you find out, yes, those that knew big O better did better, which is important, because it might be that people that did better then end up needing to know more about big O. Maybe you might give some number of randomly selected new employees a course on big O, and retest all of them afterwards to measure improvement, and then see if the ones who were trained do better then the untrained ones. At that point, it's probably worth while to test people's big O skills before hiring them. So you take your big O test and improve it so that people can't memorize the test, and start testing people. It might also be worth giving new hires a course on big O, if you can show that such training improves their value in the company. Is Big O everything somebody needs to know? Probably not. But it's still more information then you had, and you can maybe get a sense of how it's useful. Conversely you might find that people that know Big O better do worse as employees. Or that it helps certain programmers more then others... for example people making graphically intensive games or developing databases may need to know more about it then people who work with stuff that I/O bound.
jiggy2011 12y ago
Who pays for the tests? If candidates must pay then you will filter out any candidates who are unwilling or unable to invest the time and money required for completing the tests. If you ask the employer to pay then they must have some way to filter candidates before they assign them the tests, otherwise they have to pay for 100s of tests. And this is even assuming that you can construct generic tests that correspond closely to the requirements of the employer.
- lsiebert 12y ago
  I'm not sure who pays for them. I think that paying for an assessment that actually tells you what you need to improve to get work might be useful for people. I know there are plenty of people who either are too confident, or too insecure about their abilities. And if it's a better assessment of ability then what HR currently uses, then isn't that worth some of what a company pays for HR? Probably it's something that would need to be worked out by the market, hence my idea of one company that comes up with the assessment and then licenses it, and multiple licensees that use the assessment, experimenting with how to pay for it. You could also assess existing employees, and figure out ways they could improve, or why one employee does better than others. In fact that might be a good way to begin research, albeit with some clear biases based on existing hiring practices. Also such assessments would arguably be less biased by things like unconscious sexism/racism, which could be useful for figuring out raises. I don't think that such tests would necessarily replace in house interviews. The goal isn't to create a perfect assessment, the goal would be to improve assessment. Let another company do the equivalent of a phone screening. As for generic tests... there are things that companies assess. knowledge of big O for example. Also such assessments could be used to assess more then individuals, things like the relative benefit of training programs. And eventually you could assess particular assessments based on performance, so you could improve the testing itself. It just seems silly to me to have every company do ad-hoc assessments on their own, when the whole industry could benefit by reducing inefficiencies.
  - jiggy2011 12y ago
    Thing is that there are already tests out there that test for programming ability but they tend to either be very academic and mathematically focused like project euler or on trivia related to specific languages or technology. When you have a working team, chances are that you already know most peoples strengths and weaknesses as it relates to the job function, so a test isn't so useful in finding areas for improvement. When hiring someone new you are either looking for people with the same skill set or a complimentary one. The problem remains that the best hires are probably in enough demand and know that they can get a job without having to take your test, especially if they need to pay to take it, so the people most likely to try and get a good score will be the worse hires who have a strong incentive to try and game the test which would be easy if the questions don't change often. Also, if the test were proven to be a reliable assessment of skills across the entire industry then people who scored highly on the test would quickly start asking for much higher salaries. But what employers are looking for is to get a lower salary on employees who have a good skill match for the employers specific problems even if the don't score well in some areas. So there is a reason for employers to develop their own tests.
yazaddaruvala 12y ago
I think the problem here is just the complexity of the issue. I doubt there can be a standardized test to evaluate you on your software engineering ability. If such a test could exist I'm sure University Degrees and GPAs would be the only job requirement. However, not so sadly, GPAs are still only a good heuristic. I think the problem of understanding if a person is a capable engineer is so specialized that only a better engineer would be able to evaluate adequately. In which case, are these companies really going to hire and pay engineers competitively to be recruiters/evaluators? And even still, 1. their evaluation would only be for the specialization that was evaluated and 2. Depending on the amount of time spent evaluating, to predict a candidates average performance over time, through stressful times and through calm times, rather than just their perhaps peek or perhaps low performance over the stressful evaluation period, will be exceptionally hard. Maybe I'm wrong but this is just from empirical evidence that I've gathered working with people in university and in the real world during my career. Don't get me wrong, I agree, as demonstrated by its high barriers to entry, it is a real opportunity if someone is able to simplify or scale this. However, I'm not as optimistic as you seem to be that this is even possible. I hope I'm wrong, it would sure make my life even nicer. I would just start working as a freelancer, and take six month vacations since it would be so easy to "get back in the game".
- lsiebert 12y ago
  Maybe you have to be a capable engineer to evaluate others. But you could still potentially do better then HR. As for GPAs, read http://qz.com/180247/why-google-doesnt-care-about-hiring-top-college-graduates/ http://qz.com/180247/why-google-doesnt-care-about-hiring-top...
dccoolgai 12y ago
I already sort of do this with codewars, albeit not as precisely. I have my interviewees get to level 4 before I hire them - and I can go bck and see how clean their solution to problems are, generally. It's worked very we'll so far.
sunir 12y ago
Would Clearfit.com fit the bill?
altay 12y ago
(10x Management cofounder here.) Yup, this pre-vetting is a key part of the service we provide for companies. (We work for the talent, but of course we need to provide value to both sides of the marketplace.) A large company, which will remain unnamed, works with a handful of engineering contracting outfits, but they like us because we're the only one they trust to provide them with good talent out of the box. With the other companies, they have to go through the full interview rigamarole for each contractor; with 10x, they just tell us what they need and we hook it up.

imaginenore 12y ago

> I'm surprised that instead of recruiters or agents, there aren't firms dedicated to employee assessment. It's a very hard problem that hasn't been solved yet. What are you going to test? Logical problems? Programming puzzles? Knowledge of algorithms? Knowledge of technical details? GPA? None of them strictly correlate with how productive / useful a programmer is.

lsiebert 12y ago
how productive/useful a programmer is is, frankly, a slippery concept. Especially since things like attractiveness, gender, race and so on can be shown to affect such assessments unconsciously. Seriously, we live in a world where height is correlated with salary. http://www.apa.org/monitor/julaug04/standing.aspx http://www.apa.org/monitor/julaug04/standing.aspx My degree is in psychology, not computer science, so let me see if I can give you a general answer with a specific example of how you might work to figure out this. Short Answer: Research. Long Answer: You'd start with a hypothesis, and you'd test it and then improve your testing based on the results. You try to nail down what you can, and eventually you have an internally consistent system, which is about the best you can do. For example, you might say, "Knowledge of Big O should be positively correlated with job performance." First you come up with an assessment of knowledge of Big O. A test with variable questions that assesses such knowledge. You probably test the assessment by comparing how well a person does if they take the test twice (using Pearson correlation, or maybe Spearman), to see if you have test-retest reliability, and dividing the questions in half randomly, and seeing if it's internally consistent. So you know your test appears to be reliable... you can't necessarily say that it's measuring Big O perfectly, but you can argue that it's measuring Big O knowledge to some degree. You'd start out with a big company, or multiple companies, and assess the programmers there, then use something like relative income (controlling for company, gender, the afore mentioned height, whatever) to see if you are right. Or use another metric, git commits, lines of code. You are dealing with a system where nothing is perfectly correlated with being a great programmer, where how highly you assess a person may be based on characteristics that don't earn your company any money but which you are unconsciously biased towards, but you can still differentiate based on something. Let's say that yes, it seems like better programmers know big O better. Then you might assess big O skills of new hires, and track them to see how successful they are. Maybe you find out, yes, those that knew big O better did better, which is important, because it might be that people that did better then end up needing to know more about big O. Maybe you might give some number of randomly selected new employees a course on big O, and retest all of them afterwards to measure improvement, and then see if the ones who were trained do better then the untrained ones. At that point, it's probably worth while to test people's big O skills before hiring them. So you take your big O test and improve it so that people can't memorize the test, and start testing people. It might also be worth giving new hires a course on big O, if you can show that such training improves their value in the company. Is Big O everything somebody needs to know? Probably not. But it's still more information then you had, and you can maybe get a sense of how it's useful. Conversely you might find that people that know Big O better do worse as employees. Or that it helps certain programmers more then others... for example people making graphically intensive games or developing databases may need to know more about it then people who work with stuff that I/O bound.

jiggy2011 12y ago

Who pays for the tests? If candidates must pay then you will filter out any candidates who are unwilling or unable to invest the time and money required for completing the tests. If you ask the employer to pay then they must have some way to filter candidates before they assign them the tests, otherwise they have to pay for 100s of tests. And this is even assuming that you can construct generic tests that correspond closely to the requirements of the employer.

lsiebert 12y ago
I'm not sure who pays for them. I think that paying for an assessment that actually tells you what you need to improve to get work might be useful for people. I know there are plenty of people who either are too confident, or too insecure about their abilities. And if it's a better assessment of ability then what HR currently uses, then isn't that worth some of what a company pays for HR? Probably it's something that would need to be worked out by the market, hence my idea of one company that comes up with the assessment and then licenses it, and multiple licensees that use the assessment, experimenting with how to pay for it. You could also assess existing employees, and figure out ways they could improve, or why one employee does better than others. In fact that might be a good way to begin research, albeit with some clear biases based on existing hiring practices. Also such assessments would arguably be less biased by things like unconscious sexism/racism, which could be useful for figuring out raises. I don't think that such tests would necessarily replace in house interviews. The goal isn't to create a perfect assessment, the goal would be to improve assessment. Let another company do the equivalent of a phone screening. As for generic tests... there are things that companies assess. knowledge of big O for example. Also such assessments could be used to assess more then individuals, things like the relative benefit of training programs. And eventually you could assess particular assessments based on performance, so you could improve the testing itself. It just seems silly to me to have every company do ad-hoc assessments on their own, when the whole industry could benefit by reducing inefficiencies.
- jiggy2011 12y ago
  Thing is that there are already tests out there that test for programming ability but they tend to either be very academic and mathematically focused like project euler or on trivia related to specific languages or technology. When you have a working team, chances are that you already know most peoples strengths and weaknesses as it relates to the job function, so a test isn't so useful in finding areas for improvement. When hiring someone new you are either looking for people with the same skill set or a complimentary one. The problem remains that the best hires are probably in enough demand and know that they can get a job without having to take your test, especially if they need to pay to take it, so the people most likely to try and get a good score will be the worse hires who have a strong incentive to try and game the test which would be easy if the questions don't change often. Also, if the test were proven to be a reliable assessment of skills across the entire industry then people who scored highly on the test would quickly start asking for much higher salaries. But what employers are looking for is to get a lower salary on employees who have a good skill match for the employers specific problems even if the don't score well in some areas. So there is a reason for employers to develop their own tests.

yazaddaruvala 12y ago

I think the problem here is just the complexity of the issue. I doubt there can be a standardized test to evaluate you on your software engineering ability. If such a test could exist I'm sure University Degrees and GPAs would be the only job requirement. However, not so sadly, GPAs are still only a good heuristic. I think the problem of understanding if a person is a capable engineer is so specialized that only a better engineer would be able to evaluate adequately. In which case, are these companies really going to hire and pay engineers competitively to be recruiters/evaluators? And even still, 1. their evaluation would only be for the specialization that was evaluated and 2. Depending on the amount of time spent evaluating, to predict a candidates average performance over time, through stressful times and through calm times, rather than just their perhaps peek or perhaps low performance over the stressful evaluation period, will be exceptionally hard. Maybe I'm wrong but this is just from empirical evidence that I've gathered working with people in university and in the real world during my career. Don't get me wrong, I agree, as demonstrated by its high barriers to entry, it is a real opportunity if someone is able to simplify or scale this. However, I'm not as optimistic as you seem to be that this is even possible. I hope I'm wrong, it would sure make my life even nicer. I would just start working as a freelancer, and take six month vacations since it would be so easy to "get back in the game".

lsiebert 12y ago
Maybe you have to be a capable engineer to evaluate others. But you could still potentially do better then HR. As for GPAs, read http://qz.com/180247/why-google-doesnt-care-about-hiring-top-college-graduates/ http://qz.com/180247/why-google-doesnt-care-about-hiring-top...

dccoolgai 12y ago

I already sort of do this with codewars, albeit not as precisely. I have my interviewees get to level 4 before I hire them - and I can go bck and see how clean their solution to problems are, generally. It's worked very we'll so far.

sunir 12y ago

Would Clearfit.com fit the bill?

altay 12y ago

(10x Management cofounder here.) Yup, this pre-vetting is a key part of the service we provide for companies. (We work for the talent, but of course we need to provide value to both sides of the marketplace.) A large company, which will remain unnamed, works with a handful of engineering contracting outfits, but they like us because we're the only one they trust to provide them with good talent out of the box. With the other companies, they have to go through the full interview rigamarole for each contractor; with 10x, they just tell us what they need and we hook it up.