3 ms·

Reading all of these accounts of horrors in HackerNews I feel my idea of creating a "TOEFL for Developers" would be perfect. (I am hesitant to describe my idea

Reading all of these accounts of horrors in HackerNews I feel my idea of creating a "TOEFL for Developers" would be perfect. (I am hesitant to describe my idea, but WTH, I'm currently not in a position to act on it, and they say ideas are dime a dozen so...): For those of you for whom English is not your first language, you surely know what TOEFL is: Test of English as a Foreign Language. The private company provides a test where they evaluate a person for proficiency on different sets of English skills: Reading, Writing, Speaking, general comprehension. At the end of the test they give you a score for each of those and then a general score. Now, the nice thing about the TOEFL score is its reputation: When you are going to study into a University (from Oxford to Liverpool, from Harvard to Queensland), if you are not a native English speaker, the university asks you to take a TOEFL (or equivalent) and get a score higher than X. As long as you provide that score, the university won't ask any questions. They don't doubt your English proficiency skills. I want to do something like that but for development: Sure, University thought you CS101, CS201, etc. But what companies are looking for are specific skills (similar to say, https://sijinjoseph.netlify.app/programmer-competency-matrix/ https://sijinjoseph.netlify.app/programmer-competency-matrix... or https://docs.google.com/spreadsheets/d/131XZCEb8LoXqy79WWrhCX4sBnGhCM1nAIz4feFZJsEo/edit https://docs.google.com/spreadsheets/d/131XZCEb8LoXqy79WWrhC... ). These very concrete skills that can be concretely measured. I want this becase I've been dealing with recruiting, interviewing and also being interviewed for most of my career. And in my experience setting up a "recruiting wheel" in a company or startup is very painful. Most developers don't know how to perform interviews. Or don't care (who has time for that?). As an Engineer manager, having to comb through 400 resumes a week is time consuming and unproductive. So I imagine a scenario where, there is this proverbial "Full Stack Developer" certification authority which is independent, very reputable and unbiased: They won't tell me if X or Y developer will make it into my team. They will tell me how does the person score in "Algorithms/Data Structures" or "Micro Services development" or "Frontend development" and then I will choose whether I want to hire them or not (according to my current requirements). That way, I only have to do a culture-fit interview and that's it. In the past I have dealt with "Sourcing Agencies" that try to do something like that: They do the 1st technical interview, and they say that there will be alignment to what I want. But that was not the case: They send me 2 candidates and get very surprised when I reject one and not the other. And I feel our objectives are not aligned, so everybody is frustrated. Moreover, I don't want to have to deal with hiring agencies. And paying 1 to 3 salary months is too fucking expensive. Now, as a candidate/developer: I would love to have a "TOEFL" like certificate that I can show, with my proficiency. That way, I can "look for a job" passively. I don't have to be making countless of code interviews showing one time after another what I know. I only do it ONCE for an authoritative company, and everybody else can check my qualifications there. Companies like Microsoft or TripleByte have tried to do something like that, but the problem is again that they do other things: Their interests clash. Why would I trust LinkedIn? That's why TOEFL monetization make sense: The candidate has to pay for it. I am aware there are some certifications: Oracle whatever, Microsoft whatever, AWS whatever. But those are either too specific/niche (WebLogic Server 12c administration exam) or they are only a "badge" (AWS Cloud Engineer) that doesn't tell you that much. There is opportunity here for disruption. And I know that the problems that I've faced are faced by countless of other CTOs/Technology Heads.

quanticle 5y ago
Now, the nice thing about the TOEFL score is its reputation: When you are going to study into a University (from Oxford to Liverpool, from Harvard to Queensland), if you are not a native English speaker, the university asks you to take a TOEFL (or equivalent) and get a score higher than X. As long as you provide that score, the university won't ask any questions. They don't doubt your English proficiency skills. They should. My university had a minimum TOEFL score requirement. The number of grad students who passed that requirement, but then couldn't write a coherent English sentence to save their lives was still incredibly high.
nitrogen 5y ago
certification authority which is independent, very reputable and unbiased That is a big ask. There have been many attempts at CS-adjacent certifications, some of them government sponsored, but none of them find widespread use. The ones I've personally encountered were grossly out of date (e.g. a C++ assessment that didn't mention templates at all, in 2012), and weren't even mentioned during interviews by the company that demanded them. It's not just a matter of trust, either. Our industry is incredibly conceptually diverse, despite the common current of abstract patterns that underlies everything. There is no specific or generic skill test that won't be out of date in a year, and forcing the required degree of standardization on our industry to make that possible would effectively destroy it. In the mean time, the only organizations incentivized to keep their assessments up to date are the vendors themselves, who as you note, have conflicting interests. And at root, if you want to be even slightly more generic than a vendor certification, you are basically trying to test for intelligence and adaptability.
bmn__ 5y ago
> my idea of creating a "[standard aptitude test] for Developers" Already exists. https://news.ycombinator.com/item?id=27662804 https://news.ycombinator.com/item?id=27662804 Don't you have apprenticeships in your country?
ozim 5y ago
I am compelled to write all the list of things why it will not work :) unfortunately I don't have that much time in my life. Don't take it as putting you down, I would love to see such thing working and live in real world, so look at it as problems that would need to be solved while aiming for grand solution. First, there are wrong incentives at play, good guys currently don't need that as they mostly are employed and look for jobs passively. Good guys don't care about certificates and you want to certify good guys so your certificates mean something and can build trust. Weak guys will be spending money to cheat their way around just to get passing grade, which will cost money to prevent. Then you have hiring managers that will not use this exam as good enough just to hire a person unless there is big company that would do the same. So you have chicken and egg problem, because no one will trust your certification and to get the trust someone would have to use it. My idea is that you cannot effectively hire at scale. I could hand pick maybe 5 devs/year for my company, getting to real numbers in my small company I can see that we can hire maybe 2 developers per year. When We hire someone it worked for me with simple questions and attitude check. I don't see a way to hire 100 good developers in 1 year, that is not possible. Maybe at a scale you can hire 10-20 good developers a year but you have to have 3-5 people involved full time - and ideally those 3-5 people should be nice and technical - where in the end you don't want to use them only for hiring because those are probably your best developers with soft skills. For that competency matrix - that is something you can do when you have developers inside of the company and you can observe them for at least 6 months. Good luck getting that in practical testing setup for 2 or less hours. I also find those kind of matrices terrible and not useful for evaluating people or projects. I work on a project where we were not fitting in anything of what was in the matrix .. yet we were delivering good quality for business all the time, while teams that were stuck on their "competency matrix" trying to improve meaningless points ended up disbanded and not working anymore in the organization.

quanticle 5y ago

Now, the nice thing about the TOEFL score is its reputation: When you are going to study into a University (from Oxford to Liverpool, from Harvard to Queensland), if you are not a native English speaker, the university asks you to take a TOEFL (or equivalent) and get a score higher than X. As long as you provide that score, the university won't ask any questions. They don't doubt your English proficiency skills. They should. My university had a minimum TOEFL score requirement. The number of grad students who passed that requirement, but then couldn't write a coherent English sentence to save their lives was still incredibly high.

nitrogen 5y ago

certification authority which is independent, very reputable and unbiased That is a big ask. There have been many attempts at CS-adjacent certifications, some of them government sponsored, but none of them find widespread use. The ones I've personally encountered were grossly out of date (e.g. a C++ assessment that didn't mention templates at all, in 2012), and weren't even mentioned during interviews by the company that demanded them. It's not just a matter of trust, either. Our industry is incredibly conceptually diverse, despite the common current of abstract patterns that underlies everything. There is no specific or generic skill test that won't be out of date in a year, and forcing the required degree of standardization on our industry to make that possible would effectively destroy it. In the mean time, the only organizations incentivized to keep their assessments up to date are the vendors themselves, who as you note, have conflicting interests. And at root, if you want to be even slightly more generic than a vendor certification, you are basically trying to test for intelligence and adaptability.

bmn__ 5y ago

> my idea of creating a "[standard aptitude test] for Developers" Already exists. https://news.ycombinator.com/item?id=27662804 https://news.ycombinator.com/item?id=27662804 Don't you have apprenticeships in your country?

ozim 5y ago

I am compelled to write all the list of things why it will not work :) unfortunately I don't have that much time in my life. Don't take it as putting you down, I would love to see such thing working and live in real world, so look at it as problems that would need to be solved while aiming for grand solution. First, there are wrong incentives at play, good guys currently don't need that as they mostly are employed and look for jobs passively. Good guys don't care about certificates and you want to certify good guys so your certificates mean something and can build trust. Weak guys will be spending money to cheat their way around just to get passing grade, which will cost money to prevent. Then you have hiring managers that will not use this exam as good enough just to hire a person unless there is big company that would do the same. So you have chicken and egg problem, because no one will trust your certification and to get the trust someone would have to use it. My idea is that you cannot effectively hire at scale. I could hand pick maybe 5 devs/year for my company, getting to real numbers in my small company I can see that we can hire maybe 2 developers per year. When We hire someone it worked for me with simple questions and attitude check. I don't see a way to hire 100 good developers in 1 year, that is not possible. Maybe at a scale you can hire 10-20 good developers a year but you have to have 3-5 people involved full time - and ideally those 3-5 people should be nice and technical - where in the end you don't want to use them only for hiring because those are probably your best developers with soft skills. For that competency matrix - that is something you can do when you have developers inside of the company and you can observe them for at least 6 months. Good luck getting that in practical testing setup for 2 or less hours. I also find those kind of matrices terrible and not useful for evaluating people or projects. I work on a project where we were not fitting in anything of what was in the matrix .. yet we were delivering good quality for business all the time, while teams that were stuck on their "competency matrix" trying to improve meaningless points ended up disbanded and not working anymore in the organization.