7 ms·

If those tests actually filtered out incompetent programmers, I'd say they were great, but my experience is that they all test very shallow knowledge that doesn

If those tests actually filtered out incompetent programmers, I'd say they were great, but my experience is that they all test very shallow knowledge that doesn't relate to ability to produce software that performs under load. If you could come up with such a test, it ought to just be used as a standard for everybody to take. And yes, as unpopular an opinion as it is around here, I'd love to see a "bar exam"/licensing board for software developers.

juancn 5y ago
I mean, you don't need a really hard test to weed out most hacks. In any case, if the test is hard, companies that know their shit in hiring, care more about how you approach it than what the end result is. In the end, the interview tries to figure out how it's like to work with you. Do we get along? Can we communicate effectively? Will you be happy and thrive here?
- nouveaux 5y ago
  You sound like a great hiring manager/interviewer. However, I have read stories of people who got rejected only because they did not finish the coding challenge in time. The engineer reported that the candidate communicated well, but did not finish. Hiring manager rejects the candidate.
  - juancn 5y ago
    Yeah, there are bad employers and bad interviewers. Sometimes the importance of it is underplayed. I always drill others that interview duty is the most important thing you can do at a company (even though it sometimes feels like a chore). The people you help choose, influences culture and long term viability of the company. Take it seriously and be fair with candidates. Having said that, figuring out a fair and effective interview process is HARD. It requires constant tweaking and monitoring. Also, you have to start from a decent set of values at the company. For example, we do shadow interviews. When we certify interviewers, we have them come to watch and listen to an interview (we explain to the candidate beforehand what's going on), and then after the interview ends, we meet with the shadow interviewer and discuss what they picked up, and explain why I went such and such way, what I was looking for, why such and such mistake is not a big deal, and so on. After a couple of these, we have them interview, but with a certified interviewer shadowing them. Just observing (we have them practice on us their interview exercises if any before the interview). And the we give feedback, you rushed a bit here, give them more time to be at ease with you, did you notice such and such, whatever... If everything goes well, we certify them, and they can fly solo. We try to have a varied pool of interviewers to avoid biases. The downside is that you need to do more interviews as a candidate, to get a fair assessment (i.e. no single person can derail you). In the end, we've all been in a crappy interviews and try our best to not inflict that on other people.
ryandrake 5y ago
Oops, I posted my rant[1] about our need for a bar exam before realizing you got there first. Fully agree, this profession really needs this. 1: https://news.ycombinator.com/item?id=30616361 https://news.ycombinator.com/item?id=30616361
Apocryphon 5y ago
The thing about the license is that it avoids the redundancy of having to pass the same orthogonal algorithm questions at every single company an applicant chooses to apply at. That alone is a great time-saver, and allows applicants the freedom to focus on actual work.
- jxidjhdhdhdhfhf 5y ago
  Not really. You'll still have interviews, they'll just be more personality and resume-driven. Personally, I'll take writing FizzBuzz over and over again vs. "where do you see yourself in 5 years?" ("I don't know but probably not working here") and "why do you want to work for our company?" ("to earn money").
  - Apocryphon 5y ago
    Most interview loops include Leetcode for the technical screen, then at least one domain-specific coding section (like a more tense pair programming section) during the on-site, and maybe another algorithms section on the on-site (several if you’re Google). The license would at least eliminate the algorithms section to focus on the domain-specific, which is at least more reflective of real-world work.
    - jxidjhdhdhdhfhf 5y ago
      You're assuming they will actually eliminate it though rather than just pick something else to test.
      - Apocryphon 5y ago
        Sure, but at least that might be something more relevant to day to day work, and require less grinding than Leetcode.
    - jxidjhdhdhdhfhf 5y ago
      Also, its a lot easier to study leetcode problems than it is to study for domain-specific ones (since there are many different domains). I would expect having more domain-specific problems would limit you to a narrower range of jobs.
lijogdfljk 5y ago
I've had moderate success discussing Rust (a language we hire for) at moderately low levels. Ie, i've had the thought that it's difficult to bullshit topics with any depth. Even basic questions, why isn't an Rc safe to send to a thread? How does an Rc work? What is the difference between an Rc and an Arc? etc I'm happy to accept "i don't know", those are valuable too. But the whole conversation process led me to believe a lot about the candidate. We've hired 10+ juniors and seniors with this process and i've not felt cheated yet. I still feel like i need to find a way to open up people who "freeze". I do that myself, so i can relate, and my strategy of interviewing is pretty 1:1. Still though, i've had more faith in it than live/take-home tests.
AnimalMuppet 5y ago
> ... they all test very shallow knowledge that doesn't relate to ability to produce software that performs under load. That's a later filter. The first filter is, can they code at all. Like, FizzBuzz level of "at all". I mean, if you need them to produce code that performs under load, you'd better interview for that too. But people tend to first worry about whether they can clear the lowest bar before they proceed further.
- sokoloff 5y ago
  It is difficult to comprehend how low-skilled the shallow-end of the applicant pool is. I think it’s gotten worse, not better, since the original FizzBuzz essay was written.
starlust2 5y ago
They actually do. The test isn't (or shouldn't be) that you understand string manipulation (or whatever) but rather that you can work through a new problem. There are many programmers who claim many years of experience but have poor problem solving skills. They can implement a CRUD form and basic tasks, but fall flat when faced with a problem they've never encountered before.
kragen 5y ago
> I'd love to see a "bar exam"/licensing board for software developers. Can't resolve IRQ conflicts? Don't know the big-O complexity of different Delaunay-triangulation algorithms? Out of date on which features Microsoft added to LINQ in different versions of C#? Don't know where the "launch debugger" button is in Android Studio? Sorry, we aren't allowed to hire you to work on Instagram's iPhone app. Better hope the police don't find out your Android has an unlocked bootloader; I hear prosecutors are really cracking down on unlicensed software development ever since the assassination. They say unregulated software development leaked his location to the Zetas, I don't know if I believe that.
- commandlinefan 5y ago
  Yep, that's exactly the sort of ridiculous hyperbole that gets thrown around every time somebody suggests any sort of standardization around professional software development!
  - kragen 5y ago
    There's no reason to think it's hyperbolic or ridiculous. We've seen lots of sorts of standardization around professional software development, with uniformly dismal results, and lots of criminalization of experimentation by astoundingly incompetent authorities. — ⁂ — The first official standard for professional software development was COBOL, shortly followed by JOVIAL. In the 01970s the US military standardized on Ada for, among other things, their hard-real-time systems; it took about a decade of violating the standard to convert Ada into a system that was somewhat suitable for hard-real-time systems, after which an Ada-induced problem destroyed the first Ariane 5. The ACM withdrew from the SWEBOK process after it became apparent that knowledge of actual programming was going to be completely excluded; instead SWEBOK certifies that you endorse the obsolete, counterproductive management processes to which "agile development" was conceived as an antidote. (If you're familiar with the BOK for some other branch of engineering, you may be aware that, for example, the structural engineering BOK includes not only project engineering practices but also what makes structures stand up or fall down, and the civil engineering BOK includes things like engineering mechanics, strength of materials, and chemistry. Presumably this is because political decisionmakers have been able to observe the difference between successful and unsuccessful skyscrapers and dams and infer which engineers were responsible.) The A+ certification for IT work is impossible to get if you only know Linux and not Microsoft Windows, and last I saw, it did still require that you knew how to debug IRQ conflicts—long after ISA buses were limited to PC104 systems. PCI-DSS audits routinely, though not universally, mandate counterproductive security practices like installing virus scanners on Linux servers. If we'd effectively standardized professional software development in 01960 we'd still be programming in COBOL today. Moreover, the Board of Licensure could revoke our licenses when we criticized widespread bad software development practices, as has happened to Charles Marohn of Strong Towns: https://www.strongtowns.org/journal/2021/5/23/lawsuit https://www.strongtowns.org/journal/2021/5/23/lawsuit — ⁂ — You may not have commemorated it, but on March 1 we passed the 32nd anniversary of the Secret Service raiding Steve Jackson Games for publishing a "manual for computer crime", namely, the Cyberpunk module/game for the GURPS role-playing game/system. Outside the US the situation is often even worse; Saeed Malekpour was sentenced to death in 02010 because he wrote open-source image-uploading software and a porn site used it without his knowledge. (His sentence was suspended; he escaped in 02019.) Do you really think it would be a good idea for the governments that chose those prosecutors to decide who gets to program and how? — ⁂ — The bigger issue is that computers are now our exocortices; we've offloaded significant amounts of our thinking and communication onto them, as we did previously with inventions like books and clocks. Computers vastly extend our mental abilities. Without control over our computers—without the legal and practical ability to program them and to investigate what they are programmed to do—we in effect lose our freedom of speech, association, and thought. That power is far too dangerous to allow any guild or faction to monopolize it.
chrismcb 5y ago
I think you are looking at this the wrong way. the test isn't too find who can write software that performs under load. It is too find out who can write software, period. Essentially the fizzbuzz test. There are a ton of people who say they can write code, but then can't write a simple for loop.

juancn 5y ago

I mean, you don't need a really hard test to weed out most hacks. In any case, if the test is hard, companies that know their shit in hiring, care more about how you approach it than what the end result is. In the end, the interview tries to figure out how it's like to work with you. Do we get along? Can we communicate effectively? Will you be happy and thrive here?

nouveaux 5y ago
You sound like a great hiring manager/interviewer. However, I have read stories of people who got rejected only because they did not finish the coding challenge in time. The engineer reported that the candidate communicated well, but did not finish. Hiring manager rejects the candidate.
- juancn 5y ago
  Yeah, there are bad employers and bad interviewers. Sometimes the importance of it is underplayed. I always drill others that interview duty is the most important thing you can do at a company (even though it sometimes feels like a chore). The people you help choose, influences culture and long term viability of the company. Take it seriously and be fair with candidates. Having said that, figuring out a fair and effective interview process is HARD. It requires constant tweaking and monitoring. Also, you have to start from a decent set of values at the company. For example, we do shadow interviews. When we certify interviewers, we have them come to watch and listen to an interview (we explain to the candidate beforehand what's going on), and then after the interview ends, we meet with the shadow interviewer and discuss what they picked up, and explain why I went such and such way, what I was looking for, why such and such mistake is not a big deal, and so on. After a couple of these, we have them interview, but with a certified interviewer shadowing them. Just observing (we have them practice on us their interview exercises if any before the interview). And the we give feedback, you rushed a bit here, give them more time to be at ease with you, did you notice such and such, whatever... If everything goes well, we certify them, and they can fly solo. We try to have a varied pool of interviewers to avoid biases. The downside is that you need to do more interviews as a candidate, to get a fair assessment (i.e. no single person can derail you). In the end, we've all been in a crappy interviews and try our best to not inflict that on other people.

ryandrake 5y ago

Oops, I posted my rant[1] about our need for a bar exam before realizing you got there first. Fully agree, this profession really needs this. 1: https://news.ycombinator.com/item?id=30616361 https://news.ycombinator.com/item?id=30616361

Apocryphon 5y ago

The thing about the license is that it avoids the redundancy of having to pass the same orthogonal algorithm questions at every single company an applicant chooses to apply at. That alone is a great time-saver, and allows applicants the freedom to focus on actual work.

jxidjhdhdhdhfhf 5y ago
Not really. You'll still have interviews, they'll just be more personality and resume-driven. Personally, I'll take writing FizzBuzz over and over again vs. "where do you see yourself in 5 years?" ("I don't know but probably not working here") and "why do you want to work for our company?" ("to earn money").
- Apocryphon 5y ago
  Most interview loops include Leetcode for the technical screen, then at least one domain-specific coding section (like a more tense pair programming section) during the on-site, and maybe another algorithms section on the on-site (several if you’re Google). The license would at least eliminate the algorithms section to focus on the domain-specific, which is at least more reflective of real-world work.
  - jxidjhdhdhdhfhf 5y ago
    You're assuming they will actually eliminate it though rather than just pick something else to test.
    - Apocryphon 5y ago
      Sure, but at least that might be something more relevant to day to day work, and require less grinding than Leetcode.
  - jxidjhdhdhdhfhf 5y ago
    Also, its a lot easier to study leetcode problems than it is to study for domain-specific ones (since there are many different domains). I would expect having more domain-specific problems would limit you to a narrower range of jobs.

lijogdfljk 5y ago

I've had moderate success discussing Rust (a language we hire for) at moderately low levels. Ie, i've had the thought that it's difficult to bullshit topics with any depth. Even basic questions, why isn't an Rc safe to send to a thread? How does an Rc work? What is the difference between an Rc and an Arc? etc I'm happy to accept "i don't know", those are valuable too. But the whole conversation process led me to believe a lot about the candidate. We've hired 10+ juniors and seniors with this process and i've not felt cheated yet. I still feel like i need to find a way to open up people who "freeze". I do that myself, so i can relate, and my strategy of interviewing is pretty 1:1. Still though, i've had more faith in it than live/take-home tests.

AnimalMuppet 5y ago

> ... they all test very shallow knowledge that doesn't relate to ability to produce software that performs under load. That's a later filter. The first filter is, can they code at all. Like, FizzBuzz level of "at all". I mean, if you need them to produce code that performs under load, you'd better interview for that too. But people tend to first worry about whether they can clear the lowest bar before they proceed further.

sokoloff 5y ago
It is difficult to comprehend how low-skilled the shallow-end of the applicant pool is. I think it’s gotten worse, not better, since the original FizzBuzz essay was written.

starlust2 5y ago

They actually do. The test isn't (or shouldn't be) that you understand string manipulation (or whatever) but rather that you can work through a new problem. There are many programmers who claim many years of experience but have poor problem solving skills. They can implement a CRUD form and basic tasks, but fall flat when faced with a problem they've never encountered before.

kragen 5y ago

> I'd love to see a "bar exam"/licensing board for software developers. Can't resolve IRQ conflicts? Don't know the big-O complexity of different Delaunay-triangulation algorithms? Out of date on which features Microsoft added to LINQ in different versions of C#? Don't know where the "launch debugger" button is in Android Studio? Sorry, we aren't allowed to hire you to work on Instagram's iPhone app. Better hope the police don't find out your Android has an unlocked bootloader; I hear prosecutors are really cracking down on unlicensed software development ever since the assassination. They say unregulated software development leaked his location to the Zetas, I don't know if I believe that.

commandlinefan 5y ago
Yep, that's exactly the sort of ridiculous hyperbole that gets thrown around every time somebody suggests any sort of standardization around professional software development!
- kragen 5y ago
  There's no reason to think it's hyperbolic or ridiculous. We've seen lots of sorts of standardization around professional software development, with uniformly dismal results, and lots of criminalization of experimentation by astoundingly incompetent authorities. — ⁂ — The first official standard for professional software development was COBOL, shortly followed by JOVIAL. In the 01970s the US military standardized on Ada for, among other things, their hard-real-time systems; it took about a decade of violating the standard to convert Ada into a system that was somewhat suitable for hard-real-time systems, after which an Ada-induced problem destroyed the first Ariane 5. The ACM withdrew from the SWEBOK process after it became apparent that knowledge of actual programming was going to be completely excluded; instead SWEBOK certifies that you endorse the obsolete, counterproductive management processes to which "agile development" was conceived as an antidote. (If you're familiar with the BOK for some other branch of engineering, you may be aware that, for example, the structural engineering BOK includes not only project engineering practices but also what makes structures stand up or fall down, and the civil engineering BOK includes things like engineering mechanics, strength of materials, and chemistry. Presumably this is because political decisionmakers have been able to observe the difference between successful and unsuccessful skyscrapers and dams and infer which engineers were responsible.) The A+ certification for IT work is impossible to get if you only know Linux and not Microsoft Windows, and last I saw, it did still require that you knew how to debug IRQ conflicts—long after ISA buses were limited to PC104 systems. PCI-DSS audits routinely, though not universally, mandate counterproductive security practices like installing virus scanners on Linux servers. If we'd effectively standardized professional software development in 01960 we'd still be programming in COBOL today. Moreover, the Board of Licensure could revoke our licenses when we criticized widespread bad software development practices, as has happened to Charles Marohn of Strong Towns: https://www.strongtowns.org/journal/2021/5/23/lawsuit https://www.strongtowns.org/journal/2021/5/23/lawsuit — ⁂ — You may not have commemorated it, but on March 1 we passed the 32nd anniversary of the Secret Service raiding Steve Jackson Games for publishing a "manual for computer crime", namely, the Cyberpunk module/game for the GURPS role-playing game/system. Outside the US the situation is often even worse; Saeed Malekpour was sentenced to death in 02010 because he wrote open-source image-uploading software and a porn site used it without his knowledge. (His sentence was suspended; he escaped in 02019.) Do you really think it would be a good idea for the governments that chose those prosecutors to decide who gets to program and how? — ⁂ — The bigger issue is that computers are now our exocortices; we've offloaded significant amounts of our thinking and communication onto them, as we did previously with inventions like books and clocks. Computers vastly extend our mental abilities. Without control over our computers—without the legal and practical ability to program them and to investigate what they are programmed to do—we in effect lose our freedom of speech, association, and thought. That power is far too dangerous to allow any guild or faction to monopolize it.

chrismcb 5y ago

I think you are looking at this the wrong way. the test isn't too find who can write software that performs under load. It is too find out who can write software, period. Essentially the fizzbuzz test. There are a ton of people who say they can write code, but then can't write a simple for loop.