3 ms·

It's worth noting that Triplebyte has completely pivoted since they did candidate pass/fail assessments. I started using them about a year ago (first passively

It's worth noting that Triplebyte has completely pivoted since they did candidate pass/fail assessments. I started using them about a year ago (first passively looking, then actively looking) I really enjoyed the ability to be assessed on something besides Leetcode style questions. I didn't take a job through their platform (though I did get one really strong offer), but even still, found the assessments incredibly useful, since they give you a percentile distribution of your performance for each topic-specific test. After taking their assessments, when interviewers asked me how I am at, say, Python, I could tell them I have a hard time assessing my capabilities. "But hey, I took this standardized test that says I'm in the 85th percentile, not sure how good of a metric it is" (and not mentioning that I think I'm OK at best, at Python) It's the only way I've found to get a measure of your talents compared to the rest of the field (even if it might not be reliable/useful) A lot of the companies that interview through Triplebyte also skip LC mediums because they have a different signal about your potential suitability as a candidate.

malfist 4y ago
I question the ability to boil down engineering talent into something that can be represented as a "percentile distribution" Way too much of engineering is non-quantifiable. Putting a number to someone's skills is bound to be reductive at best.
- pcthrowaway 4y ago
  Sure, I'm with you, but that doesn't change the fact that enough prospective employers ask you to rate your skills with X, Y, and Z on a scale of 1 to 10. Like honestly I might think I'm a 3 at X, but if some test that thousands of other people took tells me I'm in the 90th percentile of X users, that information is still useful to me.
  - kareemsabri 4y ago
    Seems like a great bit of information for you to have. And if it's in the 30th percentile, it may let you know to either not apply to jobs that want that skill, or work to improve.
    - pcthrowaway 4y ago
      Yes, absolutely, it gave me a pretty clear indicator of where I need improvement. One complaint I do have is that (in addition to the percentile bucket) they give you a 1-5 rating, where 4 is "senior engineer level" and 5 is something "exceptional performance, a leader in the field" But the ratings seem to fall at different percentile distributions for each test. For example, I might get 80th percentile on one test, but get a 3 rating, and for another test, 80th percentile is a 5.
      - rachofsunshine 4y ago
        This is intentional. The score thresholds are set at absolute levels of skill, not by percentile - otherwise they would drift along with the population of people taking the quizzes. In general, different quizzes have vastly different populations of people attempting them. For example, our front-end quiz gets a lot of beginners and hobbyists, and thus has a very bottom-loaded score distribution. Our devops-related quizzes, on the other hand, have a population that skews skilled and senior, and has a very top-loaded score distribution. Communicating this information to our users (particularly the less-quantitatively-oriented ones on the company side) has been a source of considerable UI challenges for us.
        bb010g 4y ago
        I'd like to say that, as a developer applying for jobs, I thought your UI and documentation for this was all very well done and clear when I used it at the end of 2021. Your knowledge base was especially nice & useful.
        rachofsunshine 4y ago
        I'll pass that along to our terribly talented UX writer, who pretty much single-handedly maintains it.
        pcthrowaway 4y ago
        I'm interested in knowing how you determine what an "absolute level of skill" is though. Like, there's a nearly endless number of things to ask about front-end, and it's rapidly evolving tech. I don't know if there's a person alive who knows the ins and outs of every language, protocol, and api available on the front-end. Personally speaking, I've used Python a handful of times over the years, but never as a primary language for any work I've done. I got a 4 on the Python test. Compared to front-end, which I've been using professionally, and also dabbling in for ~20 years (still keeping up with developments in the years in which I wasn't primarily doing front-end dev professionally) I got a 3. I definitely know 100X as many random facts about front-end APIs, libraries, tooling, and technologies than I do about Python. So perhaps it just came down to luck (guessed unlucky for the front-end and lucky for Python). Or perhaps there's just so much more to know that falls in scope for the front-end quiz than there is for Python, to the point where you can spend 20 years learning the front-end technologies and still be "middle-of-the-road" in terms of "absolute level of skill". But I think that makes your descriptions of the 1-5 rankings a bit disingenuous. If people who has (what most other companies would consider) senior-level knowledge is generally considered a 3 by your system, a more honest ranking of the descriptions would involve changing "4: level expected of seniors" to "4: knows roughly ~80% or more of all things there are to know about this subject".
- kareemsabri 4y ago
  The test is not purporting to be a percentile distribute of engineering talent, it's a percentile distribution on that specific test. Certainly it's reductive, but if it's highly correlated with engineering talent it's a useful signal. I don't really see the problem, and the alternative (every company come up with their own subjective assessment) doesn't seem much better.
- rachofsunshine 4y ago
  No doubt it is reductive. It kind of has to be - real people are just way too high-dimensional for a single score. But it does correlate pretty strongly with success at onsites. We have a lot of hard data from the ATSes of companies that hire through us, and engineers with high scores on our assessments pass onsites at several times the rate of engineers without them (~2x onsite pass rate, ~7x total hire rate, which is a huge delta!). Speaking as someone who doesn't like being reductive, I've had to make my peace with the fact that "flawed" can still be "way better than the status quo". And I really do think we do much, much better than the status quo of companies putting a bunch of top schools in as linkedin keywords. If you're familiar with data science a bit - think of it as trying to project out the first couple principal components of your skills. It won't account for the whole data set, but you can go a long way with just those first couple components.

malfist 4y ago

I question the ability to boil down engineering talent into something that can be represented as a "percentile distribution" Way too much of engineering is non-quantifiable. Putting a number to someone's skills is bound to be reductive at best.

pcthrowaway 4y ago
Sure, I'm with you, but that doesn't change the fact that enough prospective employers ask you to rate your skills with X, Y, and Z on a scale of 1 to 10. Like honestly I might think I'm a 3 at X, but if some test that thousands of other people took tells me I'm in the 90th percentile of X users, that information is still useful to me.
- kareemsabri 4y ago
  Seems like a great bit of information for you to have. And if it's in the 30th percentile, it may let you know to either not apply to jobs that want that skill, or work to improve.
  - pcthrowaway 4y ago
    Yes, absolutely, it gave me a pretty clear indicator of where I need improvement. One complaint I do have is that (in addition to the percentile bucket) they give you a 1-5 rating, where 4 is "senior engineer level" and 5 is something "exceptional performance, a leader in the field" But the ratings seem to fall at different percentile distributions for each test. For example, I might get 80th percentile on one test, but get a 3 rating, and for another test, 80th percentile is a 5.
    - rachofsunshine 4y ago
      This is intentional. The score thresholds are set at absolute levels of skill, not by percentile - otherwise they would drift along with the population of people taking the quizzes. In general, different quizzes have vastly different populations of people attempting them. For example, our front-end quiz gets a lot of beginners and hobbyists, and thus has a very bottom-loaded score distribution. Our devops-related quizzes, on the other hand, have a population that skews skilled and senior, and has a very top-loaded score distribution. Communicating this information to our users (particularly the less-quantitatively-oriented ones on the company side) has been a source of considerable UI challenges for us.
      - bb010g 4y ago
        I'd like to say that, as a developer applying for jobs, I thought your UI and documentation for this was all very well done and clear when I used it at the end of 2021. Your knowledge base was especially nice & useful.
        rachofsunshine 4y ago
        I'll pass that along to our terribly talented UX writer, who pretty much single-handedly maintains it.
      - pcthrowaway 4y ago
        I'm interested in knowing how you determine what an "absolute level of skill" is though. Like, there's a nearly endless number of things to ask about front-end, and it's rapidly evolving tech. I don't know if there's a person alive who knows the ins and outs of every language, protocol, and api available on the front-end. Personally speaking, I've used Python a handful of times over the years, but never as a primary language for any work I've done. I got a 4 on the Python test. Compared to front-end, which I've been using professionally, and also dabbling in for ~20 years (still keeping up with developments in the years in which I wasn't primarily doing front-end dev professionally) I got a 3. I definitely know 100X as many random facts about front-end APIs, libraries, tooling, and technologies than I do about Python. So perhaps it just came down to luck (guessed unlucky for the front-end and lucky for Python). Or perhaps there's just so much more to know that falls in scope for the front-end quiz than there is for Python, to the point where you can spend 20 years learning the front-end technologies and still be "middle-of-the-road" in terms of "absolute level of skill". But I think that makes your descriptions of the 1-5 rankings a bit disingenuous. If people who has (what most other companies would consider) senior-level knowledge is generally considered a 3 by your system, a more honest ranking of the descriptions would involve changing "4: level expected of seniors" to "4: knows roughly ~80% or more of all things there are to know about this subject".
kareemsabri 4y ago
The test is not purporting to be a percentile distribute of engineering talent, it's a percentile distribution on that specific test. Certainly it's reductive, but if it's highly correlated with engineering talent it's a useful signal. I don't really see the problem, and the alternative (every company come up with their own subjective assessment) doesn't seem much better.
rachofsunshine 4y ago
No doubt it is reductive. It kind of has to be - real people are just way too high-dimensional for a single score. But it does correlate pretty strongly with success at onsites. We have a lot of hard data from the ATSes of companies that hire through us, and engineers with high scores on our assessments pass onsites at several times the rate of engineers without them (~2x onsite pass rate, ~7x total hire rate, which is a huge delta!). Speaking as someone who doesn't like being reductive, I've had to make my peace with the fact that "flawed" can still be "way better than the status quo". And I really do think we do much, much better than the status quo of companies putting a bunch of top schools in as linkedin keywords. If you're familiar with data science a bit - think of it as trying to project out the first couple principal components of your skills. It won't account for the whole data set, but you can go a long way with just those first couple components.