12 ms·

Tell HN: Triplebyte is, yet again, making user profiles public without consent?

610 points by teraflop 4y ago

Triplebyte (YC S15) is a tech recruiting company that operates by getting developers to take skill tests, and then using the results to match them with employers. Back in 2020, they got in a lot of hot water by suddenly announcing that user profiles -- which had been collected with assurances that the data wouldn't be shared without consent -- would be made public, unless you opted out within a week[1]. This provoked a lot of backlash, especially since the CEO seemed totally oblivious to the privacy concerns[2]. After a lot of angry comments, he publicly apologized and reversed course[3]. Then in 2021, some users started once again being notified that their profiles were automatically being made public[4]. This time, it was explained away as an "oversight" related to the fact that previously, opt-outs weren't permanent but had a hidden expiration time. Triplebyte once again apologized and promised that it wouldn't happen again, and many people seemed satisfied with the "transparency and candidness" of their response. Now it's 2022, and yesterday I got a recruiting email from a company that found me via the Triplebyte account I created back in 2019. When I logged in to check, sure enough, my profile was set to "publicly visible" and "open to new opportunities". I was pretty sure I had never made those changes, but just in case I was misremembering, I contacted Triplebyte support to find out what was going on. Today I got this response: "I was able to do some digging on to why this must have happened, It looks like before we did our last update to the platform you did not have the profile visibility set to indefinitely so the profile was turned on. Since then we have made a privacy chance once you set the profile to off there is not reset time frame it will remain off until you turn it on." (Unlike the user in [4], I never got any kind of notification that this automatic change was being made.) So despite their explicit promises, Triplebyte did not actually go back and fix the privacy settings for users who had them silently changed by the previous "dark pattern". This is a heads-up to anyone else who has a Triplebyte account and might be affected by the same issue. [1]: https://news.ycombinator.com/item?id=23279837 [2]: https://news.ycombinator.com/item?id=23280120 [3]: https://news.ycombinator.com/item?id=23303037 [4]: https://news.ycombinator.com/item?id=27255742

comboy 4y ago
Free publicity. I would love to hear some ideas for fixing this or maybe some inside stories but in my mind this kind of behavior has close to zero negative consequences for the company, even if it gets loud.
jb1991 4y ago
I've come to believe that there is a Murphy's law equivalent for tech recruiting. "If you can be abused as a technical candidate, you will be." It's just going to happen eventually that recruiters, either individual recruiters or recruitment firms/companies as a whole, will abuse your rights to just about everything or treat you with disrespect at some point or another. Developers are just commodities for these guys now.
- gnicholas 4y ago
  Not just technical candidates — I saw garbage like this when I was a lawyer, also.
  - jonny_eh 4y ago
    To recruiters it's just a numbers game. No different than sales in any other industry.
    - duxup 4y ago
      I was impressed by your resume and think you'd be perfect for <irrelevant job>. +1 Next...
- azemetre 4y ago
  IDK when it recently started but I've been getting direct messages from various recruiters to an email that I only make public on certain open source packages. As a result I remove all instances of email and just mark every recruiter that I never reached out to as spam. It's just so disrespectful, as you said.
  - chirau 4y ago
    What is disrespectful? Their job? I think it only becomes disrespectful when they keep nagging after you have told them off or something. Otherwise, they are just doing their job, seeking out good candidates. It's not like all of them are conniving to swamp you, they act individually and what you eventually hate is the sum of those individual parts.
Hellion 4y ago
I went through their process, passed in their top cohort (or so they told me, probably it was to massage my ego). And then they dropped that public profile bomb the first time around. I’d say I cost them a lot of money when I deleted my account, but they actually cost it themselves. They could have asked me if I wanted a fancy jacket, because that ended up immediately getting recycled. The book was dece tho Anyways, I ended up getting a job that I love the old fashioned way - through a friend. Great concept, shit company. Edit: Triplebyte employees reading this: you massively betrayed my trust. Had you asked me, and let me review on my own time, I might have even been proud to have a verified skill set badge page I could link to. Instead, I will probably look directly for your competitors on my next job search. You had a great idea, too bad that you seem to be irresponsible.
- jonny_eh 4y ago
  > Great concept I'm not sure it is.
  - Hellion 4y ago
    Eliminating the same boring foo bar baz coding test is a great concept. Being able to skip an hour or two out of a full interview across many companies is a great concept. Having a 3rd party verified that allows you to be quickly matched with potential employers is a great concept. Execution may be difficult, or impossible even. But that’s not an issue, when said company insists on shooting itself in the foot constantly.
- akagusu 4y ago
  > old fashioned way - through a friend I wish more companies use this method.
  - lesuorac 4y ago
    I think Nepotism generally causes legal issues for companies. https://en.wikipedia.org/wiki/Nepotism https://en.wikipedia.org/wiki/Nepotism
TechBro8615 4y ago
Good, hopefully they’ll find the image I left for them in my avatar the last time they did this.
KerryJones 4y ago
Triplebyte has given me my favorite interview to date. I love the concept, I had some interesting chats with Uber and Apple, but didn't end up going anywhere (they wanted me for positions I wasn't interested in -- currently at a different FAANG). The repeated fails on execution is pretty disheartening.
ravenstine 4y ago
I totally forgot about Triplebyte. Are they even relevant still? I remember back when it seemed like their ads were appearing everywhere and was a bit worried they were going to be the new way of hiring engineering talent. Seems like there's been nothing but crickets chirping for the last few years. Why? My experience with them was pretty bad. I took their assessment for web development, I think I even did an assignment, and got put on a video call with someone from Triplebyte. He never cracked a smile. Suddenly I got asked a bunch of CS questions that really were not very relevant to web development, some of which were entirely inappropriate like sorting a binary search tree. I even told the guy that I thought I was getting those questions wrong and he just scowled and said "well you just don't know when you're going to use this stuff." "My point exactly," I thought. Ultimately I got rejected. The whole idea that you can boil down a candidate to some coding challenges and a video quiz is bad. I do like the idea of streamlining the hiring process for developers, but there's more to it than knowing a bunch of stuff, because that can be gamed. And quizzing me on irrelevant material was a bad move. A firm like Triplebyte won't be as good at interviewing a candidate as the employer itself, and may even keep perfectly qualified candidates out of view from all employers affiliated with them.
- KerryJones 4y ago
  Wow, that sounds pretty bad. When I went through they did 4 different interviews (a little bit fuzzy; many years ago) but it did: 1) Create an incremental game (start simple, see how far you can get) 2) Live debugging (can run tests, they fail, you need to figure out why and fix it) 3) Flash rounds (Do you know what Big O is? Can you explain linked lists?) 4) ... I forget I thought it was one of the widest range of actual skills and their final assessment I agreed with. Stayed away from algorithms-y questions (which I hate)
- gcampos 4y ago
  My impression is that they are still around, but they failed to deliver the "recruiting revolution" and I think the reasons are: - The screening had a lot of false negatives. "I got rejected by Triplebyte, but got a FAANG offer" is quite common. - Most companies used Triplebyte not as an interview replacement, but as an additional screening process, which means that as a candidate, you don't have any real incentive to use them. The only real use case I heard recently about Triplebyte is to send candidate who normally you wouldn't even screen, so if they pass Triplebyte process, you know that you should consider the candidate, but if they fail is fine because you would have passed them anyways
throwaway889900 4y ago
Just delete your account this time.
- zug_zug 4y ago
  That's what I did last time :D
z9znz 4y ago
Whether they are malicious or not, I cannot say. But anecdotally, are they a time-waster? Absolutely. I took their generalist test and scored perfect. It was actually not so hard, but it tested some language things (a couple/few languages), some Unix things, SQL, etc. It was right up my alley. After completing the 30? min test, I got an email saying they were enthusiastic about me and would like to continue the process. I think the process was to become an interviewer; I forget now. Time passed... weeks, and then months. I emailed them and got nothing back. Then I checked Glassdoor (YMMV). There were a lot of negative stories about internal politics, gender issues, leadership problems, "insane" bad manager stories, etc. I don't know how Glassdoor works, but perhaps it was all lies; or perhaps it was very true; perhaps it has been erased (pay to play). Nonetheless, I parked them in the corner of my mind behind Toptal.
- pcthrowaway 4y ago
  > Then I checked Glassdoor (YMMV). There were a lot of negative stories about internal politics, gender issues, leadership problems, "insane" bad manager stories, etc. I don't know how Glassdoor works, but perhaps it was all lies; or perhaps it was very true; perhaps it has been erased (pay to play). Are we looking at the same Triplebyte? Their reviews on Glassdoor are pretty good edit: There are some nasty ones further back, the latest being Feb 2020
  - scarmig 4y ago
    Glassdoor isn't really trustworthy: companies can pay it to nuke bad reviews.
  - z9znz 4y ago
    Another thing we can do, especially on HN (but also for those of you who may be in Cali where you actually meet a lot of humans in this space) is to ask, "Have I ever met a happy Toptal consultant?" Has anyone here on HN?
    - pcthrowaway 4y ago
      I'm not really sure where Toptal fits into this. Triplebyte is not a consultant agency
greatjack613 4y ago
Triplebyte had a great model in the beginning which was focused on getting people who had skills without formal certifications jobs. That did not make enough money so the founders pivoted like 5 times to try other things and each time they made Triplebyte more and more anti candidate. Keep on going Ammon, we all know where these companies end.
econnors 4y ago
FWIW, my profile is still set to non-visible, so it's not a universal change.
- teraflop 4y ago
  Good to know. Giving them the benefit of the doubt, I can believe that at some point they fixed it so that setting your profile to invisible took effect permanently, but they never went back and retroactively applied that change to people who had previously opted out (or never opted in). Or if they did, they must have somehow skipped my account. I responded to them with a link to this thread in case they wanted to publicly comment or correct anything, but I'm not holding my breath.
  - rachofsunshine 4y ago
    We intended to make that change retroactive. I'm looking into it with our engineering team as we speak. EDIT: I can confirm that that fix was in fact retroactive. We'll have a response to OP momentarily. (EDIT2: Ammon, our CEO, has a top-level reply to OP.)
swayvil 4y ago
Tickling the fear of the day is a great way to get media attention. The obvious way is wagging your fist at all those bad guys doing bad stuff. But doing bad stuff is actually a pretty good way too. If you can manage the backlash. Works great. I mean, it worked here. Right? The last time I thought about triplebyte was the last time they did something bad. Kids do it all the time to their parents. Very hard to ignore. Something to think about.
- quickthrower2 4y ago
  Good point. A lot of candidates wont care. Like all the crazy stuff Uber did probably didn’t bother most drivers or riders. And each bad publicity is still publicity.
muh_gradle 4y ago
Ok, I"m deleting my profile. Thank you for the heads up. Well done Ammon Bartram. Great company practices.
throwaway81523 4y ago
Oh man, not again. My initial experience with triplebyte was quite good, when they did a high touch testing and interview process. It worked pretty well (at least for me) but I guess it was too expensive for them to keep doing. Later there was a glitch where they hosed me rather badly, but I could treat that as a one-off. What is it with this repeatedly opening up profiles though? I thought the saying was that it was ok to make mistakes, but.... make new ones.
MrWiffles 4y ago
Thanks op for the news. Can anyone recommend alternatives for the future? It's clear they're untrustworthy so I'm hesitant to even consider them again even though I passed years ago (2019 I think).
clpm4j 4y ago
I've come to a personal conclusion that hiring is an unsolvable "problem". When the problem and the product essentially distill to people - many individuals - it unravels. People are unique. There is never a broad, one-size-fits-all solution. Companies are just collections of people. There are way too many variables to ever reach the level of consistency that a product/solution needs to provide in order to retain customers and grow into a large, sustainable business.
rachofsunshine 4y ago
Hi everyone - I'm the head of product here at Triplebyte. We did not make any intentional change to how profile visibility functions and (to the best of my knowledge) the issue referred to in OP's support response (and mentioned in the second half of OP's post) was fixed last year. (See my comment at the top of OP's link [4] for more from then.) We'll have a more complete answer shortly. EDIT: This does not appear to be a widespread issue. Continuing to investigate. EDIT2: Full response from Ammon, our CEO, at https://news.ycombinator.com/item?id=31771836 https://news.ycombinator.com/item?id=31771836
- Hellion 4y ago
  Edit: it looks like I was wrong about some things. The team at Triplebyte has also answered my questions. In the interest of fairness, if you read this post, I encourage you to read their responses as well. ————- At this point, if you (Triplebyte) haven’t set private the profiles of everyone who didn’t explicitly consent, then you are irredeemable. You’ve had, what, two years to work this out? I would love to understand what the goals and numbers here are, and what the revenue implications of public profiles are? It seems to me that you’re trying to build a social network of some sort with verified-skill capability. I went to your site, and your about me page, to see what your business model was. What you stated, the mission and the copy, does not align with a public profile and social network style feature. If you truly are trying to make hiring better, how does public profiles achieve that? If you truly are trying to make hiring better, people on all sides of process should be benefiting. And there should be nothing to be afraid of in terms of mission. Your /manifesto document falls short. What, exactly, is your business model? You should have no shame in it. Unless you think it’s shameful, in which case, please stop. Like your manifesto claims, transparency is good!
  - ammon 4y ago
    As of 2 years ago, every profile on Triplebyte is not visible to recruiters unless an engineer explicitly makes it visible.
  - rachofsunshine 4y ago
    So, first off: our support response was incorrect. OP had in fact enabled visibility manually some time ago. But to reply to your questions: ------ > I would love to understand what the goals and numbers here are Well, zero, since nothing was broken in this case. The issue last year had affected a couple hundred people (out of a total pool in the hundreds of thousands) before it was caught and reverted; the worst-affected person was erroneously visible (to Triplebyte companies only) for about a week and a half (and was hidden again within hours of us knowing about it). > It seems to me that you’re trying to build a social network of some sort with verified-skill capability. More or less. Not a social network, necessarily, but a platform of engineers with skill data from our assessments. In brief, our goal is to make the job search process lower-grind, higher-signal, and more fair for candidates by exposing more information about companies, targeting messages more intelligently, and relying on skills over resumes. And on the company side, our goal is to provide them better and more reliable information than they can get when trying to hire elsewhere. Everyone hates the status quo of a thousand applicants or a thousand outbound messages per conversation started, it's just a hard status quo to disrupt as an individual candidate or recruiter. > If you truly are trying to make hiring better, how does public profiles achieve that? To be clear, we do *not* have public profiles in the sense of "visible to the internet as a whole". We have profiles that are or aren't visible to our (company-side) subscribers, according to each engineers' settings; a "visible" profile is visible to those subscribers and an invisible one is not. The only way a profile can currently be exposed to the world at large is if an engineer on Triplebyte explicitly enables a public link and posts it elsewhere. (Even then, I don't think they're indexable? I'm not 100% sure about that bit; this is an old feature I haven't touched during my tenure.) > What, exactly, is your business model? "Be a place where the best engineers want to look for jobs, because it's a better and fairer experience, and charge companies for access to those job seekers and for the tools we give them to surface the best matches for the role they're hiring for." In a literal financial sense, we charge a subscription fee to employers for access to our candidate database ("visible" profiles in the previous section's sense only).
rykuno 4y ago
I wish this company succeeded with its mission, but it turned into a variant of Hackerrank I feel. I hate leetcode/hackerrank type interviews. On the hiring side of things where the company I work for screens candidates this way; it sucks. I would have loved to see a test of more practical programming experience. Candidates spend their time on stupid coding questions all day instead of actually coding something useful or that they can learn from. I have a relative doing exactly this. No idea how to build a simple RESTful API, yet spends all his time on Hackerrank posting on linkedIn how he's in the top 'X' percent. When they get hired and put on a "real world" project they are absolutely lost :(. I also tried Triplebyte after seeing ads on Reddit. Passed a few tests and nothing really ever came of it other than an email.
- itsdrewmiller 4y ago
  To me a big part of the appeal to Triplebyte is that their variant is one that does reflect real world programming. Their general coding quiz shows you written code and asks you to find the bug or fill in the blank - things very similar to day to day activities when working on an existing application. AFAIK there isn't anything about tricky algorithms, either.
VirusNewbie 4y ago
I really like the idea of Triplebyte but it did not seem to be worth the effort. If it could get you past multiple tech screens at big companies it would be great. I don't really care that it saves me an hour of doing an easy coding challenge at some no name company...
TheMagicHorsey 4y ago
What are some other alternatives to Triplebyte for employers? I mean skills-based filtration systems. I often post on different forums and ask people if they want jobs, but no matter how esoteric and nuanced the forum I post to, I always seem to get a couple dozen candidates that have very rudimentary skills and end up wasting my time on zoom interviews. I'd like to be able to access a pool of candidates that I know have a certain level of expertise ... even if they have no formal training or poor academic scores.
ammon 4y ago
I’m the CEO of Triplebyte. I just looked into this, and we are not automatically making anyone's profile public without their consent. OP, it looks like you had set your profile to be visible some time ago (I can provide details via email but won’t here for privacy purposes), and just hadn’t gotten a message until now. The support response you received was incorrect; you weren’t affected by any recent changes to the site. Since there wasn’t anything broken on our end, there shouldn’t be anyone else impacted by this. But as part of making sure that OP wasn’t visible by mistake, I had my team double-checked to make sure our previous fix from last year was properly retroactive. It was. More generally, I don't think that tricking anyone is a viable way for us to run a business. We’re trying to create a marketplace that can open opportunities for engineers who wouldn’t otherwise have them, and we need the trust of engineers in order to do that.
- mettamage 4y ago
  > We’re trying to create a marketplace that can open opportunities for engineers who wouldn’t otherwise have them Provided they're in the US. My experience as a European has been: more or less everything I apply to ignores my application. I haven't checked recently, is it changing? I'm not implying you're wrong since you used the word trying to create. Moreover, you didn't specifically specify which group of engineers you're trying to create opportunities for. I don't even want to go in a right or wrong type of frame, because it doesn't matter, but my lack of eloquence might give that impression. What I am implying is that the statement is a bit broad. On a more emotional (perhaps even non-rational) note: I feel spoken to yet left out, for years now.
  - ammon 4y ago
    Most employers on Triplebyte (90%) are hiring in the US currently. However, we're working on something that us going to change that (announcement coming soon).
    - pcthrowaway 4y ago
      As a Canadian, it seemed like at least 30% of remote employers were open to hiring from Canada (although it was often marked incorrectly in your system). At the very least, a couple of U.S. companies with remote positions that claimed to be U.S. only (or "work authorization required?"), reached out to me.
egberts1 4y ago
so glad that i used a pseudonym on Triplebyte. I knew better.
aasasd 4y ago
I mean, having the hidden expiration time for the ‘non-public’ status should've been more than enough to illustrate the total lack of respect for the users.
raxxorraxor 4y ago
I am pretty sure Google leeched my contacts from my phone in a similar way once. There need to be hard penalties here since the company cannot undo the damage. It might just have been a bug on their end, but it is fairly possible implement data sharing in a way that errs on caution. Especially for a company that allegedly values private information.

clairity 4y ago

> "The whole idea that you can boil down a candidate to some coding challenges and a video quiz is bad." yes, there are too many variables between the candidate, job, company, and work environment to determine long-term fit via a test, especially for "creative" jobs. the more regimented the job (e.g., fast food cook), the lower the variability, but it's still significant. plus, such tests only evaluate technical skills, not the more important non-technical ones (like punctuality, integrity, steadfastness, etc.--note that these are a function of the involved parties and the relationship between them, not just the candidate). but also, the underlying problem of hiring is not one of trying to get the best fit, but of trying to avoid the pain of firing. that's the thing that needs to be reframed/solved, but that's a much harder and a much less technical problem (alternatively put, technical tests are marginal at best).

datavirtue 4y ago
Can a quiz or trivia determine how good a person is at the most crucial aspect of the job: discovering what needs done? I spend weeks drumming up two or three days worth of coding work. The coding aspect is basically manual labor and pedantic arguments with other devs.
- throwaway81523 4y ago
  I felt that the live interview was fairly ok at that. A staff engineer watched you code and debug stuff, asked you how you would do XYZ, etc. This took a few hours so must have cost the company $100s in engineering time. I can see why they don't do it any more. There's not much more they can do in a compressed time frame. Lately a bunch of companies want you to do unpaid multi-day take-home assignments before they even talk to you, but that is nuts. An alternative could be a paid brief engagement, like an NBA 10-day contract. I've done some things like that, which worked ok.
  - clairity 4y ago
    interviews are good for weeding out the absolute no's and that's about as good as you can get with those. a live coding session gives you marginally more information in that regard, but won't give you definitively more. a 10-day contract is better, since it's real work, for pay, but the relatively short duration doesn't tell you much about the candidate's intrinsic motivation or how relationships develop past the honeymoon stage. so really, it'd be best and easiest if we all explicitly assumed that jobs had 6-12 month trial periods, for both parties, and that after that time, either can walk away without hard feelings (or negative judgment), other than in the most egregious cases (i've seen a couple cases that'd fall in this category). again, this is primarily about jobs that have the most variability. less variable jobs don't need as long of an evaluation period (but do need more than a few weeks).
- clairity 4y ago
  that's the billion dollar question. literally, if you can solve that problem, then it's worth billions, as you'd be able to replace the indeeds of the world. i'd strongly contend that no one has even come close to actually solving this problem, which is probably in firmly the realm of P=NP. my (now failed) startup in this space attempted to answer it for less variable jobs (hourly work), where we could tease out more signal from the noise, but even that had lots of unaccountable variability.

pcthrowaway 4y ago

It's worth noting that Triplebyte has completely pivoted since they did candidate pass/fail assessments. I started using them about a year ago (first passively looking, then actively looking) I really enjoyed the ability to be assessed on something besides Leetcode style questions. I didn't take a job through their platform (though I did get one really strong offer), but even still, found the assessments incredibly useful, since they give you a percentile distribution of your performance for each topic-specific test. After taking their assessments, when interviewers asked me how I am at, say, Python, I could tell them I have a hard time assessing my capabilities. "But hey, I took this standardized test that says I'm in the 85th percentile, not sure how good of a metric it is" (and not mentioning that I think I'm OK at best, at Python) It's the only way I've found to get a measure of your talents compared to the rest of the field (even if it might not be reliable/useful) A lot of the companies that interview through Triplebyte also skip LC mediums because they have a different signal about your potential suitability as a candidate.

malfist 4y ago
I question the ability to boil down engineering talent into something that can be represented as a "percentile distribution" Way too much of engineering is non-quantifiable. Putting a number to someone's skills is bound to be reductive at best.
- pcthrowaway 4y ago
  Sure, I'm with you, but that doesn't change the fact that enough prospective employers ask you to rate your skills with X, Y, and Z on a scale of 1 to 10. Like honestly I might think I'm a 3 at X, but if some test that thousands of other people took tells me I'm in the 90th percentile of X users, that information is still useful to me.
  - kareemsabri 4y ago
    Seems like a great bit of information for you to have. And if it's in the 30th percentile, it may let you know to either not apply to jobs that want that skill, or work to improve.
    - pcthrowaway 4y ago
      Yes, absolutely, it gave me a pretty clear indicator of where I need improvement. One complaint I do have is that (in addition to the percentile bucket) they give you a 1-5 rating, where 4 is "senior engineer level" and 5 is something "exceptional performance, a leader in the field" But the ratings seem to fall at different percentile distributions for each test. For example, I might get 80th percentile on one test, but get a 3 rating, and for another test, 80th percentile is a 5.
      - rachofsunshine 4y ago
        This is intentional. The score thresholds are set at absolute levels of skill, not by percentile - otherwise they would drift along with the population of people taking the quizzes. In general, different quizzes have vastly different populations of people attempting them. For example, our front-end quiz gets a lot of beginners and hobbyists, and thus has a very bottom-loaded score distribution. Our devops-related quizzes, on the other hand, have a population that skews skilled and senior, and has a very top-loaded score distribution. Communicating this information to our users (particularly the less-quantitatively-oriented ones on the company side) has been a source of considerable UI challenges for us.
        bb010g 4y ago
        I'd like to say that, as a developer applying for jobs, I thought your UI and documentation for this was all very well done and clear when I used it at the end of 2021. Your knowledge base was especially nice & useful.
        rachofsunshine 4y ago
        I'll pass that along to our terribly talented UX writer, who pretty much single-handedly maintains it.
        pcthrowaway 4y ago
        I'm interested in knowing how you determine what an "absolute level of skill" is though. Like, there's a nearly endless number of things to ask about front-end, and it's rapidly evolving tech. I don't know if there's a person alive who knows the ins and outs of every language, protocol, and api available on the front-end. Personally speaking, I've used Python a handful of times over the years, but never as a primary language for any work I've done. I got a 4 on the Python test. Compared to front-end, which I've been using professionally, and also dabbling in for ~20 years (still keeping up with developments in the years in which I wasn't primarily doing front-end dev professionally) I got a 3. I definitely know 100X as many random facts about front-end APIs, libraries, tooling, and technologies than I do about Python. So perhaps it just came down to luck (guessed unlucky for the front-end and lucky for Python). Or perhaps there's just so much more to know that falls in scope for the front-end quiz than there is for Python, to the point where you can spend 20 years learning the front-end technologies and still be "middle-of-the-road" in terms of "absolute level of skill". But I think that makes your descriptions of the 1-5 rankings a bit disingenuous. If people who has (what most other companies would consider) senior-level knowledge is generally considered a 3 by your system, a more honest ranking of the descriptions would involve changing "4: level expected of seniors" to "4: knows roughly ~80% or more of all things there are to know about this subject".
- kareemsabri 4y ago
  The test is not purporting to be a percentile distribute of engineering talent, it's a percentile distribution on that specific test. Certainly it's reductive, but if it's highly correlated with engineering talent it's a useful signal. I don't really see the problem, and the alternative (every company come up with their own subjective assessment) doesn't seem much better.
- rachofsunshine 4y ago
  No doubt it is reductive. It kind of has to be - real people are just way too high-dimensional for a single score. But it does correlate pretty strongly with success at onsites. We have a lot of hard data from the ATSes of companies that hire through us, and engineers with high scores on our assessments pass onsites at several times the rate of engineers without them (~2x onsite pass rate, ~7x total hire rate, which is a huge delta!). Speaking as someone who doesn't like being reductive, I've had to make my peace with the fact that "flawed" can still be "way better than the status quo". And I really do think we do much, much better than the status quo of companies putting a bunch of top schools in as linkedin keywords. If you're familiar with data science a bit - think of it as trying to project out the first couple principal components of your skills. It won't account for the whole data set, but you can go a long way with just those first couple components.

armchairhacker 4y ago

To be fair, I think a shared interview system would be great. Then companies don’t have to devote time and effort into their own interview process, which turns out to be Leetcode and full of false positives / negatives anyways. But it needs to be: - In-depth. Not just a single exam or interview. You need to really know the employee’s strengths and weaknesses - Detailed. You can’t just give someone pass / fail or a single score. Not only is it mean, but you end up getting misaligned candidates anyways, because some people are really bad at some aspects of software but good at others. In fact maybe the process should ditch scores entirely and just show the recruiters the actual employee interviews, and what he/she has and has not accomplished. - Changing over time. Not a short period of time. But like, if I take the assessment, 6 months later I can take a smaller assessment and it will update my scores and log my progress. Triplebyte is not 1 or 2. Idk but I think it’s 3 and you can retake the quiz. But then it’s only telling employees if you’re basically competent for some arbitrary statistic, which doesn’t even tell if you’re basically competent at the company. I think it would be nice if i could take one thorough interview instead of several less-thorough company-specific interviews, but that’s not Triplebyte.

avereveard 4y ago
Idk, as a company I would still want to run my interview, and candidates would probably hate the double interview. Pre interview would also get stale fast. Companies wouldn't trust a third party to run binding technical assessment for them, and quality devs would probably avoid places that hire without having someone from the destination team show up I think the opposite would be more beneficial: a light check to validate work claims and some high level foundational question about code just to make sure one has basic proficiency in what he claims he has Then companies would need lot less hr pre-screening and could focus in technology and culture matching
- wan23 4y ago
  I'm very sure that a consortium of FANG type companies could get together and design a shared interview for generic technical skills, and then run normal interviews for candidates that pass that step. And when I say normal interviews, I mean the stuff that an individual company should want to do, such as behavioral, talking about past projects, etc. Making candidates do leetcode type sessions at every place they apply is a waste of time for both candidates and companies.
- pcthrowaway 4y ago
  I don't think the proposition (for employers) is really to be able to skip interviews altogether, but to find "diamonds in the rough", candidates who may have an exceptional grasp of material that might not be suggested by their resume or years of experience or whatever. If you can hire someone skilled, but who other companies might overlook, that's a huge benefit to your team. Triplebyte is really just the first screen, and the importance each company wants to put on the signal is up to them.

teraflop 4y ago

Thanks for responding. As far as I can determine, the last deliberate interaction I had with Triplebyte was in May 2020, when I got an email with your name and address on it saying that my profile was about to be made public, and I replied the following day asking you to delete my account. (I assume you never saw that email, and in any case I never bothered to follow up on it through other channels.) (EDIT: on closer inspection, the reply with my account deletion request went to candidate.support@triplebyte.com, not to you personally.) I don't recall ever logging into my Triplebyte account between then and yesterday, and I couldn't find any evidence of doing so in my browser or search history. I guess that's your word against mine, but if you have reason to believe I'm mistaken, you're welcome to send it to me privately. > The support response you received was incorrect I'm very curious as to where the incorrect information came from, then.

rachofsunshine 4y ago
We have no record of such an email under either the email on your account or your name (and we checked and we do have records of other emails in the same time window, including many other people who emailed us at the time for the same reason you did). I'm not sure why (have you perhaps changed your name or email since?). I'm going to have our support team get back in touch with you via email for further details here. Can you reply here if you haven't been able to do at least one round of back-and-forth with them within the next day or so?
- teraflop 4y ago
  > have you perhaps changed your name or email since? Nope. Just to show that I'm not yanking your chain, this is the email I'm referring to: https://imgur.com/2FpAiik https://imgur.com/2FpAiik The redacted name and email address are exactly the same as when I contacted you yesterday. To be clear, I'm not trying to criticize you for not taking action on this message, because for all I know it could have gotten dropped by a mail server along the way. I'm just using it to illustrate the fact that I wasn't even aware that I still had an active Triplebyte account, so I find it implausible that I logged in, set my profile to public, and then completely forgot about it. > Can you reply here if you haven't been able to do at least one round of back-and-forth with them within the next day or so? Sure, will do.
  - rachofsunshine 4y ago
    > To be clear, I'm not trying to criticize you for not taking action on this message, because for all I know it could have gotten dropped by a mail server along the way. I'm just using it to illustrate the fact that I wasn't even aware that I still had an active Triplebyte account, so I find it implausible that I logged in, set my profile to public, and then completely forgot about it. Yeah, of course. Totally fair. (Further comms via email)
teraflop 4y ago
Just following up on this, in case anyone's still reading. This is the response Triplebyte sent me yesterday: > As mentioned on HackerNews, we cannot locate any emails from you before today. After getting in touch with the product team, I realized I had made a mistake. Upon doing a bit of digging on the back end, you set your profile set to be visible in 2019 prior to the 2020 events. I guess I can't concretely disprove this story, but I have a really hard time buying it. I don't remember even being given the option to make my profile public when I tried out Triplebyte in 2019. To back up my recollection, the messages that they sent out about the "public profiles" feature in 2020 described it pretty unequivocally as a new feature that hadn't existed previously. Archived versions of Triplebyte's FAQ from 2019/2020 make no mention of it; they only talk about the ability to be matched with companies after completing a full interview with Triplebyte (which I never did). And if the person who initially responded to me truly did just make a mistake, that would certainly be no big deal -- but it seems like an oddly specific and convenient mistake to make. I responded about 24 hours ago saying why I found these explanations unconvincing, and haven't heard back. I'm posting here not to try to pressure Triplebyte into a response, but because (a) I don't know how much longer this thread will stay open to new comments, and (b) I don't really think it's likely that I'll get any further closure about this issue, so I don't plan to spend any more of my time and energy on it. People are welcome to read the discussion and judge for themselves.
- rachofsunshine 4y ago
  Checking back in a few days later: > To back up my recollection, the messages that they sent out about the "public profiles" feature in 2020 described it pretty unequivocally as a new feature that hadn't existed previously. The "public" profiles we announced (and rolled back) at the time were distinct from the visible-to-Triplebyte-subscriber-companies profiles we have today (and had in late 2019). "Visible" is doing double-duty here between public-to-the-internet (which we never ended up doing) and visible-on-our-platform (which predated the 2020 announcement).

rachofsunshine 4y ago
> I'm interested in knowing how you determine what an "absolute level of skill" is though. They're set by experts in the area, the same as the ones who write our question content. To give a little more detail: the tests run in a beta state for a while before being fully released. We gather a bunch of data and calibrate parameters for our IRT model based on that. So the ordinal ordering of performance is entirely mathematical and data driven. (When we were still doing in-house human interviews, those were part of the data set as well, and still are for the subjects that overlap them.) But that produces a continuous, hard-to-interpret, and population-dependent score distribution, and SMEs draw the lines with which we bucket those scores. (For those of you familiar with IRT as a framework, they set theta thresholds.) But yes, there is some chance involved. It's a tradeoff between the standard error in our scores and the length of the quiz, and we try to optimize for a sweet spot there (since most people don't want to take two hours of quizzes). And we are absolutely going to get it wrong sometimes. That's both for in-model reasons (the statistical standard error is enough that we we'll be off by a level either way around 20-25% of the time or something like that) and for out-of-model ones (maybe some of our questions just test the wrong thing in ways that don't show up in the data). Assuming your self-assessment is correct (and I will say that many peoples' are not! confidence correlates with skill, but with a whooooole lot of noise.) then yeah, you probably had a bad roll of the dice on one and not on the other. As I say a lot (in this thread and elsewhere), we can't reasonably bat 1.000: our goal is to bat better than the next guy. And I think we do do that, messy though the entire space can be in practice. --- For the record, when we talk to companies, here's what we tell them about scores: 2 = knows something in this area, but we can't say with confidence that they know enough to handle things independently. OK for entry-level roles, but lower than you'd like for others. We don't show a 2 on profiles. The only place companies see it is if they're using our screens to screen their own candidates via our Screen product. The idea being that if you have the choice of whether to take an assessment or not in the first place it shouldn't really hurt you to try. 3 ("Proficient") = professional competence in that area, can work independently in it. A score we'd expect of a mid-level engineer within their area of expertise. A recommendation for most roles, maybe not very senior ones (but not a point against even for senior roles). A score of 3 or above counts as certified, meaning it earns a shareable certificate and makes you appear in search results for a particular quiz score. 4 ("Advanced") = significant expertise, something more typical of a senior eng who really knows their way around the subject. A recc for all levels, even very senior ones. 5 ("Expert") = exceptional, above and beyond even by the standards of senior roles