6 ms·

I had an interesting experience with triplebyte which wasn't as objectively bad as yours, but it also makes me skeptical of the company. First round was multi

by zug_zug 10y ago

I had an interesting experience with triplebyte which wasn't as objectively bad as yours, but it also makes me skeptical of the company. First round was multiple choice questions, relatively straight-forward. Second-round was skype-call and just felt incredibly subjective. I was asked questions around building out memcached to support arbitrarily-sized values, and I got the same "smug" vibe you sensed. The interview style was very "Him: How would you do X?" "me: Well that's not a simple problem, there are a lot of solutions each with tradeoffs." "Him: Okay so name one" "Me: So you could do X" "Him: BUT THEN Y [GOTCHA!]" "Me: Yes, that's one of the tradeoffs of X" It wasn't clear to me what the heck he was even looking for. Was he hoping I'd list race-condition problems? Had he not even considered race-condition problems? Was he looking for a theoretical solution or a real-world solution? Also he kept going on random tangents ("That brings me to an interesting question, how would you shift a gigabyte of memory 1 bit?"). He seemed very concerned with efficiently bit-packing the header in this problem, which seems silly to me when we're talking about storing gigabytes. My understanding was that triplebyte was seeking to be the SATs of engineering, however SATs do heavy validation with test-retest reliability and such, I had no particular reason to suspect triplebyte's interview was any more objective than any other company's.

ammon 10y ago
We actually put a bunch of effort into consistency/repeatability checks. Every interview is recorded (video), and we re-watch and re-grade a percentage of them to measure the consistency. A long-term experiment we're running is comparing qualitative scores (code quality, good process, how good did the interviewer feel the candidate was) with quantitative features (which tests passed, how long did it take, what design--picked from a decision tree--did the candidate take). We calibrate the qualitative scores with the recorded interviews. So far, quantitative scoring is winning (when judged against predicting interview results at companies). We're waiting, however, until we can see which better predicts job success.
- sillysaurus3 10y ago
  Notably absent from that list is "Are we verifying we're doing a good job as interviewers?" It doesn't matter how good the interviewer feels the candidate is, or whether a design was picked from a decision tree. All that matters is whether the candidate can do the work at actual companies. I think people here are reacting to irrelevancies during the interview process -- questions which cannot possibly be reflective of a candidate's real-world competency. (When was the last time you shifted a gigabyte of memory? And even if you did, that's not what companies are going to employ people to do. So why ask the question? Are you sure it isn't trivia?)
  - ammon 10y ago
    Interviews absolutely should be grounded in trying to predict how a candidate would do on a job. That's the whole ballgame. The question is how to best do that. First, you need to run a repeatable process (my previews comment). Second, you need to look at the right skills. The approach we take is to track a lot, and figure out what works the best over time. What we've found to be most predictive (so far) is a base level of coding competency, plus max skill (how good an engineer is at what they are best at). So (beyond the coding portion) we don't actually care very much about what a candidate is bad at. We care about how good they are at what they are good at. To give as many candidates as possible the opportunity to show strength, we cover a number of areas. This includes back-end web development, distributed systems, debugging a large codebase, algorithms, and -- yes -- low-level systems (currency, memory, bit and bytes). We do not expect any one engineer to be strong in all of these areas (I'm weak on some of them). But they all are perfectly valid areas to show strength (and we work with companies that value each of the areas). We've recently moved to a new interview processes organized around this idea of max skill. It's working great in terms of company matching and predictive ability. However, it seems we may have underestimated the cost to candidates of being asked about areas where they are weak. There's more negative feedback here than we've seen in previous HN discussions, and I think that the interview change may be behind that. I'm taking that to heart. I think we can probably articulate it better (that we measure in a bunch of areas and look for max strength). We're also running an experiment now where we ask engineers are the start of the interview which sections they think they'll do best on. I'm excited about this. If engineers can self-identify their strongest areas, we'll be able to make the process shorter and much more pleasant! So, the bit shift question: that come up down one branch of a system design question that we used for a while (we've since moved to a more targeted version that is more repeatable). The (sub)issue involved adding a binary flag to a large data blob (this came up as part of a solution to a real-world caching problem). Adding a single bit flag to the front of a 1GB blob has a problem. To really add just one bit, you'd have to bitshift the entire 1GB. This is clearly not worth it to save 7 bits of storage (ignoring that that would not be saved in any case). You can just use a byte (or word), or add the flag at the end. When candidates suggested adding a bit flag at the front, we would follow up asking them how they'd do it (to unearth if they were using 'bit' as a shorthand for a reasonable solution, or if they really are a little weak in binary data manipulation). This was one small part of our interview. By itself it in no way determined the outcome of the interview, or even of the low-level systems section. Plenty of great engineers might get it wrong. But I don't think it was unfair.
- mbesto 10y ago
  > better predicts job success How do you rate job success?
  - ammon 10y ago
    being happy with the job and not having left after 1 year
    - mbesto 10y ago
      Wait so job success is based on the interviewee, and not the company?
- pyb 10y ago
  Wasn't there a study by Google a while back, where they found, as a trend, that their most successful people had only marginally passed their job interview ?
  - astrange 10y ago
    That sounds like the Pareto-optimal solution to a job interview. It's like how you're not doing grad school properly if your grades are better than C. Also, candidates who do too well could be too good for the job, since Google supposedly likes having incredibly overqualified people maintain do-nothing internal apps.
  - jameside 10y ago
    The finding was that people who had received a "no hire" recommendation yet still received an offer tended to do well. The reason being: to compensate for the poor feedback, someone else on the hiring loop believed in the candidate so much and saw something exceptional and were willing to go to bat for them.
- pfarnsworth 10y ago
  It sounds like your ability as an interviewer is pretty poor. There are several examples of the same smug behavior. What sort of training have you gone through to ensure that you're actually an appropriate and qualified person to be interviewing?
  - dkarapetyan 10y ago
    I interviewed with Ammon. He did not come across as smug. I went through the process just as a curiosity because I was fed up with the typical interview process. I haven't followed through anything else related to Triplebyte in terms of an actual job in case people think I have a favorable view because I got a job through them.
  - Harj 10y ago
    I'm his co-founder and this comment is unnecessarily personal. Ammon has done over 900 technical interviews (https://www.reddit.com/r/cscareerquestions/comments/5y95x6/i_am_ammon_bartram_and_i_have_done_900/ https://www.reddit.com/r/cscareerquestions/comments/5y95x6/i...) and there's one negative reference to him specifically on this thread.
    - pfarnsworth 10y ago
      Nah, you can't use forum threads as proof. You need to do the analysis scientifically, which based on the feedback, it sounds like you need to do. You don't bother quantifying the ability of you and your interviewers, you just assume you're great.
jtchang 10y ago
How would you actually bit shift 1 gigabyte worth of data? Would you start at the end and read in words? Possibly using some temp variable?
liveoneggs 10y ago
so like every other interview ever?

by zug_zug 10y ago

ammon 10y ago

We actually put a bunch of effort into consistency/repeatability checks. Every interview is recorded (video), and we re-watch and re-grade a percentage of them to measure the consistency. A long-term experiment we're running is comparing qualitative scores (code quality, good process, how good did the interviewer feel the candidate was) with quantitative features (which tests passed, how long did it take, what design--picked from a decision tree--did the candidate take). We calibrate the qualitative scores with the recorded interviews. So far, quantitative scoring is winning (when judged against predicting interview results at companies). We're waiting, however, until we can see which better predicts job success.

sillysaurus3 10y ago
Notably absent from that list is "Are we verifying we're doing a good job as interviewers?" It doesn't matter how good the interviewer feels the candidate is, or whether a design was picked from a decision tree. All that matters is whether the candidate can do the work at actual companies. I think people here are reacting to irrelevancies during the interview process -- questions which cannot possibly be reflective of a candidate's real-world competency. (When was the last time you shifted a gigabyte of memory? And even if you did, that's not what companies are going to employ people to do. So why ask the question? Are you sure it isn't trivia?)
- ammon 10y ago
  Interviews absolutely should be grounded in trying to predict how a candidate would do on a job. That's the whole ballgame. The question is how to best do that. First, you need to run a repeatable process (my previews comment). Second, you need to look at the right skills. The approach we take is to track a lot, and figure out what works the best over time. What we've found to be most predictive (so far) is a base level of coding competency, plus max skill (how good an engineer is at what they are best at). So (beyond the coding portion) we don't actually care very much about what a candidate is bad at. We care about how good they are at what they are good at. To give as many candidates as possible the opportunity to show strength, we cover a number of areas. This includes back-end web development, distributed systems, debugging a large codebase, algorithms, and -- yes -- low-level systems (currency, memory, bit and bytes). We do not expect any one engineer to be strong in all of these areas (I'm weak on some of them). But they all are perfectly valid areas to show strength (and we work with companies that value each of the areas). We've recently moved to a new interview processes organized around this idea of max skill. It's working great in terms of company matching and predictive ability. However, it seems we may have underestimated the cost to candidates of being asked about areas where they are weak. There's more negative feedback here than we've seen in previous HN discussions, and I think that the interview change may be behind that. I'm taking that to heart. I think we can probably articulate it better (that we measure in a bunch of areas and look for max strength). We're also running an experiment now where we ask engineers are the start of the interview which sections they think they'll do best on. I'm excited about this. If engineers can self-identify their strongest areas, we'll be able to make the process shorter and much more pleasant! So, the bit shift question: that come up down one branch of a system design question that we used for a while (we've since moved to a more targeted version that is more repeatable). The (sub)issue involved adding a binary flag to a large data blob (this came up as part of a solution to a real-world caching problem). Adding a single bit flag to the front of a 1GB blob has a problem. To really add just one bit, you'd have to bitshift the entire 1GB. This is clearly not worth it to save 7 bits of storage (ignoring that that would not be saved in any case). You can just use a byte (or word), or add the flag at the end. When candidates suggested adding a bit flag at the front, we would follow up asking them how they'd do it (to unearth if they were using 'bit' as a shorthand for a reasonable solution, or if they really are a little weak in binary data manipulation). This was one small part of our interview. By itself it in no way determined the outcome of the interview, or even of the low-level systems section. Plenty of great engineers might get it wrong. But I don't think it was unfair.
mbesto 10y ago
> better predicts job success How do you rate job success?
- ammon 10y ago
  being happy with the job and not having left after 1 year
  - mbesto 10y ago
    Wait so job success is based on the interviewee, and not the company?
pyb 10y ago
Wasn't there a study by Google a while back, where they found, as a trend, that their most successful people had only marginally passed their job interview ?
- astrange 10y ago
  That sounds like the Pareto-optimal solution to a job interview. It's like how you're not doing grad school properly if your grades are better than C. Also, candidates who do too well could be too good for the job, since Google supposedly likes having incredibly overqualified people maintain do-nothing internal apps.
- jameside 10y ago
  The finding was that people who had received a "no hire" recommendation yet still received an offer tended to do well. The reason being: to compensate for the poor feedback, someone else on the hiring loop believed in the candidate so much and saw something exceptional and were willing to go to bat for them.
pfarnsworth 10y ago
It sounds like your ability as an interviewer is pretty poor. There are several examples of the same smug behavior. What sort of training have you gone through to ensure that you're actually an appropriate and qualified person to be interviewing?
- dkarapetyan 10y ago
  I interviewed with Ammon. He did not come across as smug. I went through the process just as a curiosity because I was fed up with the typical interview process. I haven't followed through anything else related to Triplebyte in terms of an actual job in case people think I have a favorable view because I got a job through them.
- Harj 10y ago
  I'm his co-founder and this comment is unnecessarily personal. Ammon has done over 900 technical interviews (https://www.reddit.com/r/cscareerquestions/comments/5y95x6/i_am_ammon_bartram_and_i_have_done_900/ https://www.reddit.com/r/cscareerquestions/comments/5y95x6/i...) and there's one negative reference to him specifically on this thread.
  - pfarnsworth 10y ago
    Nah, you can't use forum threads as proof. You need to do the analysis scientifically, which based on the feedback, it sounds like you need to do. You don't bother quantifying the ability of you and your interviewers, you just assume you're great.

jtchang 10y ago

How would you actually bit shift 1 gigabyte worth of data? Would you start at the end and read in words? Possibly using some temp variable?

liveoneggs 10y ago

so like every other interview ever?