5 ms·

So you communicated via email and the only consideration was the quality of their sample? Nobody at any point literally spoke to this person? We've had a ton

by swozey 3y ago

So you communicated via email and the only consideration was the quality of their sample? Nobody at any point literally spoke to this person? We've had a ton of issues at my corp of people having other people handle their interview questions, code, or having AI listen to the conversation and answer questions by text as if we prompted it. Then they show up online after being hired and they have webcam problems or never go on cam, talk as little as possible, basically never communicate to anyone really, miss a lot of meetings, etc. We've had probably 5 instances of this last year we're absolutely super strict about things now. Our assumption is that they want to get in, potentially get any bonuses if there are any, get a few huge (for their call center) paychecks until they get caught. Do no or almost no work. And it's a call center you can sometimes hear tons of convos going on in the background so they're just doing this en masse. This is happening ALL over since covid. I think some of them may have even filed unjust termination sort of things to the workforce commission but I may be wrong. Huge companies are dealing with this at a really rough scale.

tptacek 3y ago
Our process runs mostly through email until the last challenge, which is interactive (for every role in the company we simulate some kind of design or problem solving exercise on Slack with an random member of our team, which is then rated based on a rubric by a panel that reviews the transcript; it differs from an interview in a variety of ways, most notably that the team member delivering the exercise doesn't ever interrogate the candidate). Most of our work sample stuff is design/architecture based. I'm not worried about how LLMs impact it. This is the sole way we've hired engineers at Fly.io since 2020 and we have never once brought someone on who turned out to be 3 raccoons in a trenchcoat. Raising the bar further on ourselves: we hire globally, at SF scale salary worldwide.
- potamic 3y ago
  That's an interesting approach. In an interview, you generally drive the discussion a certain way to gather data points for your rubric. Wouldn't this happen in your case? If yes, how is it different from an interview? Otherwise, how do you evaluate candidates who solve the problem but don't seem to communicate much about it.
  - tptacek 3y ago
    Nope! It's complicated. We put a lot of effort into deprogramming candidates before the simulation (we call it a "workday"). If you join the Slack channel for a workday and wait for us to ask you questions, almost nothing will happen. The candidate drives, apart from the brief we provide at the beginning of the session. The rubric heavily draws from what questions the candidate asks, and which topics they choose to zoom in and spend time on. It's interesting; I warn candidates that studying for it ahead of time (for instance: for a long time our workdays were all based on a block storage problem) puts you at a disadvantage, by robbing you of good questions to ask. It's not a big disadvantage, but interview-style preparation isn't doing much to help you either. I'm always reluctant to talk too much about the workday process, in particular because the more I describe it the less chill it sounds. In reality, it's you joining a Slack channel with an engineer from our team and just shooting the shit about something we're (in the simulation) planning to build. A lot of what that challenge is evaluating is communication. We're an all-remote company with team members in virtually ever time zone; being able to work out in the open in the environment we've created is important. I think our workdays are a pretty quirky feature of our process. You could instead do a scripted interview, where everybody gets precisely the same questions or prompts. The big thing we do with the workday that I think everyone should clone is: deliver the interview over Slack, and don't allow the person delivering the interview to grade it; instead, have someone else grade from the transcript. It was like a light went off (on?) for us when we figured out to stop having workday participants offer performance feedback about candidates. (One thing you immediately get from it is a process to iterate on the quality of your staff delivering the workday/interview; you can immediately see which team members click with candidates and which don't, and then nudge people in the right direction. It is wild to me that people do free-form interviews without this kind of iteration system in place.)
    - afpx 3y ago
      I love that high-profile people like you are taking risks and innovating with hiring. Thank you. I'm hoping to see some change in the industry. There are probably a lot of high-performing people who have little motivation to go through the normal interview process.
- tasubotadas 3y ago
  Could you give some exercise examples? How complex do you make? Does it feel like doing work for you?
  - tptacek 3y ago
    It feels like what working here feels like, but it is strictly not doing work "for" us.
    - tasubotadas 3y ago
      How different are the take home exercises from the workday?

by swozey 3y ago

tptacek 3y ago

Our process runs mostly through email until the last challenge, which is interactive (for every role in the company we simulate some kind of design or problem solving exercise on Slack with an random member of our team, which is then rated based on a rubric by a panel that reviews the transcript; it differs from an interview in a variety of ways, most notably that the team member delivering the exercise doesn't ever interrogate the candidate). Most of our work sample stuff is design/architecture based. I'm not worried about how LLMs impact it. This is the sole way we've hired engineers at Fly.io since 2020 and we have never once brought someone on who turned out to be 3 raccoons in a trenchcoat. Raising the bar further on ourselves: we hire globally, at SF scale salary worldwide.

potamic 3y ago
That's an interesting approach. In an interview, you generally drive the discussion a certain way to gather data points for your rubric. Wouldn't this happen in your case? If yes, how is it different from an interview? Otherwise, how do you evaluate candidates who solve the problem but don't seem to communicate much about it.
- tptacek 3y ago
  Nope! It's complicated. We put a lot of effort into deprogramming candidates before the simulation (we call it a "workday"). If you join the Slack channel for a workday and wait for us to ask you questions, almost nothing will happen. The candidate drives, apart from the brief we provide at the beginning of the session. The rubric heavily draws from what questions the candidate asks, and which topics they choose to zoom in and spend time on. It's interesting; I warn candidates that studying for it ahead of time (for instance: for a long time our workdays were all based on a block storage problem) puts you at a disadvantage, by robbing you of good questions to ask. It's not a big disadvantage, but interview-style preparation isn't doing much to help you either. I'm always reluctant to talk too much about the workday process, in particular because the more I describe it the less chill it sounds. In reality, it's you joining a Slack channel with an engineer from our team and just shooting the shit about something we're (in the simulation) planning to build. A lot of what that challenge is evaluating is communication. We're an all-remote company with team members in virtually ever time zone; being able to work out in the open in the environment we've created is important. I think our workdays are a pretty quirky feature of our process. You could instead do a scripted interview, where everybody gets precisely the same questions or prompts. The big thing we do with the workday that I think everyone should clone is: deliver the interview over Slack, and don't allow the person delivering the interview to grade it; instead, have someone else grade from the transcript. It was like a light went off (on?) for us when we figured out to stop having workday participants offer performance feedback about candidates. (One thing you immediately get from it is a process to iterate on the quality of your staff delivering the workday/interview; you can immediately see which team members click with candidates and which don't, and then nudge people in the right direction. It is wild to me that people do free-form interviews without this kind of iteration system in place.)
  - afpx 3y ago
    I love that high-profile people like you are taking risks and innovating with hiring. Thank you. I'm hoping to see some change in the industry. There are probably a lot of high-performing people who have little motivation to go through the normal interview process.
tasubotadas 3y ago
Could you give some exercise examples? How complex do you make? Does it feel like doing work for you?
- tptacek 3y ago
  It feels like what working here feels like, but it is strictly not doing work "for" us.
  - tasubotadas 3y ago
    How different are the take home exercises from the workday?