7 ms·

Ask HN: Anyone Interviewed at Fly.io?

24 points by aosmith 4y ago

I sent a resume and got an (almost) immediate response which included a lot of credits ($500 USD). The "code challenge" they sent doesn't make practical sense and the person responding isn't from HR. They're a "Retail Sales Manager at Fly.io". This whole thing just feels off, like some weird sales tactic. Does anyone know these people?

mbrain 4y ago
I've also had a not so cool experience with them, the interviewer canceled 30 mins before the call and mentioned they will send an email to reschedule tomorrow. That was in September 2021, haven't heard back again :)
- aosmith 4y ago
  I was applying for a infra role and they asked me to setup an app and ELK stack in nomad on fly.io... I don't think their target market has infrastructure people and it just feels like Russian dolls.
  - detaro 4y ago
    I don't get the problem. If you applied for an infra role, you are clearly not representing their target market, but they want you to show an infra thing on some VMs. And it makes sense for them to let you run those VMs on their platform, where they can easily pay for it?
    - aosmith 4y ago
      Fly is a no infra platform like heroku. It's like being asked to setup k8s on heroku.
  - deleted 4y ago
    [deleted]
- tptacek 4y ago
  I see the hiring conversation you had back in September '21; I don't see us canceling a meeting, but I'm sure that happened. That sucks! Getting an inscrutable Russian Nesting Doll response from the erstwhile retail sales manager of a company asking you to set up Nomad on top of Nomad and ELK on top ELK is an off-putting experience, but it's the kind of off-putting experience we're comfortable giving people: what we're talking about is either going to click, or we're probably currently not a good fit, and that's fine. On the other hand: the experience you apparently had, of going through our process and being unceremoniously jettisoned without explanation is off-putting in a different and less tolerable way. It's something we try hard never to do, but we're a small team (much smaller in 2021) and we screw up sometimes. I apologize.
  - Cheezewheel 4y ago
    Apologies are pretty damn cheap. The excuse of being a 'small team' is bizarre. Shouldn't a 'small team', of all places at least have the courtesy of not unceremoniously jettisoning candidates? It seems to be poorly indicative of a culture to both have the callous disregard for candidate experience like a large corporation while also lacking having the resources of one.
  - quickthrower2 4y ago
    ELK on top on Nomad, surely?
    - tptacek 4y ago
      Just trying to capture the spirit of the original poster's concern. :)
jjice 4y ago
They were polite and gave credits as well, but I really thought the programming challenge was ill defined and broad. From the interviews and challenges I've run, it was much too broad and really required rereading and a lot of assumptions. Their response wasn't very clear either. Could have just been me, but I've never run into that situation anywhere else/ I will say though, they were very polite.
- aosmith 4y ago
  They have been very friendly with me as well but your feedback echos my experience thus far.
- tptacek 4y ago
  I have a lot to say about "ill defined and broad", starting with "I agree with you" and followed immediately by "c'est la vie". In a nutshell: we are extremely limited in how much time we can ask of candidates: our budget is the typical amount of time we believe those candidates would spend in more typical interview process (phone screens and an on-site interview loop). That means our programming work samples (WSTs) are small, which means we have to take every opportunity to extract signal from them, which in turn limits exactly how much we can document about our expectations. We try to be clear in our assessments (and to give some feedback about our decisions). I feel like we do a better-than-average job of being transparent, responsive, and fair, but I would feel that way: I'm the Retail Sales Manager.
  - jjice 4y ago
    For sure, I know the struggle of interviewing and doing your best to make sure to save everyone's time during the process and there's a hard balance to achieve. I'd say Fly had an above average process overall from my personal experience.
    - tptacek 4y ago
      That's nice of you to say. Thank you! The other thing I'll say about our process: we don't have it exactly right yet. We learn some new big thing about once a month or so, and it doesn't seem to be slowing down; every time we learn something, by implication, it means we've been doing something pretty wrong previous to that point.
brundolf 4y ago
Maybe they're outsourcing their hiring to some third-party firm?
- aosmith 4y ago
  Chicago based per linkedin.
  - tptacek 4y ago
    The funny thing is, the term "Chicago" is how I found this post.
fpater 4y ago
I interviewed for a role there, got quick replies to my emails too. Didn't need to use any credits to do the challenge (but they mentioned that I could ask for it if I wanted to). The challenge itself was not easy, and I assume that was exactly what they were aiming for. Besides the interview I've already talked with them and they were really friendly and eager to help (it was not a business talk, I'm doing some academic research and wanted to discuss some things regarding networks/infra-sec). I believe you just had the wrong idea, specially when you mention "some weird sales tactic". Edit: in my experience, the code challenge made a lot of sense.
- aosmith 4y ago
  It just felt wrong, thanks for letting me know. Seems weird to run an orchestration system in another one. Fly wasn't meant to be a VPS / server host and using it as one seems odd to me.
  - anuragsoni 4y ago
    I don't work for fly, but I'll say that while the end-user of fly.io isn't expected to run an orchestration system, fly.io themselves runs an orchestration system (nomad from what I can tell), so it seems like a fairly reasonable idea for a take home evaluation for an infrastructure role at the company to evaluate skills/knowledge about orchestration systems.
    - pcthrowaway 4y ago
      They posted maybe 6 months ago that they were migrating from Nomad to K8s (and expected it to take a while) so it's possible they're not on Nomad anymore.
      - kasey_junk 4y ago
        I would be _shocked_ if they moved to K8s.
        4y ago
throwaway81523 4y ago
I sent an email a year or so ago about their unpaid pre-interview homework project (a non-starter to me). I got back an email from a real person, though I didn't pursue it because of the homework and an unrelated issue.
- tptacek 4y ago
  We don't do interviews at all; there's no "pre-interview" anything here.
  - throwaway81523 4y ago
    I see. Well, I didn't proceed past that email exchange, so I don't know what would have come next. What does your process consist of if there is no interview? It can't be just the homework, I hope.
    - tptacek 4y ago
      It's all work samples. What else would it be? Interviews don't work; they select for people who interview well. The work we do is software development, not interviewing.
      - throwaway81523 4y ago
        I see. I could imagine submitting something I had done in the past, but the idea of doing custom work for free doesn't sound equitable. There was another issue too, but that should probably stay private.
        tptacek 4y ago
        I'm easy to contact, if there's anything else you want to tell us.
        throwaway81523 4y ago
tptacek 4y ago
I'm the Retail Sales Manager at Fly.io. How can I help you? We're happy to take the credits back if they make you uncomfortable. (I kid. I kid.)
- nullfield 4y ago
  So far, your comments and engagement with this thread are close to making me apply…
  - quickthrower2 4y ago
    I had a look at their hiring page, and I would apply because it looks pretty awesome to apply. But I would feel bad to do that unless I was serious about changing jobs!
half0wl 4y ago
I'm currently doing so. I'm pretty early in the process, and I've had a great experience thus far: quick replies, nice challenge (it was actually a challenge for me because I haven't touched the relevant language/framework in many years), and I generally appreciate these sort of interviews (I know I'm an outlier; unpaid work isn't a huge deal for me). Out of curiosity, what did they send you, and what do you find weird/off about the process? It's definitely not the norm [0] but isn't it better than spending a bunch of time working through back-and-forths in "normal" hiring pipelines? [0] https://fly.io/docs/hiring/hiring/ https://fly.io/docs/hiring/hiring/
karel_3d 4y ago
I have, for an engineering position. I put the experience on GlassDoors, but in general... it's fine, but took way too long. On plus side. The coding challenge was amazing and I learned a lot doing it. I passed the first code challenge, then I didn't pass the followup exercise which was on slack. It was very clear that the exercises are directly related what I would be doing. On the negative side, the whole process took far too long and there were around 1 month delays in all the messaging. That made me think I don't want to apply there again, because I don't have infinite time in my life. However, if you can afford to wait, it is fine.
- karel_3d 4y ago
  btw their challenges are public on github https://github.com/fly-hiring https://github.com/fly-hiring
- tptacek 4y ago
  Yours definitely took too long; based on feedback that included yours, we changed up the process. We're not perfect by any stretch, but I think we're much, much faster now. (I'm writing a post about the iterations we've done on this process since 2021.) Long story short: you're right; you had an experience with us that we're not OK with.
  - karel_3d 4y ago
    That's nice to hear. The actual interview was really fun (although it was at the end clear you are looking for someone a little different) and I learned about eBPF, which was great. If you fixed the delays, then it's even better.
    - tptacek 4y ago
      Fixed? No. But we're much better now than we were, and getting better. Again, we're not OK with dragging this out for people, and we definitely did that to you.
aosmith 4y ago
For posterity I talked to them tonight, this was legit and it's something they're working on internally. No love lost, just a little confusion.
sosodev 4y ago
I recently did the hiring challenge for the platform role. My solution worked really well and I thought the code was clean but they told me that I didn't score well enough to move on to the next round. I asked about leveling since the fly.io hiring page claimed to be hiring levels 1-3 and I didn't see how I could be screened out of consideration for level 1. They did not reply.
- tptacek 4y ago
  If you didn't clear the platform code challenge, leveling didn't come into it. We do respond to responses we get after rejecting people, but it can take us a few days; we tend to do batches of them all at once. My take is: while people are still in consideration, we have an obligation (we don't always meet) to be responsive, because people are pending their career decisions on our own decisions. Once we've communicated that decision, we're less obligated to be zippy, because you have the high-order bit of what you need from us. I don't know anything about your submission and deliberately didn't try to look it up, because our grading of your work sample is nobody's business here, and I don't want to create the impression of communicating anything about it. Having said that, I can tell you what we tell everyone who asks why their code sample didn't pass: We've made a decision to budget the amount of time we ask from candidates, end to end, from application to final decision, to the amount of time they would spend in a hiring loop at a typical company that just did interviews and resume screens. That implies that we have to keep our coding challenges short and succinct. I can't ask you to spend days coding something; that's more than an interview would ask of you. A consequence of a short coding sample is that there's less surface area for us to derive signal from. We have to be pretty greedy about selecting coding decisions to judge, because there isn't all that much code to work from (for anybody who doesn't know, our current platform code challenge is to write a TCP proxy). A consequence of having to jealously collect signal from coding challenges is that we're limited in what we can tell you about our grading. We can't offer you the rubric we use (there is one; reviews are mostly blind, and done by a pool of engineers), because that deprives us signal. We'd learn how good you are at following directions, but not much about how you approach systems programming problems. I'd want a scoring rubric (and fast responses to emails!) too. I see where you're coming from. All I can do is tell you where we're at.
saclark11 4y ago
I applied to Fly.io and a big part of that was because of their hiring process. Reading about the way they think about finding good candidates resonated with me. The role for which I applied was something I am deeply interested in, and would love to get into, might _possibly_ have the chops, but for which I probably do not yet have "the" resume. It felt like a Hail Mary to apply but I figured if nothing else I'd learn alot in the process and if I'm really lucky, might even get some constructive feedback. I didn't clear the work sample but man, I had fun doing it. I learned a ton. I most appreciated that they gave me legitimate feedback as to why they decided not to move ahead with me. It highlighted a gap in my current knowledge/experience that I was glad to discover. That being said, it did take a very long time to get a final answer back from them, but overall I'm a happpy reject. Could even see myself applying again some day.
- tptacek 4y ago
  Please do! I'm at pains to tell people that our criteria are in part based on where we are now as a team (L3/L1 balance, our current idiosyncratic product management strategy, &c) and your results might be different even if everything plays out the same way. We try to be up front about stuff if you ask us about re-applying. Thanks for bearing with us!

sosodev 4y ago

"If you didn't clear the platform code challenge, leveling didn't come into it." How does that make sense? I was told that you got a ton of applications and that I didn't meet the cutoff for the pool. Surely you can't just keep the top X (open positions + margin) and try to place them into appropriate levels if they make it through the remaining rounds? You very likely won't have an appropriate mapping of skills to your open roles and candidates will be left dissatisfied, right? Isn't a hiring challenge supposed to be a more objective assessment of engineering skill? If there's less "there's less surface area for us to derive signal from" then maybe it's not better than the traditional model. To be told that I didn't do well enough to be considered for a junior position sucks because I'm a mid level engineer. I'm certain that my proxy worked well. I was able to push ~7 GB/s through it spread across 1000 connections on my local machine. It can do that with reasonable resource utilization, good latency, no race conditions, and no errors. I also wouldn't say my notes were skimpy. I could have written more but I wanted to be respectful of the reviewers time. In fact I would rather you openly shame my submission here because at least then I would have the oppourtinity to reflect and improve. I hope this level of criticism isn't too much. I just wish that software engineering interviews could be more rational.

tptacek 4y ago
That coding challenge just isn't where we make leveling decisions. The difference between an L1 and an L3 at Fly.io is mostly not about how you write proxy code, but rather how you manage scope, how you think about what customers need, and how you design systems. We didn't tell you you weren't good enough for a junior position. That's not what our actual email response was to you, and it's not what I said in my previous comment. The single exercise you did with us doesn't currently have a leveling component, so there's nothing I can tell you about what level you'd have wound up at had you passed it. I understand that you're giving me permission to relate more about how we scored your submission, but, again, I'm unwilling to do that, for all the reasons I gave upthread.
- sosodev 4y ago
  "We didn't tell you you weren't good enough for a junior position." Not explicitly but isn't that the implication if I applied to a position open to levels 1-3?
  - tptacek 4y ago
    Our whole hiring process does level people, but the one challenge you did does not. It's pass/fail. So: no, there is no such implication in being rejected at that point. I'm saying that explicitly right now, so there needn't be any uncertainty going forward. I think it's also pretty important to understand that different companies have different leveling functions. Not being L3 at Fly.io in 2023 doesn't say much about what level you'd be anywhere else.