15 ms·

Ask HN: Was any Starfighter postmortem ever published?

186 points by xoranth 3y ago

About seven years ago, HN veterans 'elptacek, 'patio11 and 'tptacek launched an hiring startup called Starfighter. Their approach aimed to shift away from traditional interviews and instead employ a CTF/Microcorruption kind of test, inspired by 'tptacek experience at Matasano. Unfortunately, Starfighter eventually wound down. Was there ever any post-mortem written on Starfighter? If not, would 'elptacek, 'patio11 and 'tptacek be willing to provide any details?

tptacek 3y ago
Nope. We always meant to, but got sucked into new stuff shortly after.
- fragmede 3y ago
  No time like the present! The platform itself seemed like it worked great, so the question is what did it look like on the other end - were you able to get any companies to actually take on candidates who passed the test? The software's the easy part, it's everything else that's the problem.
  - elptacek 3y ago
    The way I remember it, the general experience was that companies reached out to us and were very excited by the idea, but when we actually sent a candidate, they scotched it. I did make exactly one placement where I waived the fee (because friends). They were super happy with him but it was too little, too late.
tjansen 3y ago
See also this: https://news.ycombinator.com/item?id=12415786 https://news.ycombinator.com/item?id=12415786
- thaumasiotes 3y ago
  tptacek's comments there repeatedly state that he is (seven years ago) about to publish a postmortem on Starfighter. https://www.kalzumeus.com/2016/12/30/kalzumeus-software-year-in-review-2016/ https://www.kalzumeus.com/2016/12/30/kalzumeus-software-year... says the same thing. Did that ever happen?
  - tptacek 3y ago
    Nope. There's a lot of stuff I've meant to write!
cushychicken 3y ago
Seen some of patio11’s comments and they suggest there was some part of the business that wasn’t working. What it is exactly was never said. Could have been a myriad of things: financials, product market fit, founder relationships.
andrewalan 3y ago
[dead]
kevin_nisbet 3y ago
There is a small blurb about it in patio11's year in review. https://www.kalzumeus.com/2016/12/30/kalzumeus-software-year-in-review-2016/ https://www.kalzumeus.com/2016/12/30/kalzumeus-software-year... I think I might've been the one referred to as submitting the excel file with the explanation... as I think that was how I solved the last problem on the stockfighter thing.
- deleted 3y ago
  [deleted]
dinp 3y ago
In case the founders read this post: what would you do differently if you could start over and would this idea work today? Slightly OT: everyone feels hiring is broken, can you list some things that are annoying from the employee and employer perspective? Here are some points: - the process often stretches out over weeks and often months - job posts often get 100s of applications, a lot are low effort applications, it just muddies the water for both sides - ATS systems/job boards are annoying with the need to create an account on many sites, some forms have more than 20 questions, often asking what's already there in the resume. A question to everyone: What would a good application process look like? For me, it should just solve the above mentioned problems. I send an email with my resume, a few sentences about why I might be a good fit for the role/what interests me about the company. The jobs@<company>.com email address could be linked to some Saas product which makes it easy for the employer to go through the applications and further communication about video calls or take home assignments or whatever are all in this email thread. The employer can set the stage of the application such as 2/5 or whatever, they can mark it as rejected or accepted after all rounds to trigger automated emails etc. Is there any Saas like this? (I can build this in a week if it doesn't exist, but no clue how to market it/get users, any pointers in case me or someone else builds this?)
- romanhn 3y ago
  You're describing basically every ATS (applicant tracking system). The problem is that those are all internal-facing, so there's no incentive to make your life easier as a candidate. I'm trying to tackle this with https://rolepad.com https://rolepad.com, a job application tracking system that can be connected to an employer portal and provide greater transparency and just make the whole candidate experience much better (the employer part is still under development though).
  - IMTDb 3y ago
    > so there's no incentive to make your life easier as a candidate Especially because lowering the barrier usually means getting more low effort application. People either put a lot of effort on few applications and invest significant amount of time in each one hoping to increase the chance they get picked up; or they put a lot of effort on reaching out as much as possible, knowing they have a very small chance on each application but playing on volume. Most recruiters are only interested in the first category of candidates. Having a long initial process is a feature. The goal is that the second category considers that given their chance of getting picked up, it's not worth filling the form.
slater 3y ago
OT, but what's with the weird apostrophe/tick before each name in OP?
- kristjansson 3y ago
  I read it as a LISP-y (and therefore HN-y) version of @
  - solardev 3y ago
    I read it as an unterminated string and kept looking for the end =/
  - password4321 3y ago
    I vote for ƛ
- quesera 3y ago
  One of the founders of Starfighter wanted to popularize the leading tick as the HN-equivalent of the @ on other social networks. I read it as an homage to HN days of yore.
  - tptacek 3y ago
    It's better than @! @tptacek isn't me!
monero-xmr 3y ago
No one can ever "solve" the recruiting issue. We are exactly where we were 15 years ago - not one thing has changed except certain screen-share coding tests, which are basically the same as white boards. Can you reduce hiring an airplane engine designer to a set of tests? Or a custom home builder? If so, why is there not just a stack ranking of best ones, or an algorithmic way of hiring one? Software engineering at elite levels is an art. It is conducted by master craftsmen. On top of it the best people are often quirky / weird and the social aspect when communication of needs is so essential makes it even more difficult to hire "the best". The business models of recruiting is always either like sales (recruiting firms) or marketing (LinkedIn). I would stay away from the recruiting industry as a software startup - difficult to make any money here in a novel way.
- solardev 3y ago
  I wish we had industry-standard certifications that actually meant something, and were widely accepted. The closest I've seen in the real world is just "degree in comp sci". Other orgs have tried to come up with various certs, but they never seem to have any sticking power. I guess it's a hard thing to certify when the landscape itself is constantly evolving and skills are obsoleted in 3-4 years. It seems even the pace of change is itself speeding up, and every year I see more technologies come to popularity (if not quite maturity) than I can even remember, much less learn. And it's not just languages and frameworks, but entire architectures... today's best practices are tomorrow's tech debt. There should always be room for innovators to push the bleeding edge forward, but many businesses don't really need that. Instead, if there was just some shared certification for "good enough engineering" that can deliver a reliable, maintainable product somewhat on time, reproducibly, hiring and job-hunting would be so much easier. I'm so tired of trying to charm recruiters, impress managers, placate HR, all while hoping against hope that the actual dev 2 or 3 interviews later happen to have some overlap in languages, toolsets, shared pain points, etc. It's all such a dog and pony show, with many teams having ill-fitting skillsets and cultures between members because nobody really knows what they're signing up for or getting.
  - monero-xmr 3y ago
    It’s simply too free-form and the industries software is used in are… all of them. I believe it is better to certify the outputs (product features, uptime SLAs, security) than the inputs and how the sausage is made. Software engineering is building a factory. The inputs are data. The outputs are data. All of the stuff inside the application are the different conveyor belts, elevators, gears, rollers, pounders, etchers, etc. to move and manipulate the data to make the output valuable. The factory can be constructed infinite ways without any care about physical dimensions, and the raw material can be duplicated to infinite amounts. The situation is beyond regulation and it should not be regulated. Only the product output should be regulated.
danielvf 3y ago
Starfighter was a blast to play, and taught me a lot about the back end of stock trading. I did a lot of silly things after winning the initial play through. - Solving the final challenge using a basysian spam filter on trade directions triples. - Live 3D market visualization in Minecraft - Buddy of mine built a stock exchange stack, speaking the same protocol, and we had PvP contests on who could market make the best on it. I won with a PHP powered marketing making bot with live code reloading. - Built a cardboard box out of scrap electronics from adafruit that would let me adjust a market maker bots net position by tilting the box from side to side. - Was either #2 or #3 to solve all the challenges On the business side, if I recall correctly, most companies wanted to just shove the winners down their existing bad hiring pipeline process, and a lot of successful contestants who showed up already had great jobs, and so weren't actually looking.
- armitron 3y ago
  Sounds like fun. To founders: Any plans to open source it so that it’s not lost in the sands of time?
  - clusterhacks 3y ago
    It was fun! I did all the stock trading challenges but none of the embedded device challenges. patio11 once said that the code wasn't in an open-source ready state. There were some folks who went so far as re-implementing/duplicating the backend trade simulator in other languages. Some of those were on github . . .
- clusterhacks 3y ago
  Totally agree it was a blast. I learned a ton about trading just reading various recommended resources (Harris Trading and Exchanges was a solid intro) and then solving the levels. I don't think very many players did all the Starfighter trading challenges. patio11 or tptacek might be able to confirm, but I vaguely recall being one of like 100 or so to do all six levels of out of 10,000 that started. But I also had free time for playful coding and was nerding out on online challenges back then. When Starfighter came out, I was energized and hopeful that hiring would go in this direction for programmers or that it would open doors to me as a job seeker that I couldn't every push through. No such luck. From the hiring side, I have been on two hiring committees this year. Most applicants lacked any meaningful programming projects they could use to demonstrate skills. Something like completing a series of online challenges would be a difference maker. That said, one candidate mentioned doing HackerRank challenges to learn Python and SQL, but totally flamed out on being able to even talk through a for loop in Python or come up with a simple select against a table in the in-person interview. Maybe nerves, but it also felt like the candidate's experience was 100% as just a warm body in various cubicle farms. It was very awkward. Several non-developers on the hiring committee felt like the candidate had actively lied about previous experience . . .
fwungy 3y ago
Here for the F-104 discussion. Fascinating plane.
- MadnessASAP 3y ago
  Ah yes, the ol' Lockheed Lawndart. All the fun of flying supersonic at low altitudes and none of that turning nonsense. A masterpiece of flying by the seat of your pants, especially since you definitely wouldn't want to take your eyes off the windshield to actually look at your instruments.
  - nocoiner 3y ago
    Apparently a developer is releasing an F-104 module for DCS. I cannot imagine that taking off, flying in a straight line and crashing somewhere in Germany could be in any way “fun” (even accounting for possibly distorted DCS definitions of “fun”.) I’ll almost certainly buy it, though.
ufmace 3y ago
Reading between the lines a bit on the existing postings about it, I'm guessing that they were unable to persuade any businesses to actually relax their interviewing standards for candidates who passed the Starfighter tests. Which then puts any candidates in the position of, why should I spend time and effort on this little game if it doesn't lead to any advantage at all in actually getting good positions? And so, the only people who messed with it were developers who had the free time to play with stuff like this and didn't particularly care that it wouldn't lead to any advantage in getting jobs. Which means, it's a cool toy that was high-effort to build and probably to maintain, but had no viable path to earn money from anybody.
- thaumasiotes 3y ago
  > I'm guessing that they were unable to persuade any businesses to actually relax their interviewing standards for candidates who passed the Starfighter tests. > Which then puts any candidates in the position of, why should I spend time and effort on this little game if it doesn't lead to any advantage at all in actually getting good positions? This is the same position TripleByte was in. They advertised themselves to developers as "show us you can code", but they advertised themselves to companies as "we will find candidates who pass your existing hiring process". And when pressed on which of those things they were actually trying to do, they admitted it was the second one.
  - deleted 3y ago
    [deleted]
  - aleph_minus_one 3y ago
    > but they advertised themselves to companies as "we will find candidates who pass your existing hiring process" If TripleByte really succeeded with this, the customer companys could save quite some money by simplifying their hiring processes. The problem in my opinion rather is that most company's hiring processes have an official and a hidden agenda, and TripleByte only attempted to find candidates that might succeed in the official agenda.
    - 3y ago
itslennysfault 3y ago
I have no relation to Starfighter, but I was part of a similar startup almost a decade ago, and learned a few things that are likely similar to what they faced. We started with this dream of completely revolutionizing the way people hire in tech (with hopes of expanding to other fields). First of all, we learned that for 99% of companies they don't have a screening problem, but they have a sourcing problem. Our original company was along the lines of HackerRank / LeetCode / etc, and created due to the founder's experience as a manager at Amazon where he'd get 1000s of resumes that needed to be screened and the process was eating up too much dev resource. A product that solves this problem is appealing to companies that get 1000s of resumes per posting, but basically useless to most companies who get very few candidates, and even less appealing to recruiters who care much more about top of the funnel than the screening process. After that we pivoted to an event format. The idea being that we could attract 100s of applicants to do a challenge in realtime. This reduces cheating risk and allowed us to act as a means of sourcing candidates (instead of just a screening platform). The idea was 100s of candidates would enter and the companies we got to sign up would compete for the top applicants. It sorta worked. The platform was good, companies were interested, and we got some people placed. HOWEVER, the profile of people that will do a multi-hour open ended challenge for the chance to maybe get hired by some companies largely leans toward entry-level people. We had some experienced people that did it because they were bored or just curious, but generally seasoned engineers aren't going to waste their time, and forget about "passive job seekers" which is what all recruiters want most of all. Ultimately, the company ended due to a lot of personal issues between the founding team, but the product itself was good, and the market fit was decent, but far from perfect. I'm pretty sure if we would've kept it together we would've eventually became very similar to HackerRank just offering a platform where people can conduct the same crappy tech interviews we've been using forever.
- ska 3y ago
  > the profile of people that will do a multi-hour open ended challenge for the chance to maybe get hired by some companies largely leans toward entry-level people. Wasn't this assumed to be the case going it to it when you discussed the business cases? It's hard to imagine any other result would be likely.
- 3y ago
tptacek 3y ago
Postmortem: * Our go-to-market was contingency recruiting; we'd get paid for placements. Our CTF and HN calling cards easily got us added to the recruiting funnels of big tech companies (I'm assuming any of you could have done that too; I'm just saying, we never had to cold call). But it didn't change anything about how those companies qualified candidates, so to make that business successful we'd have had to do the same dialing-for-dollars matchmaking work any recruiter does, with comparable results. * We took things very personally. You build relationships with your earliest users (candidates, in our case). Recruiting funnels at our clients had no such connection. The impedance mismatch was demoralizing; our lived experience was that every time a candidate we presented to a client got rejected, we had ourselves failed (I mean, really, we had!). Similarly, the idea that we were just another hoop that candidates had to hop through (but only the non-name-brand candidates who didn't know how to skip us to get an interview at a client) grated on us. * A lot of second-system-syndrome on the implementation side; we'd done successful recruiting CTFs before, and set what was in retrospect a completely bizarre goal of outdoing ourselves at that job. That mistake is all me. * Japan/Chicago remote founding team was very difficult, most especially for Patrick. There are teams that could handle that, but their members are more punctual about routine meetings than I am. Both the US and the Japan side felt checked out to each other. This didn't kill us; other things did, and we could have pushed through this problem. But it made things miserable for everyone. Again: this is all me. * Erin & I were self-funding the company, which was expensive (just in terms of paying everybody's personal bills). Towards the end, it had become clear that we weren't going to drive 3 FTE worth of revenue on contingency placements in the next 12 months. We could have switched up the business model, driven user growth metrics, and gone out for seed funding, but none of us at that point were convicted enough of the business to stake our reputations on that. For all 3 of us, our time was worth too much. I think we made the right call. It's been almost a decade since we did this, and those years haven't been especially kind to recruiting startups. There are knobs we could have turned to keep things chugging, and probably chugging comfortably, for many more years. But ultimately, I don't think there were any significant rewards to win by building a tech-company-specific recruiting firm. I think the company was ultimately fated to wind down sooner or later. If you're in that situation, sooner is always the better option! Life is short! The underlying idea behind Starfighter, about the right way (or, at least, a right-er way) to hire people, has proven out repeatedly since then. We hired resume-blind and with work-sample challenges at Latacora, the next company me and Erin started, and we hire resume-blind and with work-sample challenges at Fly.io. Recruiting this way is a competitive advantage. The degree of advantage varies with the market; when the market is weak you can hire good people with terrible processes. But the advantage is never marginal, because bad hires are expensive, and there's so much hidden talent out there.
WA 3y ago
It was called Stockfighter, not Starfighter. I played a few levels and then it got taken down unfortunately. Would love to try again. Edit: patio11 called it Stockfighter, but others didn’t? Here’s the original launch post: https://news.ycombinator.com/item?id=10724592 https://news.ycombinator.com/item?id=10724592
wkat4242 3y ago
Lol I thought this was about the infamous F-104 Starfighter that killed hundreds of pilots in Europe. And it was only purchased because Lockheed paid millions in bribes to officials for which AFAIK none of them (nor the company) were ever punished. So in terms of post-mortem I was thinking of something official like that :) Ps its successor the F-16 was a lot better though and is still going strong today.
- deleted 3y ago
  [deleted]
- Miraste 3y ago
  >killed hundreds of pilots Wow, I thought you must be exaggerating, but it was literally hundreds. Some air forces lost nearly half the planes!
  - wkat4242 3y ago
    Yeah it was basically the G version for export, the plane was originally designed as a fast interceptor. But it was converted to a fighter/bomber for NATO allies that needed one. With its tiny wings it was just not great for low & slow bombing runs in poor visibility. Added to that were inexperienced crews and well, disaster ensued :P They called it the lawn dart or widowmaker in Germany. I think this is why the F-16 was so much more popular in Europe, it was actually designed as multirole and was and is good at it.
neilv 3y ago
They had a great name. For anyone who didn't see the '80s light sf movie, The Last Starfighter, the premise is appealing and classic: (minor SPOILERS) young adult kid stuck living and working in a trailer park, wants more, there's a Starfighter video arcade game set up in the park, one day he goes for the record on the game, and turns out the game was a recruitment tool for something much bigger than his humble trailer park, and he passed, rises to subsequent challenges, and becomes a hero. (Co-stars Robert Preston, as a very enterprising and scrappy recruiting headhunter, a bit like his The Music Man film character, but without the musical numbers.)
- ghostpepper 3y ago
  > without the musical numbers Just imagine how good that could've been though
- toomuchtodo 3y ago
  https://www.youtube.com/watch?v=4Q-Qs3loKoc https://www.youtube.com/watch?v=4Q-Qs3loKoc https://en.wikipedia.org/wiki/The_Last_Starfighter https://en.wikipedia.org/wiki/The_Last_Starfighter https://www.imdb.com/title/tt0087597/ https://www.imdb.com/title/tt0087597/ "Greetings, Starfighter. You have been recruited by the Star League to defend the Frontier against Xur and the Ko-dan Armada."
- elptacek 3y ago
  I think if we'd survived there would have come a day when we got a letter from somebody's lawyer. But as a kid who grew up below the poverty level, the idea of an epic escape remains near and dear to my heart. I've probably seen it 50x.
  - neilv 3y ago
    Computer programming back then was a great way for a kid to escape to a middle class livelihood, just based on getting machine access somehow, and what they could learn on their own. (It seemed to change, after the dotcom boom, when the jobs turned into a lot more money that attracted a lot more people, and class barriers were erected to many of the better jobs. There's still a chance for upward mobility, despite artificial class barriers, but people will tend to burn their energy mimicking the shibboleths and practicing for the hazing rituals, rather than on things that would attract and teach them. Which brings us back to this application domain, of helping companies to hire people who would be great software developers.)
weinzierl 3y ago
I always wanted to play it but my life was quite busy back then. Is there any chance we could have it released as purely a game?

indymike 3y ago
> Having a long initial process is a feature. Reality is: 1. Screening tools (background check, skills analysis and testing) are really good. There's really no reason to put someone through a six-week gauntlet just to find out if they can do the job. 2. Interviews are really a tool for seeing if coworkers and managers get along with the candidate, and the candidate gets along with their future coworkers and managers. In many companies interview == where manager has discretion to hire the people they want. Lots of interviews usually means lots of managers who all want to put their stamp on the hire. 3. Being good at getting hired is a function of experience getting hired. The skills needed to be a good candidate do not align with the skills needed to be a good worker in most roles.
- ska 3y ago
  In my experience: (1) just isn't true. It's at best a mixed bag. (2) is somewhat true. Team fit is pretty hard to figure out without interactions, and agree sometimes managers just want to get their oar in. (3) is oversimplification. There are definitely people who are better a getting hired than doing the job, and they are hard to weed out early in their career. If you aren't able to identify good candidates in front of you though (e.g. the FN, not FP) there are problems in your process, not the candidates.
  - indymike 3y ago
    1. I'm pretty sure that background checks work beyond doubt. I'm pretty sure that skills tests work beyond doubt. I'm less confident in AI resume/profile analysis and skills matching tools - both are getting better. 3. Agree, it's simplified, but for most companies, field and "hiring" managers are barely able to do an effective interview, let alone smoke out someone lacking interview skills from one lacking communication skills.
    - ska 3y ago
      Agree both background checks and skill tests can work very well in an ideal form, but it's expensive (in time, mostly). As practised, they are very much a mixed bag. The latter is a problem past early career stuff anyway, as you aren't going to learn much in a couple hours.

michaelt 3y ago

> A question to everyone: What would a good application process look like? A complete overhaul of credentialing, so that when I have a degree in computer science from a top university and decades of industry experience, employers don't need to independently confirm that I know how to reverse a string or traverse a tree or define what XSS is. Also, make a time machine so we can make this change 30 years in the past, if you wouldn't mind.

ghiculescu 3y ago
The problem with credentialism is that lots of people with degrees from reputable universities can’t code. Similarly lots of people with significant industry experience… also can’t code.
- zo1 3y ago
  [flagged]
  - ghiculescu 3y ago
    That happens too but I think the underlying reasons are simpler. Everyone involved in hiring should read “The Case Against Education” by Caplan.
  - nemothekid 3y ago
    The FizzBuzz test came about for similar reasons in 2007 - long before government pressures for DEI quotas[1]. The problem is computer science university degrees aren't enterprise training academies. "Computer Science is no more about computers than astronomy is about telescopes." Universities aren't incentivized to fix this because universities are research institutions - they don't make money from placing students in jobs, so trying to solve this is a dead end. [1] https://blog.codinghorror.com/why-cant-programmers-program/ https://blog.codinghorror.com/why-cant-programmers-program/
    - thaumasiotes 3y ago
      > The FizzBuzz test came about for similar reasons in 2007 - long before government pressures for DEI quotas[1]. Government pressure for hiring quotas goes back to the 1970s.
      - nemothekid 3y ago
        It's moreso I don't believe there was a ton of government pressure being applied to startups with 15 people.
        thaumasiotes 3y ago
        That's also still true today.
- red-iron-pine 3y ago
  this has been solved -- with take home coding or whiteboarding. in the networking side of these we used to have a couple of Cisco 2960s with slightly borked configs and our interviewees had to log in and tell us what's wrong. 5 minute activity, but often took some folks 30 minutes.
  - ghiculescu 3y ago
    Yeah I think that’s what Starfighter did. And we are discussing how it failed, so…
- mprovost 3y ago
  What does "reputable" mean in this case when these universities are handing diplomas to graduates who can't code? Somehow we've gotten ourselves into the situation where we force someone with a CS degree from Stanford write some leetcode, even when the people on the hiring panel themselves have degrees from Stanford. Do they still not trust their own university? And yet, when scanning resumes, people often look for prestigious/"reputable" universities as a filter. Is the assumption that if they have a degree from Stanford and can reverse a string then that is a positive signal, but if they can't reverse a string that is somehow a personal failing and not a failure of the university to educate (and then evaluate) the candidate?
  - jfim 3y ago
    > What does "reputable" mean in this case when these universities are handing diplomas to graduates who can't code? Somehow we've gotten ourselves into the situation where we force someone with a CS degree from Stanford write some leetcode, even when the people on the hiring panel themselves have degrees from Stanford. Do they still not trust their own university? What's the alternative? Just give leetcode tests to people who didn't go to the same university as the hiring panel? The fact that we have the same tests for everyone no matter where they studied at is a good thing and prevents nepotism in hiring decisions.
    - michaelt 3y ago
      A professional organisation like the ACM which 'accredits' college courses, making sure that 3.5 GPA from MIT is equivalent to a 3.5 GPA from Podunk University.
      - throwaway2037 3y ago
        As someone who attended two different universities for my undergrad, the first was way harder than the second. Basically, I failed out of my first, and cruised through my second. I intentionally selected a much lower ranked university for my second attempt. I felt like I was buying a university degree at my second place. And, yes, both were ACM accredited. It means little. What I am noticing on all threads about HN hiring: Hiring knowledge workers is fundamentally a difficult, fuzzy process. Few want to admit it. And, there are some people are naturally much better at it because they can control themselves very well during an interview. People who are very strong at solving puzzles will nearly always outperform during interviews for hands-on roles. More than anything, interviewing feels like dating. Everyone hates it until you find what you are looking for... then it becomes: "Oh, it worked fine for me."
- michaelt 3y ago
  The reform I'm proposing is that reputable universities should only issue CS degrees to people who can, in fact, code. That's why the time machine is so important - to get back any CS degrees issued to people who can't code. So I think you are actually agreeing with me?
  - vunderba 3y ago
    This sounds more like a PE Exam equivalent for software engineering which I honestly think should be put into place since most traditional engineering industries have this already.
    - __turbobrew__ 3y ago
      Yea, I think that is the alternative. A professional body which accredits its members. It is a system which is known to work for other disciplines.
enraged_camel 3y ago
I used to share your opinions, until I sat at the other end of the table and interviewed candidates. The problem is that people lie constantly about their credentials, experience and knowledge. They lie on their resumes, they make up fake references (sometimes going so far as to pay their friends to lie for them), they might even have someone create fake portfolio websites for them and showcase those as their own. There will never be a way to detect such deception without asking candidates basic questions as a way to at least "smoke test".
- wombatpm 3y ago
  A resume is not just a piece of paper. It’s a piece of paper with lies on it. And it can be the difference between not getting a job and not even coming close. -Dave Barry
- scarface_74 3y ago
  I’ve been interviewing people as either just another person in the loop or the person who could give the thumbs up about whether I wanted someone on my team and the manager just rubber stamped it for over 20 years - no coding just conversations. I’ve never regretted a single person I hired and they have always been able to do the simple CRUD work that most of the 2.7 million developers are doing. And before the gate keeping starts. I’ve been through the “Making Great Hiring Decisions” training and have conducted behavioral and system design interviews at AWS.
  - lmm 3y ago
    > I’ve been interviewing people as either just another person in the loop or the person who could give the thumbs up about whether I wanted someone on my team and the manager just rubber stamped it for over 20 years - no coding just conversations. Sounds like a good way to filter for the right class background. (Which, given how well that's correlated with intelligence, is probably as good a hiring method as any)
    - scarface_74 3y ago
      What do you mean by the right class background? If you think I’m a white guy who graduated from a prestigious school, I can guarantee you that’s a far from accurate assumption. Let’s just say when I walk through my neighborhood I’ve read comments about me in my own NextDoor group about someone “suspicious” and realized they were talking about me or my son…
scarface_74 3y ago
It still doesn’t tell me whether you kept up with what the market wants, you have the specific skills I care about or whether you have the behavioral traits I am hiring for.
quesera 3y ago
> a degree in computer science from a top university and decades of industry experience > know how to reverse a string or traverse a tree or define what XSS is. In my experience, it is absolutely not safe to assume a correlation between these two sets of traits.
VirusNewbie 3y ago
>A complete overhaul of credentialing, so that when I have a degree in computer science from a top university and decades of industry experience, employers don't need to independently confirm that I know how to reverse a string or traverse a tree or define what XSS is. But 'top universities' don't know how to teach CS. That's the point. Don't waste your time there. Maybe we should have a 'bar exam' type system, but the university system is a joke at this point, and there's no point trying to put lipstick on a pig.
VirusNewbie 3y ago
>A complete overhaul of credentialing, so that when I have a degree in computer science from a top university and decades of industry experience, employers don't need to independently confirm that I know how to reverse a string or traverse a tree or define what XSS is. But 'top universities' don't know how to teach CS. That's the point. Don't waste your time there. Maybe we should have a 'bar exam' type system, but the university system is a joke at this point, and there's no point trying to put lipstick on a pig.

euos 3y ago

> A question to everyone: What would a good application process look like? I work in a top company with infamously difficult interview. I made major contributions to several opensource projects that modern tech runs on. I can easily show a clean standalone pull request attributed to me that introduces complex multithreaded code. That code runs daily on millions instances. Why the hell do I need to spend hours answering “what a mutex is”? Also, understand that a lot of stuff is much less obvious and clear cut (e.g. how HTTP protocol works) for me than for your engineer. They only saw the basic case. For me the answer will be “it depends”. Please, respect the candidate time and experience.

lmm 3y ago
> I work in a top company with infamously difficult interview. I made major contributions to several opensource projects that modern tech runs on. I can easily show a clean standalone pull request attributed to me that introduces complex multithreaded code. That code runs daily on millions instances. A lot of applicants will lie or bullshit something similar. How would you suggest an employer distinguish someone like you from someone pretending to be someone like you?
- blincoln 3y ago
  This. Once you've spent enough time on the other side of the hiring process and/or in a leadership role, the reasons behind aspects that seem annoying and trivial as a job candidate become clear. Any interesting technical position posted by a company with a good reputation gets so many unqualified applicants that I wouldn't believe it if I hadn't seen it over and over myself at different organizations. Every HR department I've worked with has done a good job of filtering out the ones that didn't even look good on paper. Out of the ones that get past that stage, I'd estimate maybe 1-5% actually have the technical qualifications for the job. Of course, even if they have the technical qualifications, they may be lacking in other areas, like professionalism. The last three companies I've worked for have all used a custom CTF as one of the interview phases, and it's been invaluable at screening out people who were good at talking about the work, but not good at actually doing the work. So, superficially, I'd love to be able to use something like Starfighter instead of having to maintain one in-house. But how likely is it that a company like that would build truly unique challenges for every customer? IMO it would cause cheating to be easier, because people doing the CTF in bad faith could just work from lists of known elements used in previous CTFs from the same company.
- euos 3y ago
  References. My GitHub profile. LinkedIn.
  - lmm 3y ago
    All extremely easy to fake.

arp242 3y ago

Credentialism is not good either. And in general I've seen little correlation between credentials and quality, both in the software industry and elsewhere, so I don't think any such certifications will really solve anything either.

solardev 3y ago
I think a more nuanced take on this is that very few companies need top quality, but nobody wants bottom quality. Top-end devs make a name for themselves and don't really have to worry about hiring. Companies compete for them with crazy high offers. Bottom-end devs try to make up for their lack of experience or skill by "hacking" the application or interview process somehow. In between, "good enough" devs power the world's bog-standard apps. They're not going to win any awards or lead any talks, but they can do the jobs well enough to make the company money. They know they aren't top-end and can't just say "I'm quitting, who's hiring?" on twitter, but they are also competing with the hordes of less-skilled devs because there's no easy way to tell them apart. I've never heard of "credentialism", but why doesn't it work in this instance, when it does for doctors, lawyers, engineers, etc.?
- arp242 3y ago
  > I've never heard of "credentialism", but why doesn't it work in this instance, when it does for doctors, lawyers, engineers, etc.? There are loads of bad doctors though, and bad lawyers, and bad engineers. "Credentialism" refers to over-reliance on credentials, rather than just looking at what someone can do. I'm not against education, but education is not the same as credentials. People that can hack the interview process can also hack the credentials. I've seen people with college degrees in IT employed as programmers that literally cannot program. I'm not talking "bad programmers" or "brain fart moment", I'm talking "consistently being unable to grasp even basic control flow over a period of months or years". Also some good programmers can be hard to work with for reasons other than their programming ability.[1] Plus not every programmer is suitable for every position or every type of programming or "jibes" with every other programmer in their sensibilities and approach.[2] The hiring process is just as much about that as testing basic skills. No one thinks credentials are a guarantee of quality (people will always slip through the cracks no matter what), but I'm not sure they're even a strong indicator of quality either, certainly not for programming. This is hard to quantify of course, and just based on my own observations over my career. [1]: https://news.ycombinator.com/item?id=21585958 https://news.ycombinator.com/item?id=21585958 [2] Simple example: I'm firmly in team "short names", but I know some people really hate it, but I'm pragmatic if nothing else and don't mind doing whatever works well for the entire team. On one extreme end I've worked in a team that insisted on the longest possible name for everything: every variable had to spell out the full name of the class where it came from, often >20 chars, otherwise people considered it "unclear what anything refers to!" I once added a mkdir() function, this was considered "obscure" so it had to be "CreateDir()", then someone said it may not be clear what "dir" is so "CreateDirectory()", and then another remarked it actually makes all intermediate directories too, so we ended up with "CreateRecusiveDirectoryTree()" – I'm not saying this is "wrong", but it does not jibe well with my sensibilities. Several other things also didn't, so I didn't last too long).
  - solardev 3y ago
    I hear ya. Essentially, credentials aren't a good way to measure -- much less guarantee -- quality? That makes sense. I've encountered several doctors, etc. who weren't very good, including one who shouldn't have been allowed to practice, IMO. Point taken. Anyway, in your experience, what actually WOULD be a good indicator of candidate quality?
  - tl 3y ago
    > I've seen people with college degrees in IT employed as programmers that literally cannot program. I'm not talking "bad programmers" or "brain fart moment", I'm talking "consistently being unable to grasp even basic control flow over a period of months or years". Also some good programmers can be hard to work with for reasons other than their programming ability. Your link talks about "good programmers that lack social skills". In my experience, politeness is a finite resource and dealing with people on a regular basis fully unable to do their listed job is a severe tax on mine.
    - arp242 3y ago
      I wasn't talking about people who are well-intentioned but a bit abrasive if they're having a bad day, I was talking about people who are so unpleasant they "completely crush morale for the rest of the team or company".
- aleph_minus_one 3y ago
  > Top-end devs make a name for themselves and don't really have to worry about hiring. Companies compete for them with crazy high offers. Those who don't have to worry about hiring are not (necessarily) top-end devs, but good self-promoters and networkers. Not every top-end dev is a good self-promoter or good at networking.
  - solardev 3y ago
    Fair point.

xyzzy_plugh 3y ago

Indeed. I think tptacek's perspective on hiring is quite radical, certainly for established shops where doing things differently than the rest of the industry just adds friction elsewhere. My read on the approach is that it works very well for certain types of engagements, but other types, such as where you are trying to build a solid team of peers, requires something _closer_ to the traditional methods. That being said the Starfighter approach is probably right most of the time -- teams are almost certainly never stable, management usually treats ICs as fungible cogs and moves them around, in which case individual value should trump things like team fit or culture. But we're human. Appropaches like Google's work better (n.b. I am far from a proponent of what they do) because the interviewers are human and despite the whole blind-panel thing if the interviewer and interviewee can't get along the interview won't go well. So culture and team fit _is_ evaluated significantly more than something like Starfighter. People care a lot about that implicit bit, in my experience.

tptacek 3y ago
Fly.io is several solid teams of peers, built on resume-blind work-sample hiring.
- xyzzy_plugh 3y ago
  For sure! But the company was built that way. An existing 1000 person corp works differently. Everyone has to be on board for it to work, I'd wager.
wglb 3y ago
I used this approach at a company of about 750 folks, bending HR to bypass some of their stuff. E.g., they insisted on collecting resumes, but I generally discarded them. Instead, I did a one-on-one phone call for the purpose of them to ask me questions--not to grill them, and to ask them if they would like go to the next step and do a work sample test. I tailored the work sample test to the role. I did it for the compliance hires and for the project manager. The results were unexpectedly excellent. As a result of this process, and having a solid company culture, there are around ten or twelve folks that have a new career in the security field that would likely not have had another opportunity in the classical hiring scheme. I think the state of the industry with respect to hiring and evaluating folks is broken across the whole spectrum. It is like once someone becomes established in a field, they invent all manner of gatekeeping tests that they didn't pass to get where they are. The whole story of sabermetrics is another example.

dmoy 3y ago

Yea it's a problem. You're trying to hire from a pool that includes very competent people, and also people who legitimately can't write any code at all and are just trying to sneak in for 6mo to get $100k before they get fired, or think that "fake it til you make it" is a real thing. But at the same time, unlike other engineering disciples there's no PE licensing exams (or whatever the FAA does, I forget), no equivalent degree requirements (for good reason - one can learn to code without it, and a huge portion of a CS degree is very theoretical), but at the same time is qjite difficult, nothing like a plumber/electrician/etc certification level with accompanying work requirements and an apprenticeship program, and has very high pay and demand. For very small shops willing to incur legal risk or limit the application pool, you can do take home tests and/or lean on side projects, with an hour of explaining the nitty gritty details and choices. At scale it becomes untenable, because you open yourself up for discrimination lawsuits, and you can't convince a lot of people who already have jobs and families to do lengthier take home stuff. If I had to bet, I'd bet some day it ends up like plumbing. You work as an apprentice, learn real stuff on the job, and then there's some kind of state certification which itself is a joke, but requires you to have gone through an apprenticeship program already where all the real shit happens. Companies trying to hire "the best" or whatever are still going to have problems, so as long as there's not much regulation increasing cost of dev and uniformity by an alarming degree, we'll still have insane hiring (and insane pay too, regs will make it unprofitable eventually).

jvickers 3y ago
> because you open yourself up for discrimination lawsuits Is that because when looking to make a possibly subjective judgement on the performance in a test and especially what a side project shows, it then becomes more difficult to prove that the judgement was not instead made because of some protected characteristic of the candidate?
- dmoy 3y ago
  That, and a big chunk of it is kids. In the same way you can't decline to hire someone because they're pregnant or may soon become pregnant (or are the father side of that equation) - requiring side projects is a very thin/loose proxy for "doesn't have kids". It's not a big enough problem to really stop people, but a big company's lawyers will stop it internally.
fdye 3y ago
So my $0.02 from being on both the hiring and trying to get hired side of the fence. Also I can't do a whiteboard interview to save my life, never could. I mostly think the process everyone is running makes a bit more sense for entry-level applicants. You are dealing with a candidate pool exactly like what you describe above. However, for anything above a junior dev it is horribly inefficient. Instead I'm always surprised more places don't rely on references and prior experience. Yes people lie, but in my experience its relatively easy to tell the difference with a simple glance at their LinkedIn. If I look at a candidate that spent 2-3 yrs as a Software Engineer at some company that I'm relatively familiar with, and they seem to have 2nds and thirds to people I know in the industry, then pretty good chance they aren't lying. Same for people taking the time out to write recommendations for them. Even bigger signal if they want setup some calls with their prior co-workers who can vouch for them, to me that means they stand by their work and reputation. I recently went through about seven rounds for a senior role. During that time I repeatedly offered to setup some time with my prior coworkers from those I directly managed, to peers, to those I reported to (executive team). My thinking being that they could hear from the horses mouth how I lead a team, my work ethic, etc. They did not take me up on the offer, which to me was crazy. Yes, I could be running some machiavellian scam with 2-3 people who also made a fake LinkedIn, I also could have put up fake articles in PR Newswire announcing my last position, could have spoofed all those blogs I co-authored from the company I worked at 2 jobs ago, etc. But really, wouldn't it be a better signal for a candidate to offer and have all these things? Instead you see an industry that puts someone with ~10 yrs experience through a whiteboard interview. It makes no sense.
- eldavido 3y ago
  I think this is more in line with how hiring works for senior jobs. I've been part of 50+ hiring decisions and do it like this. Social stuff is absolutely the first thing I'm checking. It's a quick test to spot total lies, e.g. you claim you worked somewhere but are connected with zero people on any social network, or claim attendance at a school with zero connection to anyone. If nothing else, it helps to build rapport for the interview. I would 100%, absolutely trust a personal recommendation over a dumb whiteboard interview.
Alex3917 3y ago
> For very small shops willing to incur legal risk or limit the application pool, you can do take home tests and/or lean on side projects, with an hour of explaining the nitty gritty details and choices. At scale it becomes untenable, because you open yourself up for discrimination lawsuits, and you can't convince a lot of people who already have jobs and families to do lengthier take home stuff. At the end of the day, programming is just a writing job. So if you apply for non-technical writing jobs, e.g. to be a columnist at The New York Times or The New Yorker, do they not read any of your preexisting work on the grounds that it could open them up to discrimination lawsuits? Similarly, do folks who hire designers typically not look at their portfolios?
- dmoy 3y ago
  > So if you apply for non-technical writing jobs, e.g. to be a columnist at The New York Times or The New Yorker, do they not read any of your preexisting work on the grounds that it could open them up to discrimination lawsuits? If you work in an industry where nothing is published publicly, and then demand to see people's side blogs and use that for hiring decisions, then yea that'll make lawyers at a big company a little skittish. It's a different thing when your job is to write things that everyone can see, vs having your company own everything you do.
scarby2 3y ago
> or think that "fake it til you make it" is a real thing. Fake it to you make it is actually a real thing. Though you actually have to be able to make it. You can overstate qualifications and experience but not competence. I don't lie about anything but i've definitely known people who have lied in order to get hired who have gone on to perform extremely well.
- dmoy 3y ago
  Sure I mean the extreme form, as in "literally cannot write any code at all" yet getting interviewed at XYZ tech company, and then being shocked when there's a coding interview. Bonus points when they've somehow talked there way all the way onsite, wasting 10+ hours of engineering time, and 48/72+ hours of their own time, before it's discovered that they can't in fact write code at all in any language. (I mean like 1 variable for-loop pseudocode, think fizzbuzz except without any conditional logic, even easier). People absolutely embellish and get away with it.
fragmede 3y ago
Day to day, how much code do you think principle and staff engineers write? Senior architects and team leads? The problem is I don't think we, as an engineering discipline and career path, are willing to recognize that the job of being a coder stops being about writing code! Yes there are deadbeats who don't contribute (some of them can code just fine), as there are in every industry, but until you've been in a project that failed because there was no senior architect, someone getting paid a comfortable salary to be the "senior architect" and not write code sounds like a waste of the company's money to the intern. At the start of a SWE career, I don't think things broken. Study up some leetcode and write code on a whiteboard to prove that you, as a candidate, are smart and motivated enough to play the game. It's after that, that things get out of sync and you get absolutely amazing people who have written world changing software who can't pass your stupid puzzle of a whiteboard interview. And then what do you do? This person failed your test but you want to hire them. Here's the part where we fail, where we are too stuck on following the rules. They failed the test so you can't make an offer? No. Move heaven and Earth to bend the rules, and make them the offer anyway. Don't hide behind corporate handbooks and the HR or the legal department. Obviously this power can be abused, but I don't take it as a given that it must be, given a hiring panel. Google could have made an offer to Homebrew's Max Howell but chose not to, which speaks volumes.
- aleph_minus_one 3y ago
  > Google could have made an offer to Homebrew's Max Howell but chose not to, which speaks volumes. Programming is divided into a multitude of various cultures with their own (often incompatible) rules. Not without reason, you tend to talk of a "Java shop", "C# shop", "Python shop", ... So, when looking for a good programmer, you either look for a beginner who is still "malleable" into your desired culture, or has the same right pedigree as your company. In this sense, I a priori have no doubt that Max Howell is a decent programmer, but I do believe that he comes from and is attached to a very different programming culture than Google's; thus he would likely not be a good fit.
- lmm 3y ago
  > Yes there are deadbeats who don't contribute (some of them can code just fine), as there are in every industry, but until you've been in a project that failed because there was no senior architect, someone getting paid a comfortable salary to be the "senior architect" and not write code sounds like a waste of the company's money to the intern. Oh, if only the worst was not contributing. I've been at a company full of brilliant programmers that was just driven into the ground by the senior architect, who wasn't necessary. Hiring someone who can't Fizzbuzz for that position (and giving him the authority to overrule the development leads) was absolutely the single decision that sank that company.
Apocryphon 3y ago
The threat of litigation does not actually exist in terms of software engineering interviews: https://news.ycombinator.com/item?id=22258113 https://news.ycombinator.com/item?id=22258113
- dmoy 3y ago
  But people have absolutely sued, successfully, for discrimination. Just not in the context of being treated like an adult by an interviewing system that gives actual feedback. Notably virtually no big companies will give feedback by policy, because of their own lawyers. All the engineers on the other side would freaking love to give constructive feedback. Their own lawyers prevent them from doing so.
  - Apocryphon 3y ago
    Source? Which tech companies? Or is it just the amorphous possibility of suits?
    - dmoy 3y ago
      Yes tech companies get sued for discrimination in hiring on a routine basis. The issue with giving feedback is the you hand evidence to the opposing side in a future lawsuit. There's very little benefit for the company, and a lot of risk. Microsoft settled three years ago on a race discrimination in hiring thing. And two years ago for discriminating against non-citizens (and in some twist, Facebook settled the same year for discriminating against citizens, so... damned either way). Just this year Workday got sued for discrimination in hiring, which is mildly hilarious because they're the HR-as-a-service company. > Or is it just the amorphous possibility of suits? It's mostly this, driven by the lawyers. They've seen a few previous settlements and just decide "well, none of that here". I don't know if you've ever done hiring at a big company, or sat through the legal training things, but even if you don't draw a lawyer who tells it to you straight, it's pretty easy to infer what's going on.
      - dmoy 3y ago

sam_lowry_ 3y ago
Sometimes I flip on obvious intreview questions while solving same problems during the flow of programming almost instinctively.
deleted 3y ago
[deleted]
ApolloFortyNine 3y ago
The number of cs majors I've had fail a for loop function floors me. The company I was at was paying 105k starting at that time 4 years ago for junior devs fresh out of college, so not FANG tier but plenty good for the area. At least 30% bombed writing a for loop and if statement though. And we let them pick any language they wanted, so that wasn't it.
- toast0 3y ago
  I've only done interviewing for high tier companies, so my bar was pretty high, I wanted candidatws to be able to demonstrate knowledge and ability of a loop with a loop inside. Our screening was pretty good, so most (but not all) candidates could demonstrate one loop, but an unfortunate number couldn't do the nested loop. shrug
- clusterhacks 3y ago
  I taught an entry-level programming class in a university setting. There were weekly programming practice "labs" and a more involved independent project with several check-ins with me over the semester. I gave an open-book final exam that asked students to write a for loop on one of the questions. I would say around 20% bombed the question despite the very book sitting beside them having a chapter on looping. With examples. After the class ended, I ultimately concluded a non-trivial number of students probably didn't do any of their own coding during the course. I'm pretty sure I have worked with at least two or three people in coding positions who couldn't actually code and relied on outside (spouse? out-sourced?) support.
acdha 3y ago
I hate the way cheaters spoil the well for people with legitimate issues. I always want to be considerate of people with anxiety, processing issues, etc. but I have also seen multiple candidates where everyone thought they were using someone else’s resume (e.g. 10+ years of experience, couldn’t describe _anything_ they did) and were trying to pass it off as ESL issues (our interview panel included a fluent speaker of that language who did not find this credible), and I wish they appreciated that they are making it harder for everyone whose resume looks like the one they used
nonameiguess 3y ago
They're not necessarily "lying" in the strictest sense. I've seen this problem in very large defense contractors like Lockheed and Raytheon that combine two things: 1) a need to hire a lot of cleared developers very quickly when a new contract is won and 2) internal HR processes that make it nearly impossible to fire someone. The former makes it possible for underqualified or even flat-out incompetent people to get hired in the first place just because they're cleared and live close enough to a SCIF and the latter means they stick. At best, they'll do literally nothing and just silently sit through meetings and complete extremely menial tickets a toddler could do in their sleep. At worst, management insists they need to do real work to justify charging for their labor hours, but then they suck so much energy out of everyone else who has to hold their hand and cover their mistakes that their net contribution becomes negative. Nonetheless, it is very easy to transfer them to another team or another project that doesn't know any better and has to take whoever some higher layer of management tells them to take, and much harder to take the time building up the case to fire them, so they stick with the company for a long time. In doing so, they can build quite a long list of bullet points in terms of projects they've worked on, technologies they've used, and what not, even though they never actually learned anything useful from any of them and could not explain what any of the software they worked on actually did or how. This tends to be especially bad in companies that use a matrix organization where your department or career manager is in a totally separate hierarchy from the project-level managers you actually receive taskings from because the latter know your performance traits but the former decide where you get assigned to work and whether you can be fired or not, and they tend to have a huge incentive to staff critical programs, even if "staff" means fill in with net-negative warm bodies, because that's at least faster than filling with new hires, which also may not even be possible when you have very specific location and security requirements.
aleph_minus_one 3y ago
> I don't think very many players did all the Starfighter trading challenges. patio11 or tptacek might be able to confirm, but I vaguely recall being one of like 100 or so to do all six levels of out of 10,000 that started As far as I remember, at the beginning and for quite some time the Stockfighter trading challenges had reliability and performance issues (server capacity). So I decided to wait a little bit for the product to become more stable, and then get into the challenges. Unluckily, at that time Stockfighter was already decided to become shut down.
- elptacek 3y ago
  Invisible, forgotten third partner here... more made it through all of them than you might think. They even set up their own Slack community to help each other out.
  - clusterhacks 3y ago
    Apologies! Not invisible or forgotten!! Do you remember any rough numbers on started vs finished on the two CTF tracks y'all put up? FYI, the six stockfighter challenges were some of the most fun I ever had hacking around. I enjoy puzzle games like Exit and happily spend money on them. I would probably have paid to play the starfighter challenges even though the embedded one wasn't particularly my cup of tea. I wonder how many other nuts like me might have happily paid a monthly subscription for a few releases per year . . . I also remember you saying in an early thread that clients' hiring processes were a mess and I think implied most treated Starfighter as just another candidate funnel. Did any of your clients take up a challenge/CTF process after working with Starfighter? I know Thomas says hiring with challenges/CTFs is a competitive advantage but I'm not so sure. If it was true, wouldn't we see more of it out in the real world?
    - elptacek 3y ago
      No, we never really got far enough that we were able to collect that kind of data. I'm glad it was fun; certainly doing it for a living is entertaining. Mostly. IMO, the messiness of hiring processes and the lack of adoption of work sample testing have a lot to do with each other. One of the questions I regularly ask in interviews (and have for a long time) is, "What are the first 5 things you'll have me work on?" During the last round of interviews, no one I talked to could answer that. I even had a couple of interviewers who didn't even know the name of the hiring manager or the position. Then you have other artificial filters like preferred certifications where the assumption seems to be that whatever training was required to pass can be mapped onto every organisation... or that simply possessing a cert magically grants insight into how most companies do stuff. The point is that companies don't really seem to have a good idea of what they're actually hiring for, but they get by hiring more senior people who have enough experience (hopefully) that they can figure it out on their own. One of the things we discussed early on was avoiding having Starfighter turn into a certification process and maybe that's one of the reasons why we stumbled around a bit. I am aware that certifications have their place, but they're not really a substitute for industry experience and they definitely aren't a guarantee that a candidate can quickly figure out how a company's tech is glued together. IMO that's the power of work sample testing: you're giving a candidate a challenge where they can demonstrate aptitude. But if you don't know what actually needs to be done, you can't create that challenge, much less a rubric for describing expectations.

aleph_minus_one 3y ago
> If all TripleByte does is find candidates who pass your process, you can't simplify your hiring process without affecting the quality of the candidates you hire through references from TripleByte. If TripleByte is capable of finding candidates who would pass a specific, described process, you theoretically can omit the whole process because by definition, you know that the candidate is able to pass it. In practise, for good reasons, you don't want to do this, but it already save quite some money to shorten the process or just do plausibility checks. If these plausibility checks make you realize that TripleByte was overselling (or to put it much more directly: are fraudsters), this should hopefully cause sufficient reputation damage for TripleByte that they go bust; thus TripleByte hopefully has a strong incentive not to oversell.
- thaumasiotes 3y ago
  > If TripleByte is capable of finding candidates who would pass a specific, described process If you (as a hiring company) can describe your hiring process accurately enough that it's possible to specify it, you don't need recruiting at all. You can just take applications directly. What you're imagining is not related to the problems TripleByte faced. If your goal is, as TripleByte's was, to find people who can pass a particular company's hiring process, the only way for you to know what that process is like is by experimenting with it empirically. > you theoretically can omit the whole process because by definition, you know that the candidate is able to pass it. Once again, you (aleph_minus_one) are describing the goal of "find candidates who are good at the job", not the goal of "find candidates who will pass the hiring process". > If these plausibility checks make you realize that TripleByte was overselling (or to put it much more directly: are fraudsters) No, that's just what it looks like to you (the company trying to fill spots). The problem people are attempting to address is that the company's hiring process is unrelated to what the company wants in a candidate. You are the fraudster; TripleByte in this scenario is the scapegoat for the bad job you do at candidate evaluation.

clusterhacks 3y ago
Thanks!! No surprises here from what I read, but definitely appreciate your thoughts. Got a short list of companies in addition to Fly.io that are using a hiring process you like or love?
neilv 3y ago
Excellent thoughts, thank you. > We took things very personally. You build relationships with your earliest users (candidates, in our case). Recruiting funnels at our clients had no such connection. The impedance mismatch was demoralizing; our lived experience was that every time a candidate we presented to a client got rejected, we had ourselves failed (I mean, really, we had!). Similarly, the idea that we were just another hoop that candidates had to hop through (but only the non-name-brand candidates who didn't know how to skip us to get an interview at a client) grated on us. That sounds rough. If you could do it over, do you see a way you could get more buy-in from some clients, such that they commit to "advanced placement" in their process, to candidates you provide? Or are pretty much all companies going to have their entire hiring process set in concrete? What about interfacing at the hiring manager level, at companies where the hiring manager normally has some discretion to vouch for the candidate and bypass a lot of process? Could you get sufficient trust with those? Still the rejection problem?
- tptacek 3y ago
  No, I don't think we could ever have "fixed" hiring at companies with enough scale to be lighthouse customers for a business like Starfighter, which was I think a conclusion all 3 of us simultaneously landed on. The engineering team has to be committed to doing things that way. For instance: prior to Starfighter, I hadn't had the misfortune of working in a "leveled" engineering team (I'd worked at big tech companies before, but "levels.fyi"-style leveling, which is universal now, wasn't a thing back when I was still working at bigcos). Work-sample hiring doesn't just impact how you make yes/no decisions on candidates, but also how you level them. If I was going to do it over again --- and, again, I wouldn't, because I feel like I also learned this just isn't a great business to be in --- I would have done something much closer to "Stack Exchange, but for CTFs". Run a couple of pure investment years, build up a base of users and some public profile, and then do lightweight recruiting stuff (probably just curated job ads for awhile) on top of that.
JSeymourATL 3y ago
> But it didn't change anything about how those companies qualified candidates... Turns out, finding/discovering hidden tech talent is relatively easy. Finding the "Economic Buyer", a true hiring executive with budget and authority to make timely hiring decisions gets worse every year. Bureaucratic Red Flag, when HR Bozos start 'adding value' to the process. As companies grow, they mindlessly get in their own way.
zaphar 3y ago
I remember you and Patrick hosting a meetup of HN folks once where we talked about Starfighter. I remember thinking that it was an excellent way to hire but every place I've been I've had trouble getting management and HR to buy into it as a strategy. Even when I made it up to VP I still made little to no headway. It's very hard to shift companies that are eigher larger or with more internal history than a startup to try something like this.
clavoie 3y ago
For those of us that missed out, were the CTFs ever documented? Or otherwise recorded somewhere to see what the hoopla was all about? Curious minds inquire ;)
- danielvf 3y ago
  You basically got Stripe level documentation (high praise) of an API endpoint that would let you get open trade offers, and place trade offers. There was a minimal UI that you could manually operate the system with. Actually understanding what was happening in the markets required you to write something to visualize it with data from API calls. Each level had an increasing level of what you needed to do, with the first level being to buy some stock (which you could do from the UI). Most levels had you operating some kind of marketing making system, where you had to constantly update passive positions that made money as bots traded against them, and the prices shifted. The last level required you to "hack" the system to get info you weren't supposed to have, in order to identify the insider trader in the middle of 50 other bots. It was quite different from all the other levels. Here was the level solving code from my second run through: https://gist.github.com/DanielVF/d43e079050d1cf3c05d6 https://gist.github.com/DanielVF/d43e079050d1cf3c05d6
  - clavoie 3y ago
    Thx!

lmm 3y ago
> What do you mean by the right class background? If you think I’m a white guy This is why I find it so frustrating to talk to Americans. > Let’s just say when I walk through my neighborhood I’ve read comments about me in my own NextDoor group about someone “suspicious” and realized they were talking about me or my son… So you're a member of the class that lives in "good" neighbourhoods where it's suspicious for certain types of people to be? I think you've proved my point.
- scarface_74 3y ago
  I think you’re more showing your own biases. I never said that the neighborhood was “good”. I just said that there aren’t that many people with my skin color… But back on topic. I base my interviews on behavioral traits that for framing, can be sussed out by asking “tell me about a time when” type questions. You did see the part about I’ve gone through the interview training by the second largest employer in the US didn’t you? It doesn’t matter if you can reverse a binary tree on the whiteboard if I’m looking for someone who “takes ownership”, “digs deep”, and has shown the ability to deal at the level of “scope” that I’m looking for an knows how to “deal with ambiguity”
  - blincoln 3y ago
    I like your "tell me about a time when" approach,[1] and I've used it successfully in the past, but I've never seen a good way to formalize it. Have you ever been able to delegate that responsibility to a team of other people and get results more or less consistent with what would have happened if you'd conducted all of the interviews? My current opinion is that while it's effective, it means there's a significant bottleneck (sometimes a single person) for the hiring pipeline. i.e. you still need at least one earlier tier of interview to winnow the candidates down to a volume that's manageable for the bottleneck, and the earlier tier needs to be something more standardized that can be delegated to a larger group of people. i.e. "can this person explain common technical details A, B, and C of work in this department well enough to demonstrate solid understanding?" [1] I read years ago that border guards in some countries use a similar technique, because if someone is being deceptive, it's unlikely they can handle a random walk to an arbitrary depth of detail.
    - scarface_74 3y ago
      I have a framework now that I’ve been developing based on my time at AWS. I’m now at a company where I’m building out an “application modernization” practice. It’s based on the Leadership Principles at Amazon. Not that Amazon is a great place to work in practice (I just got Anazoned two months ago). But they talk good game. https://managementconsulted.com/amazon-leadership-principles/ https://managementconsulted.com/amazon-leadership-principles... I have a sticky note of the LPs I care about. A lot of times when I just ask a generic “tell me about yourself” question, they will talk about their experience and I can just ask them more details around their project that let’s me get the information I want. Other times, I have to specifically pull out my questions. For instance, I really care about having someone who is willing to speak up in meetings and is willing to express their opinions and even someone who is willing to tell me they think I’m wrong. I speak very authoritatively about my areas of expertise. But I always try to emphasize “strong opinions weakly held”. In LPs terms it’s called “Have backbone disagree and commit”. My favorite question for that is “Say I’m your manager and you are in the meeting with me , the CTO and other managers and we come up with an idea that you think is completely stupid. What would you do?” And that starts an entire conversation. I’ve given a thumbs down to people who give me an answer that I don’t think is a fit for what I’m looking for. I love for people to give me feedback and I hire people who help my team fill in technical gaps. I need people to speak up
    - aleph_minus_one 3y ago
      > [1] I read years ago that border guards in some countries use a similar technique, because if someone is being deceptive, it's unlikely they can handle a random walk to an arbitrary depth of detail. Such people are not necessarily deceptive; in my experience, much more often - the person does not have the best memory, but loves to talk - the person has a tendency to "hallucinate" (i.e. has some mild (typically harmless) schizophrenic traits)
  - lmm 3y ago
    > But back on topic. I base my interviews on behavioral traits that for framing, can be sussed out by asking “tell me about a time when” type questions. > It doesn’t matter if you can reverse a binary tree on the whiteboard if I’m looking for someone who “takes ownership”, “digs deep”, and has shown the ability to deal at the level of “scope” that I’m looking for an knows how to “deal with ambiguity” That definitely sounds like something that would filter more for "went to the right schools" (or, more generically, "mingled with the right people", but schools are the easy way). I could certainly believe it's as effective as "reverse a binary tree", but I very much doubt it's fairer.
    - scarface_74 3y ago
      How does “going to the right school” have anything to do with having a project at work that you took ownership of and you took the time to learn a new to you technology? Do you realize that scope, impact and dealing with ambiguity is literally in the leveling guidelines stated in some way or the other in every tech company? https://dropbox.github.io/dbx-career-framework/ https://dropbox.github.io/dbx-career-framework/ I’m not looking for someone who just graduated from college.
      - lmm 3y ago
        > How does “going to the right school” have anything to do with having a project at work that you took ownership of and you took the time to learn a new to you technology? The right schools give you the opportunity and mentality to do that (and before that, even more fundamentally, so does having a comfortable upbringing), and then you'll compound that with your experiences in the good jobs that going to the right schools gets you. > Do you realize that scope, impact and dealing with ambiguity is literally in the leveling guidelines stated in some way or the other in every tech company? Everyone does it, that's exactly the point - you limit class mobility by ensuring that the good jobs are only available to the people from the right class. Even if someone from the lower classes has the skills to excel, they never get the chance to take ownership, learn new skills, and expand the scope of their impact because they're never hired for a position in which they can take ownership, take the time to learn, and have impact beyond their narrow position, because they never have a way to show that they can take ownership and learn new skills...

scarface_74 3y ago
What do you mean you don’t get the chance to learn new skills? There are plenty of free resources on the internet to teach yourself new skills. When I’m looking for the behavioral traits I wouldn’t care if they could only tell me stories about being the shift lead at McDonalds. You realize I only got a job at a company that anyone has ever heard of at 46? It’s called “work ethic”. Right now, I am building a team where the actual coding is really simple in the grand scheme of things. Since ChatGPT is well trained on the AWS SDK, the CDK, etc. you could literally use it to do 90% of the work. But the organizational complexity, business rules, etc are complicated and we are still figuring everything out. I need someone with the following behavioral traits. I’m going to frame them in terms of the Amazon LPs since it gives everyone a publicly available point of reference: - Customer Obsession - Ownership - Learn and Be Curious - Bias for Action - Dive Deep - Have Backbone; Disagree and Commit - Deliver Results If you have any kind of work experience at the mid or senior level you should be able to demonstrate those. I could answer “tell me about a time when” type questions three years out of college graduating from my no name HBCU in the south after my first job.
- lmm 3y ago
  > What do you mean you don’t get the chance to learn new skills? There are plenty of free resources on the internet to teach yourself new skills. Which is fine if you've got a job with reasonable hours that doesn't leave you drained at the end of it, and the rest of your life is stable enough for that, and... > I ... got a job at a company that anyone has ever heard of at 46 > It’s called “work ethic”. Maybe. Or maybe it's luck. People are particularly bad at assessing which when it comes to their own history. > I need someone with the following behavioral traits. Or you need someone with something else that's correlated with them. Or what you need isn't really related to them at all. > If you have any kind of work experience at the mid or senior level you should be able to demonstrate those. Right. Once you've made the leap into the good jobs it's easy to stay there (and if you went to the right schools then it's easy to get the first good job, and if you're born to the right family then it's easy to get into the right schools). The trouble is getting there at all. > I could answer “tell me about a time when” type questions three years out of college graduating from my no name HBCU in the south after my first job. Sounds like you had a good first job.
  - scarface_74 3y ago
    > Which is fine if you've got a job with reasonable hours that doesn't leave you drained at the end of it, and the rest of your life is stable enough for that, and... It sounds like you’re making excuses. You mean you spend every hour of your life working? Are you suggesting that we should have no hiring criteria? > Maybe. Or maybe it's luck. People are particularly bad at assessing which when it comes to their own history You did see the part where I would find it perfectly acceptable to show the behavioral traits by working any job? Of course you do have to have technical experience - not even in the stack we are using. I can teach the specific technology. > Sounds like you had a good first job. My first job was a computer operator working on mainframes.
    - lmm 3y ago
      > Are you suggesting that we should have no hiring criteria? > You did see the part where I would find it perfectly acceptable to show the behavioral traits by working any job? I'm suggesting that hiring criteria that reflect more directly on the candidate's abilities, and less on what opportunities they've had to date, are fairer. (And, distinctly but relatedly, the more objective the assessment is, the better for those from lower-class backgrounds). For someone who's grown up in a working class household where showing initiative had negative consequences and then gone through public schools where showing initiative had negative consequences through to line-level jobs where showing initiative had negative consequences, a tech interview that focuses primarily on whether they've got the skills, and lets them show it through their fizzbuzz/leetcode/hackerrank/what-have-you, is one of the few ways to make it out.
      - scarface_74 3y ago
        A few issue: 1. People who were too busy to study to learn how to code would definitely be too busy to grind leetcode as well as adults who are already in the industry with families 2. Okay you can reverse a binary tree on the whiteboard. That tells me nothing about whether you are the kind of person that I can assign a task like “a customer has some vague idea of what they want. I need you to figure out the details and come up with a proposal and a design and present it to them and get feedback”. Then if they don’t know the technology, I need them to be able to learn it. This is what I mean by “Customer Obsession”, “Learn and be curious”, “Digging Deep” and dealing with ambiguity. If I need them to lead a project, I need to know they are able to handle items dealing with “scope”. I want to know if they have a history of “delivering results “. These are all traits that you were required to show to get a return offer as an intern for a full time job as an L4 consultant working in Professional Services at AWS. I threw tasks like this at a first year employee who I had mentored as intern and they passed everything I threw at then with flying colors. At my current job, I’m working with a green field project where everything is ambiguous and I need someone I can throw a poorly defined objective at and they can take the ball and run with it. Leetcode isn’t going to tell me anything.

lmm 3y ago
> 1. People who were too busy to study to learn how to code would definitely be too busy to grind leetcode as well as adults who are already in the industry with families I think many more people have the opportunity to grind leetcode than have the opportunity to "take the time to learn a new to you technology" that they can then apply to a work project - which requires not just time but also having the right work environment in which they can apply it. Of course you're right that many people don't have the time for either, and a handful of professional-managerial class people who are already in good jobs might be able to do the latter during work hours but not the former. > whether you are the kind of person that I can assign a task like “a customer has some vague idea of what they want. I need you to figure out the details and come up with a proposal and a design and present it to them and get feedback”. > I want to know if they have a history of “delivering results “. Do you think you're measuring something objective about their ability to do the job? To the extent that you are, do you think what you're measuring is something different / better than "has decent general intelligence + has existed in professional-managerial-class spaces (or, more specifically, in jobs like this one) for a while?" Do you have evidence about the answers to either of those questions?
- scarface_74 3y ago
  > I think many more people have the opportunity to grind leetcode than have the opportunity to "take the time to learn a new to you technology The difference is that you are an active programmer, you’re actually programming. I graduated in 1996. I had to learn “new to me technology* as part of my job for 25 years on the job. I’ve never had to do anything approaching leetcode on my day to day job. I’ve learned on the job: - infrastructure and networking - C++ - Perl - JavaScript - VB6 - C# - Python - litterally 3 dozen AWS technologies - databases (OLAP, OLTP, key/value, document, etc) - how to manage large projects And pick up soft skills > Do you think you're measuring something objective about their ability to do the job? The best predictor of future behavior is past behavior. That’s where behavioral question comes from. > To the extent that you are, do you think what you're measuring is something different / better than "has decent general intelligence None of the behavioral traits I mention have any correlation to intelligence. I didn’t wake up one morning and learn how to communicate effectively .
  - lmm 3y ago
    > I had to learn “new to me technology* as part of my job for 25 years on the job. Right, I'm saying that's a rarer privilege than being able to do leetcode or similar (whether during work hours or not). > None of the behavioral traits I mention have any correlation to intelligence. I didn’t wake up one morning and learn how to communicate effectively . If you think being able to communicate effectively has nothing to do with intelligence you're crazy. (Of course, it probably has have even more to do with having a common class experience than with intelligence)
    - scarface_74 3y ago
      > Right, I'm saying that's a rarer privilege than being able to do leetcode or similar (whether during work hours or not). How long have you been working? I doubt very seriously that anyone who works professionally hasn’t had to learn some new technology on the job. Out of the 2.7 million working software developers, I could say that almost all of them had to learn new to them technologies on the job. Also almost none of them did anything approaching leetcode on the job. > If you think being able to communicate effectively has nothing to do with intelligence you're crazy. (Of course, it probably has have even more to do with having a common class experience than with intelligence) What exactly point are you trying to make? That if I need someone who can communicate affectingly I should test that by using leetcode? Yes, you can learn how to be an effective communicator much easier than you can learn how to pass a coding interview.
      - lmm 3y ago
        > I doubt very seriously that anyone who works professionally hasn’t had to learn some new technology on the job. Out of the 2.7 million working software developers, I could say that almost all of them had to learn new to them technologies on the job. Sure. It's a great way to hire people who already have professional-class jobs, mostly already software developers. > Yes, you can learn how to be an effective communicator much easier than you can learn how to pass a coding interview. People from the "wrong" class aren't generally ineffective communicators, they just have a communication style that's adapted to their circumstances. PMC communication styles are easy to learn if you have access to an environment where everyone else is using them, but very hard otherwise. (Also, by your own logic if the communication style is easier to learn wouldn't you want to hire people for the harder skill and teach them the communication style on the job?)