6 ms·
Hackerrank was broken – but now it's harmful
- yieldcrv 2y agoI think its time for the federal government to get involved, and it already has leverage If your company benefits or plans to benefit from QSBS tax treatment, then evaluate engineers on things they’ll be doing in the company: the application development process, sprint ceremonies, asynchronous communication and tooling they wont be implementing data structures often, and if they are reinventing the wheel they are wasting company time, if your company needs a more efficient data structures for its super scalable problem - it should trigger a process audit to be honest.
- 123yawaworht456 2y agolooking forward to Leetcode Prohibition Act of 2029
- henry2023 2y agoFirst let them get rid of non compete agreements. Haha.
- yieldcrv 2y ago2025 lets go, new administration is up for anything and has no clue what theyre doing The center is unguarded I bet I could get the 119th congress to pass anything since all the moneyed interests went to far extremes of their parties and will never get their causes passed the filibuster
- jitl 2y agoI don't understand how QSBS has anything to do with interview practices. Are you saying the IRS should somehow audit any QSBS company to make sure they're not implementing any data structures or algorithms? Like, new companies must be audited if they try to use computer science?
- yieldcrv 2y agoControl behavior by regulating the intermediary, constitutional under any governance system, my favorite approach to anything is to look at what the thing relies on, and controlling access to that thing. We can even add whistleblower bounties to current and prospective employees. The IRS already has a bounty program, probably doesn't even need Congress for this one. Its under the Treasury Department right? Can probably still Scott Bessent on linkedin about implementing it on Day 1 So, yes. You got it. Play devil’s advocate all you want, I’m ready. More likely you’ll have to put your reservations in during the comment period.
- jitl 2y ago"Let's effectively prohibit all non-CRUD apps" that's hilarious. "If your startup needs to solve hard computer problems, you're going to get an IRS audit". wild. It's like the 90s cryptography export laws, except it applies to domestic development too!
- yieldcrv 2y agoit can promote training employees again instead of “screening against false negatives” or whatever dumb interviewing rationale needs to be eradicated
- naet 2y agoThe author says "whiteboard tests" are broken, but it seems like they're arguing that online coding assessments are broken, not in person interviews using an actual whiteboard. Doing an in person interview on a whiteboard sidesteps the AI issue. As someone who's done a large number of remote interviews, there are some clear signs that some candidates try to cheat online tech interviews. I wonder if the trend will fuel more of returns to the office, or at least a return to in-person interviewing for more companies.
- bsder 2y ago> or at least a return to in-person interviewing for more companies. This has been broken for a while now, and companies still haven't reset to deal with it. The incentives to the contrary are too large.
- unavoidable 2y agoThe disincentives are huge though. Hiring a bad employee is a very expensive problem and hard to get rid of.
- ipaddr 2y agoIsn't it as simple as going on pip for fangs, a short conversation for a founder of a startup and a few weeks notice pay?
- viraptor 2y agoThat comes after the decision that you can't fix the situation, which comes after you discovered that the hire was bad, which comes after a number of visible failures. That's a lot of wasted time/effort, even if the firing itself is simple.
- gopher_space 2y agoThe cost of hiring, firing, rehiring approximates the position’s yearly salary.
- jorblumesea 2y agoDon't onsites or virtual onsites also have coding rounds? would be pretty easy to figure out who is using llm and similar tools.
- rich_sasha 2y agoIME you screen people online, get some reasonable candidates, get them onsite and they can't code to save their life. But it's your problem then.
- LeftHandPath 2y agoI recall having to implement A* to search a nxn character grid in my AI course a few years ago. It took me close to a full day to wrap my head around the concepts, get used to python (we usually worked in C++), and actually implement the algorithm. Nowadays, an LLM can spit out a working implementation in seconds. I think that's a big part of the issue with tests like Hackerrank - LLMs have been trained on a lot of the DSAs that those questions try to hit. Whereas, if you ask an LLM for a truly novel solution, it's much more likely to spit out a garbled mess. For example, earlier today, Google's search AI gave me this nonsense example of how to fix a dangling participle: > To correct a dangling participle, you can revise the sentence to give the dangling modifier a noun to modify. For example, you can change the sentence "Walking through the kitchen, the smoke alarm was going off" to "Speeding down the hallway, he saw the door come into view". LLMs have effectively made it impossible to test candidates for crystalline intelligence (e.g. remembering how to write specific DSAs quickly) remotely. Maybe the best solution would be to measure fluid intelligence instead, or front-load on personality/culture assessments and only rigorously assess coding ability in-person towards the end of the interview cycle.
- badgersnake 2y agoSo if I asked you how A* works in an interview you you be able to explain it. Johnny ChatGPT would not.
- deprecative 2y agoIn most cases it really doesn't matter. You wanted A* and you got it. Understanding isn't important if the product works.
- nkrisc 2y agoSeems a problem that will sort itself out if companies are in fact hiring under-qualified cheaters.
- acjohnson55 2y agoThe dirty secret of hiring processes is that the main goal of the earliest stages is to get from a lot of applications to something less than a lot, not to screen for fit. This has long been true, and it is now super true. Every job posting gets flooded with hundreds of applicants, if not into the thousands. Most of those people are coming through the front door with no one to vouch for them and probably nothing on their resume that makes them a must-see candidate. Most managers and HR teams probably don't even explicitly think about it this way, but by pure pragmatism have evolved processes that act as flow control. The unacknowledged result is that the company will reject 90+% of applicants, regardless of fit, under the assumption that the filtering process will allow enough good people into the actual interview rounds that the team will be able to find someone they want. From this perspective, Hackerrank is not broken, it's doing exactly what is required of it by companies. I say all this because people who are in job search processes should frame the process accurately in their mind. It hopefully will help with not taking the process so personally or not getting so infuriated with it. It may also help you strategize how to find your way into the roles you want, if crushing these tests isn't your strong suit. People who are vouched for get to bypass all of this. The more confidently you are vouchced for by a trusted party, the more benefit of the doubt you get in the hiring process. One might ask is there a better way to do this? Probably so. But if it were easy, it would already exist.
- drjasonharrison 2y agoThere would need to be an easier and cheaper way to filter applications. One option for candidates is networking, this gets you in through the "vouched" side door. This potentially means that the company should be encouraging employees with more than financial incentives to find candidates and recommend them. This means networking workshops, time swaps for attending networking/recruiting events, understanding the need to make synchronous contact with people who might be good candidates. If you are interviewing based on what you do, and what the job application (which has been mutated to get through HR's posting requirements, but don't understand why and how you should be interviewing you are more likely to bring your past experience and biases to the interview. This is bad for your company and for candidates.
- 2y ago
- alephxyz 2y ago>Since everyone started using AI, more candidates started clearing the first round with flying colors. The platforms had to recalibrate to let in their target percentage. Is there any proof or data on this? Not saying it's wrong but I'm curious how big the effect actually is.
- gitremote 2y agoI'm not convinced it would be caused by AI. More people are laid off, so more unemployed people can grind leetcode all day, and more people are competing for fewer jobs.
- Eumenes 2y agoi screen candidates all the time (not involved in technical rounds) and notice chatgpt cheating all the time but its easy to spot because the candidates read off answers in mechanical ways right after a flurry of typing
- clark010 2y agoAI is likely to shape the future, gradually embedding itself across industries, fields, processes, tools, and systems. Similar to the introduction of electricity, where identifying practical applications was inevitable, trial and error will be an essential part of its evolution. While concerns about its impact are understandable, it's important to recognize that each generation often worries about the capabilities or standards of the next. However, such concerns are typically rooted in current frameworks of thought, which may not remain relevant in the future.
- swatcoder 2y agoRight now, generative AI looks to be more like microwave ovens than electricity -- usefully novel but not obviously revolutionary or all-pervasive. Even with the remarkable convenience and new opportunities that microwave ovens brought to kitchens, it still made a lot of sense for commercial kitchens and culinary schools to keep focused on more traditional fundamentals and their mastery when hiring and training candidates. Same applies to our field for the foreseeable future.
- Yoric 2y agoI like this metaphor. I suspect that GenAI will eventually reach a point where it's actually really useful, but we aren't quite there yet.
- deleted 2y ago[deleted]
- deleted 2y ago[deleted]
- paxys 2y agoIs it already that time of the week? People were complaining about whiteboard coding interviews in the 1990s and they are complaining about it today. Meanwhile the tech industry has managed to hire millions of incredibly smart people and done pretty well for itself in that period. The interview process isn't going to change. There is no reason for it to change. Your options are to either suck it up and brush up on basic data structures and algorithms or interview at places that don't test for it. Just don't hold your breath waiting for the world to change for you.
- 1123581321 2y agoThat’s not what this article is about. It actually suggests people who love whiteboard interviews are losing here.
- amatuer_sodapop 2y ago> Your options are to either suck it up and brush up on basic data structures and algorithms or interview at places that don't test for it. The article is about the fact that if you did just this, you're at a disadvantage because people use Chat GPT to cheat on these and get ahead.
- vunderba 2y agoThe article is bizarrely conflating whiteboard interviews with online coding tests and even admits such: Some people have (rightly) pointed out that I'm interchanging whiteboarding (writing on a physical whiteboard) and online coding tests interchangeably
- amatuer_sodapop 2y agoYour point being? I'm not commenting about the white boarding.
- patchymcnoodles 2y agoI hired hundreds of people without coding sessions, because although you are right, they also hire a lot of bad people, who are just good at selling themselves. As bigger the company gets, as easier it is to deal with the unnecessary cost of wrong hirings, it's just a number game. So no, I didn't wait for the world to change for me, I changed it myself ;).
- cute_boi 2y agoplease bring in person interview. I am sick and tired of cheaters. They rig every thing ....
- snickell 2y agoTL;DR: best case LLMs break algo-whiteboarding interviews so visibly we stop using them (they're already invisibly broken). Some engineers are genuinely better at using ChatGPT and friends to solve problems, and that's a skill worth including in evaluation of potential software engineers. I would advocate interviewers go with the flow, allow use of LLMs, but have candidates do it on the "shared screen" to see how they think - just like using a web browser in interviews. I've been a professional programmer since 25 BAI (25 years before AI). I see junior engineers who use it disastrously to create piles of messed up spaghetti code, and junior engineers who use it skillfully and judiciously to create good code and solve problems far above their years of experience. Both of them will use LLMs in their daily work. One of them will destroy the codebase, and the other will far exceed the expectations we had for a "2 yoe eng" only 2 years ago. I want to identify the latter, and filter out the ones that aren't good with ChatGPT and will use it to make messes. I want to identify experienced Senior Engineers who kick ass. They can do it with or without ChatGPT, I don't care which, and I don't want an interview format that doesn't let those who use ChatGPT show their true strength. I simply don't care how well you can recreate (and pretend to have rediscovered lol) PhD thesis algorithms with no internet access on memory alone: that is not a relevant skill in 2024. I won't cry if "allow LLMs but watch how they use it / think" doesn't scale to "no skin in the game" offline tests where interviewers try to get candidates to spend an hour without the company matching that hour with an hour of employee/interviewer time. In fact, if (and it probably will) this kills offline testing, I think that's a win for labour. What ChatGPT "cheating" reveals, imo, is how weak and artificial "whiteboarding" interview questions are: they claim to be problem solving tests but they're actually algorithm memorization tests. Interviewing without allowing use of LLMs is like requiring "no web browsing during interview": it artificially biases toward test-specific skills and away from real world strengths. If an LLM can straight up solve the problem, its not a very good match for skills required in 2024: its now a problem that SHOULD be solved via LLM. It turns out that LLMs are better at rote memorization than we are. Good! The "whiteboarding algos" interview technique was already a bad match to real world SWE skills except in relatively rare job roles, and its a good thing that its artificiality caused it now be irreparably broken by ChatGPT. Bye bye, won't miss you ;-) Most bog-standard software engineering work is boring things like good variable naming, effective use (and non-overuse) of encapsulation, designing a schema that won't make you hate life in 5 years, wise selection of dependencies, etc etc. Testing live with a candidate, watching them use ChatGPT, seeing how they think, this is what gets you a job-relevant-skill signal.
- vunderba 2y agoI've said it before and I'll say it again here - the best job interview I ever had for a software engineering position was about 10 years ago. I was shown a piece of code (directly from their codebase) which was about 50 lines long and we had a free wheeling technical conversation around it in which the interviewer asked me the following questions: - What does this code appear to do (both technically and in the context of the business domain of the company) - Are there are any red flags, edge cases, etc. that this code might not cover? - If I were to rewrite it in the using modern libraries/tooling/best practices, what might I have done differently? High-level - we're not looking for you to physically write syntactically correct code here. This is the same interview process that I now use on prospective applicants. In a single 30-60 minute dialogue it helps me answer questions such as: - Can they size up and summarize existing unfamiliar code bases? In enterprise the ability to deal with legacy code is often more common than writing new code. - Can they re-interpret old code using a modern lens? For example if we're discussing JS/TS (requires vs imports, callback hell vs async await, XMLHttpRequest vs fetch). For python (nested lists vs dataframes, simple numpy regression vs pytorch, etc) - Are they able to communicate and convey their thoughts in a coherent and logical manner?