"The ideal candidate is somebody deeply familiar with the tech stack we are currently using, preferably having spent at least 6 months working on our code base.

"The ideal candidate is somebody deeply familiar with the tech stack we are currently using, preferably having spent at least 6 months working on our code base." Seriously though, the performance of an interviewee in such conditions are mostly a function of how well the interviewer can "onboard" the interviewee within 4 hours, or how much the interviewer is willing to actively guide the interviewee to the solution (as opposed to letting them figure it out). Even with a single interviewer this easily introduces bias. I've interviewed many many people over the years, and sometimes I wonder whether I'm giving slightly more hints or less hints based on how subjectively I "like" the candidate. I mean, I try hard to be fair, but I don't follow a strict script, and the variability occasionally makes me doubt myself. For such open ended tasks as 4 hours of essentially pair programming, I don't know how any interviewer could be objective and fair. Especially presuming that you wouldn't be re-using the same task once it's actually been solved/fixed... (otherwise that's just another artificial problem)