I'm enough of a domain expert in certain niches that I've participated in creating certification exams. The reason it can develop into a negative signal: the qu

I'm enough of a domain expert in certain niches that I've participated in creating certification exams. The reason it can develop into a negative signal: the question banks are rapidly exhausted by the exam dump industry, and for even some esoteric exams 90%+ of the question bank with exactly correct answers are generated within a year of the exam's introduction. With the popular exams, it is mind-numbingly fast. Think days to weeks. Crowdsourcing / MTurking the question banks is depressingly effective if you sweated over nearly a thousand question bank items for a few months with a few other expensive experts. Labs in the exams are a way to blunt this, but I think the exam dump industry has probably come up with a way to defeat labs as well by now, because the labs run a fixed set of scenarios with variations in parameters but not the general gist. It's better, but still hackable. What I've come up with requires discipline by a team documenting the issues they've fixed in the past, but it is so far 100% foolproof. Pick a random relatively self-contained issue your team fixed in the past; this is the most critical step governing the quality of this technique's results. Reproduce the issue in a scratch environment. Sit down the candidate in front of a workstation set up to their preferences you gathered in advance, with all the tooling they prefer, simplifying your superfluous environment specifics where possible (like logging them into various accounts ahead of time). Set their expectations ahead of time that they will be team / pair troubleshooting for half an hour, that you aren't evaluating whether they find the root cause or not, you're evaluating how well they work with others troubleshooting an issue in some system/language/etc. they claim they are a domain expert in. Dive in. Without fail, the ones who exam dumped their way to a certification will thrash about. Hard. Most common is they will not know where to even start, deer-in-the-headlights. Even telling the candidate straight out where to start, by using your 20/20 hindsight and artificially picking a starting point 1-2 steps away from the root cause still does not unlock them; jumping to a known good starting point is useful for the experienced candidates who freeze under interview pressure, as they usually unfreeze when given such a big head start from your IRL issue that you ran into for the first time. You can often get a good gauge of how much and how deeply the candidate worked with the stack they claim competency in by how they navigate around, ask questions, probe for error messages, etc.