I would question what signaling effects you think you're actually measuring when you're presenting contrived scenarios for which you already know the answers.

I would question what signaling effects you think you're actually measuring when you're presenting contrived scenarios for which you already know the answers. There's a tremendous amount of variability in whether or not the artificial test that's been created actually maps appropriately to what you're trying to assess. It's extremely difficult to do well. There's further variability in the proctoring and posturing in how the assessment is performed, not just the assessment itself. Almost certainly you've added layers of obfuscation to the thing you actually want to know, unless what you want to know is how good the person is at taking your specific test, which may or may not usefully test anything, and which may or may not be administered in a way that could get useful results even if the test itself is meticulously well-crafted. If you're intent on, or can't avoid, operating a more traditional looking hiring process, then I'd suggest at least trying this. Even if the assessment is contrived, then use questions that nobody in the room knows the answers to, including yourself. It doesn't relate to the work directly necessarily, but it can at least help mitigate the effects of survivorship bias.