I do something similar; building an arbitrary thing gives you a good window into how they actually approach solving a problem. One key difference - I use the s

I do something similar; building an arbitrary thing gives you a good window into how they actually approach solving a problem. One key difference - I use the same task for each interview, which while boring for me, makes it easier to transfer my score between candidates. I initially had some colleagues that I've worked with for years do the interview to calibrate what a ninja would look like on that test, which I've found very useful. I've also considered doing real-world problems, but my concern is that 1) they tend to have too much back-story and context, and 2) it would be hard to calibrate the difficulty. Perhaps that's unfounded.