This is fun. I managed a data entry project at a previous employer that was kind of like this. We had non technical subject matter experts that I sneakily got

This is fun. I managed a data entry project at a previous employer that was kind of like this. We had non technical subject matter experts that I sneakily got to write several thousand unit tests. We had some quantitative and qualitative data we wanted annotated on a substantial dataset of scanned documents. I built a tool that let them type natural language descriptors for everything. I then converted that to javascript unit tests by way of some regex and coffeescript. It was possible to bootstrap classifiers that way! Seems like kind of old tech in an era of unsupervised megaML, but I've not been using that part of my resume for a year or so.