I've experienced this pushback against the algorithm firsthand. I work in NLP, and I was asked to write a script to select data for annotation based on a number

I've experienced this pushback against the algorithm firsthand. I work in NLP, and I was asked to write a script to select data for annotation based on a number of criteria, including some automated balancing of a few factors. It turns out, this was very unsatisfying to some. It led to questions like "why is this document being selected, but not this one?" without an answer that was really satisfying. I ended up changing it to give total control to the person running the script, and removed the self-balancing feature. Since then, I pushback against any sort of black-box auto-selection sort of ideas, because I know that, while they might result in faster and ultimately equivalent results, people want control.