I wouldn’t do this with stock trading because I’m risking everything. But I wouldn’t call the people adjusting search engine parameters completely untrained eit

I wouldn’t do this with stock trading because I’m risking everything. But I wouldn’t call the people adjusting search engine parameters completely untrained either, simply not using a methodology that tests their changes against every query. I’ve found that for libraries, at least, the search engines folks are used to are simply SO BAD, so unoptimized, that a little hand tweaking and prioritizing of exact title matches will go a long way. And you’re confusing manual testing with no testing—they would very carefully watch for counter examples with a list of known good titles to search for and get back an expected set of results and were known to rollback changes when they had unexpected consequences. Effectively they have the risk appetite to test in production because the cost to end users is minimal, and the assumption when search doesn’t work is, “oh, they must not have that book” or “oh, they need to fix this particular search” and not “oh, they broke search completely and must be fired” (no one says that last one)