4 ms·

But it works? If you measure your tuning based on which rank an item is when a user clicks you might get feedback that would obviously help find more examples o

But it works? If you measure your tuning based on which rank an item is when a user clicks you might get feedback that would obviously help find more examples or create some kind of self-learning system, but if you only care about say, boosting common best sellers periodically, well, isn’t that good for (almost) everyone? ;-) My suggestion for GoodReads would be to rank exact title matches higher than partial or reordered titles, but that would just be a start. Of course, evaluate what’s working and what isn’t, and maybe do test cases where you see if lower rankings have improved. But any attempt at a fix is likely better than doing nothing, as you’re likely to spend energy fixing popular books folks want to find...

mlthoughts2018 7y ago
> “But it works?” This is the big red flag, when non-specialists hacking on Solr boosts are claiming something works because of a few qualitative test cases. “It works” is a statement that only applies after you’ve done qualitative and quantitative goodness of fit testing. You wouldn’t have a random IT employee make a stock-trading algorithm and then test it on a month of data and call it a success. For a search solution to “work,” it needs to pass quantitative and qualitative checking, and be explainable to stakeholders and be reproducible / incrementally updateable. The training and arrival at hyperparameters all need to be reproducible and based on the outcome criteria they are meant to solve. Making some hacks into a demo that superficially looks good is not at all the same as “it works.”
- lstamour 7y ago
  I wouldn’t do this with stock trading because I’m risking everything. But I wouldn’t call the people adjusting search engine parameters completely untrained either, simply not using a methodology that tests their changes against every query. I’ve found that for libraries, at least, the search engines folks are used to are simply SO BAD, so unoptimized, that a little hand tweaking and prioritizing of exact title matches will go a long way. And you’re confusing manual testing with no testing—they would very carefully watch for counter examples with a list of known good titles to search for and get back an expected set of results and were known to rollback changes when they had unexpected consequences. Effectively they have the risk appetite to test in production because the cost to end users is minimal, and the assumption when search doesn’t work is, “oh, they must not have that book” or “oh, they need to fix this particular search” and not “oh, they broke search completely and must be fired” (no one says that last one)

mlthoughts2018 7y ago

> “But it works?” This is the big red flag, when non-specialists hacking on Solr boosts are claiming something works because of a few qualitative test cases. “It works” is a statement that only applies after you’ve done qualitative and quantitative goodness of fit testing. You wouldn’t have a random IT employee make a stock-trading algorithm and then test it on a month of data and call it a success. For a search solution to “work,” it needs to pass quantitative and qualitative checking, and be explainable to stakeholders and be reproducible / incrementally updateable. The training and arrival at hyperparameters all need to be reproducible and based on the outcome criteria they are meant to solve. Making some hacks into a demo that superficially looks good is not at all the same as “it works.”

lstamour 7y ago
I wouldn’t do this with stock trading because I’m risking everything. But I wouldn’t call the people adjusting search engine parameters completely untrained either, simply not using a methodology that tests their changes against every query. I’ve found that for libraries, at least, the search engines folks are used to are simply SO BAD, so unoptimized, that a little hand tweaking and prioritizing of exact title matches will go a long way. And you’re confusing manual testing with no testing—they would very carefully watch for counter examples with a list of known good titles to search for and get back an expected set of results and were known to rollback changes when they had unexpected consequences. Effectively they have the risk appetite to test in production because the cost to end users is minimal, and the assumption when search doesn’t work is, “oh, they must not have that book” or “oh, they need to fix this particular search” and not “oh, they broke search completely and must be fired” (no one says that last one)