3 ms·

I build search engines for a living. While I appreciate the hacker spirit of your project with Solr, I also see this as a huge problem that leads to bad search

by mlthoughts2018 7y ago

I build search engines for a living. While I appreciate the hacker spirit of your project with Solr, I also see this as a huge problem that leads to bad search experiences. Tuning boosts in Solr is not even close to a reasonable way to solve problems like this. Arguably not even for an underfunded library, but certainly not for a high web traffic consumer website. For one, you need disciplined acceptance criteria in the form of both qualitative standards (things a non-technical manager can look at and say yes or no) as well as various relevancy measurements like mean reciprocal rank and normalized discounted cumulative gain (via acquiring human annotated data if needed). When people only focus on qualitative feedback on top of boosts and hacks in an off the shelf tool, they usually end up with some weird witches’ brew of bizarre boosts and time-decay weighting that is extremely fragile and can’t be robustly changed or even understood without the qualitative performance going haywire. You need disciplined study of quantitative ranking metrics to know the drivers of performance, fall off as you move down the ranking position, and to make search index updates reproducible and make incremental improvement measurable. Meanwhile if you only focus on quantitative metrics, you might miss obvious red flags. The relevance score used for NDCG might be biased some way. You might surface highly relevant results to only one context or sense of the words in a query (like only showing fruit for “apple” and never tech gadgets). You need people who make the subjective appraisal of quality for users to be looped in. Here’s the point. When this is all missing, you will lose credibility with people making the decisions. They’ll hear some engineer babble about NDCG but then say the darn thing doesn’t work in the QA testing. Or they’ll say the qualitative results look OK and get angry when weird counter-examples pop up in the second or third results pages, which might have been measured with quantitative metrics. When this happens, executives and managers just want to punt. They want the “nobody ever got fired for buying IBM” equivalent for search, and that’s how you end up with Confluence still only supporting exact title matching and having no ability for actual content relevancy search. In this sense, the little projects showing “look what us non-specialists could cook up by hacking some boosts in Solr!” do a lot of harm and should not be considered as the plucky success stories they are often painted to be.

lstamour 7y ago
But it works? If you measure your tuning based on which rank an item is when a user clicks you might get feedback that would obviously help find more examples or create some kind of self-learning system, but if you only care about say, boosting common best sellers periodically, well, isn’t that good for (almost) everyone? ;-) My suggestion for GoodReads would be to rank exact title matches higher than partial or reordered titles, but that would just be a start. Of course, evaluate what’s working and what isn’t, and maybe do test cases where you see if lower rankings have improved. But any attempt at a fix is likely better than doing nothing, as you’re likely to spend energy fixing popular books folks want to find...
- mlthoughts2018 7y ago
  > “But it works?” This is the big red flag, when non-specialists hacking on Solr boosts are claiming something works because of a few qualitative test cases. “It works” is a statement that only applies after you’ve done qualitative and quantitative goodness of fit testing. You wouldn’t have a random IT employee make a stock-trading algorithm and then test it on a month of data and call it a success. For a search solution to “work,” it needs to pass quantitative and qualitative checking, and be explainable to stakeholders and be reproducible / incrementally updateable. The training and arrival at hyperparameters all need to be reproducible and based on the outcome criteria they are meant to solve. Making some hacks into a demo that superficially looks good is not at all the same as “it works.”
  - lstamour 7y ago
    I wouldn’t do this with stock trading because I’m risking everything. But I wouldn’t call the people adjusting search engine parameters completely untrained either, simply not using a methodology that tests their changes against every query. I’ve found that for libraries, at least, the search engines folks are used to are simply SO BAD, so unoptimized, that a little hand tweaking and prioritizing of exact title matches will go a long way. And you’re confusing manual testing with no testing—they would very carefully watch for counter examples with a list of known good titles to search for and get back an expected set of results and were known to rollback changes when they had unexpected consequences. Effectively they have the risk appetite to test in production because the cost to end users is minimal, and the assumption when search doesn’t work is, “oh, they must not have that book” or “oh, they need to fix this particular search” and not “oh, they broke search completely and must be fired” (no one says that last one)

lstamour 7y ago

But it works? If you measure your tuning based on which rank an item is when a user clicks you might get feedback that would obviously help find more examples or create some kind of self-learning system, but if you only care about say, boosting common best sellers periodically, well, isn’t that good for (almost) everyone? ;-) My suggestion for GoodReads would be to rank exact title matches higher than partial or reordered titles, but that would just be a start. Of course, evaluate what’s working and what isn’t, and maybe do test cases where you see if lower rankings have improved. But any attempt at a fix is likely better than doing nothing, as you’re likely to spend energy fixing popular books folks want to find...

mlthoughts2018 7y ago
> “But it works?” This is the big red flag, when non-specialists hacking on Solr boosts are claiming something works because of a few qualitative test cases. “It works” is a statement that only applies after you’ve done qualitative and quantitative goodness of fit testing. You wouldn’t have a random IT employee make a stock-trading algorithm and then test it on a month of data and call it a success. For a search solution to “work,” it needs to pass quantitative and qualitative checking, and be explainable to stakeholders and be reproducible / incrementally updateable. The training and arrival at hyperparameters all need to be reproducible and based on the outcome criteria they are meant to solve. Making some hacks into a demo that superficially looks good is not at all the same as “it works.”
- lstamour 7y ago
  I wouldn’t do this with stock trading because I’m risking everything. But I wouldn’t call the people adjusting search engine parameters completely untrained either, simply not using a methodology that tests their changes against every query. I’ve found that for libraries, at least, the search engines folks are used to are simply SO BAD, so unoptimized, that a little hand tweaking and prioritizing of exact title matches will go a long way. And you’re confusing manual testing with no testing—they would very carefully watch for counter examples with a list of known good titles to search for and get back an expected set of results and were known to rollback changes when they had unexpected consequences. Effectively they have the risk appetite to test in production because the cost to end users is minimal, and the assumption when search doesn’t work is, “oh, they must not have that book” or “oh, they need to fix this particular search” and not “oh, they broke search completely and must be fired” (no one says that last one)