3 ms·

I work professionally in this space and I can say over the past ~10 years, my full time job at 4 companies has been almost entirely to migrate away from SOLR /

by mlthoughts2018 7y ago

I work professionally in this space and I can say over the past ~10 years, my full time job at 4 companies has been almost entirely to migrate away from SOLR / Lucene solutions and implement custom in-house search indexes. Most of the time it has been for performance reasons. SOLR / Lucene have very poor performance characteristics, especially when needing to support custom sort ordering and heavy use of filters. On other occasions it has been because you can’t easily extend search indexes to more advanced use cases, like similarity-based reverse item search, collaborative filtering, more advanced treatment of cold start issues / bias towards existing popular content / trending search. A lot of small or medium sized companies naively figure they’ll just use SOLR etc to get something out of the box, or for side channel problems that are smaller scale. But you come to regret it pretty fast because you end up needing one standardized way to build and deploy search indices and it has to support all the bells and whistles that SOLR can’t _and_ be faster than SOLR, for the big product use cases. A lot of product companies now are beginning to use word-vector approaches with nearest neighbor libraries like ANNOY, as the first solution instead of the solution you eventually have to migrate to when you realize SOLR does not actually support your use case, not even as a means to get it up & running quickly.

peterdm 7y ago
Are you seeing a blueprint start to emerge for a standardized way to build and deploy search indices in the context of applications that need vector-space features? (E.g. if you start with ANNOY, you get kNN but then how do you add in the ability to refine, filter, rescore, sort with text, etc?)
- mlthoughts2018 7y ago
  I actually have seen the opposite trend. You don’t want to standardize the search engine and that has been a major problem with SOLR. Instead, you want to custom build the data retrieval system so it’s tailored to your use case. One example from experience was needing to add hard filterable metadata to an in-house search index. We solved this by actually calculating bit masks that represented all the filtering criteria and having a frontend preprocessor that would first restrict to the filtered subset and then do TFIDF-based relevance sorting. Creating the bit mask tooling ourselves (instead of relying on whatever baked-in method of scanning items and filtering that comes with out of the box search engine tools) allowed us complete control over the trade-offs, particularly managing document deletions and optimizing run time performance in certain ways that just weren’t available in out of the box tools, as well as being able to integrate any in-house code into the search engine as needed (since the whole system was in-house code). You want to create data models that are highly application specific, and then route data into them. The mistaken approach of one-size-fits-all tools, especially in information retrieval, is to pre-define the supported behavior of the application, like a web service wrapping a search index, with baked-in assumptions about the trade-offs and only limited support to modify or configure the trade-offs under the hood. The gravest mistake is thinking just because your use case seems to function OK with those assumptions now, that you can marry yourself to the underlying data model. Then in the future you’ll hit the point where you have to throw it away and create something custom, but it will be far more costly to do so and extremely hard to migrate gracefully and ensure integrations are working.

peterdm 7y ago

Are you seeing a blueprint start to emerge for a standardized way to build and deploy search indices in the context of applications that need vector-space features? (E.g. if you start with ANNOY, you get kNN but then how do you add in the ability to refine, filter, rescore, sort with text, etc?)

mlthoughts2018 7y ago
I actually have seen the opposite trend. You don’t want to standardize the search engine and that has been a major problem with SOLR. Instead, you want to custom build the data retrieval system so it’s tailored to your use case. One example from experience was needing to add hard filterable metadata to an in-house search index. We solved this by actually calculating bit masks that represented all the filtering criteria and having a frontend preprocessor that would first restrict to the filtered subset and then do TFIDF-based relevance sorting. Creating the bit mask tooling ourselves (instead of relying on whatever baked-in method of scanning items and filtering that comes with out of the box search engine tools) allowed us complete control over the trade-offs, particularly managing document deletions and optimizing run time performance in certain ways that just weren’t available in out of the box tools, as well as being able to integrate any in-house code into the search engine as needed (since the whole system was in-house code). You want to create data models that are highly application specific, and then route data into them. The mistaken approach of one-size-fits-all tools, especially in information retrieval, is to pre-define the supported behavior of the application, like a web service wrapping a search index, with baked-in assumptions about the trade-offs and only limited support to modify or configure the trade-offs under the hood. The gravest mistake is thinking just because your use case seems to function OK with those assumptions now, that you can marry yourself to the underlying data model. Then in the future you’ll hit the point where you have to throw it away and create something custom, but it will be far more costly to do so and extremely hard to migrate gracefully and ensure integrations are working.