4 ms·
Great post. I'm working on another open source search engine on top of Redis (http://redisearch.io http://redisearch.io), mostly focused on index building and s
by dvirsky 9y ago
Great post. I'm working on another open source search engine on top of Redis (http://redisearch.io http://redisearch.io), mostly focused on index building and serving, and real-time updates of the data. The part about queries being highly varying is extremely challenging. You have to deal with simple "foo bar" queries and complex queries with intricate filtering and crazy stuff (I have a user doing an AND intersection of 17 OR unions, each of 32 terms, while checking the ordering of term offsets!). It's super fun to work on this stuff.
Another recommended book that was not mentioned in the post: Search Engines: Information Retrieval in Practice https://www.amazon.com/Search-Engines-Information-Retrieval-Practice/dp/0136072240 https://www.amazon.com/Search-Engines-Information-Retrieval-...
- sorokod 9y agoIf you allow users to execute arbitrary queries, how do you protect your server resources from being totally consumed by a small subset of your users?
- dvirsky 9y agoIn terms of redis being single threaded, we put a lot of work into making the search concurrent, see here for details: https://redislabs.com/blog/making-redis-concurrent-with-modules/ https://redislabs.com/blog/making-redis-concurrent-with-modu.... Basically we rotate between long queries and short queries, making long queries run slower but not block short ones. Right now rate limiting is left to the user and not solved at the engine level.
- sorokod 9y agoThis is about the complexity of queries (which, without knowing your system, may be unbound) - rate limiting will not help.
- dvirsky 9y agoYeah, I was also thinking about adding execution cap to a single query, that's not hard to add as there is a simple scheduler that checks their progress constantly. So just terminating a long running query is simple.