5 ms·

From an information retrieval perspective, this is embarrassing, and the benchmarks aren’t close to being apples and oranges—it’s more like apples and Mack truc

From an information retrieval perspective, this is embarrassing, and the benchmarks aren’t close to being apples and oranges—it’s more like apples and Mack trucks. The linked benchmark is for a toy data set of 5.6GB, which we all know Redis will store in RAM; yet they don’t say whether they set “index.storage.type: memory” on ElasticSearch (guessing they didn’t). At the same time, ElasticSearch/Lucene puts considerable effort into analysis at both indexing time and querying time that goes into ranking the search results. RediSearch’s ranking is “user provided”—how does that work exactly? What does that even mean? Ranking is at the heart of information retrieval—of what value is it to allow for 4x more queries to be run on a cluster when the result sets are terrible? ElasticSearch could be better at scaling up on nodes to handle more query operations on a given cluster. However, if you care about full text search, this won’t do the job.

jiofih 6y ago
It’s amazing that you reached the conclusion that it won’t do the job after reading a benchmark for 2 minutes! The benchmarks are quite realistic (5m docs, 25m products) for document search, and the settings reflect real usage. There is no “memory” storage type for ES, it has “mmapfs” which the documentation explicitly says is dangerous and might be removed. The default `hybridfs` already has the ability to use memory mapped files as an optimization. Forcing mmap would actually make it an unrealistic benchmark. > ElasticSearch could be better at scaling up on nodes to handle more query operations on a given cluster Based on? They are both running with 5 shards for the benchmark.
- joking 6y ago
  Well, you have MemoryIndex, which is different proposition (it's used for the percolate feature). I can get that redis search can be faster than Lucene based search engines, but actually running the search is the easiest part, there's a lot of thought and effort on making a decent search engine, and the article doesn't tell me if I can get facets, order by any field, make both fulltext and geospatial queries, and many more things important when you make a search engine. I rememeber long ago when SQL Server included full text, we thought that our time fiddling with lucene.net had become to an end, but when we tried it failed miserably because the full text engine and the relational one where like 2 different parts, and if you wanted to make a full text search and order by a numeric field, it would have to make a temporal table with all the results of the full text and order that. Those are the things tha lucene solves so well, that I'm reticent to think that redis search has managed to make them ok at the first try. So, not apples to oranges benchmark, but if you have already redis and the search capabilities that you want to add are fullfiled by redis search, it can be a good product.
  - kristoff_it 6y ago
    > I'm reticent to think that redis search has managed to make them ok at the first try. Your example doesn't really match the reality of Redis. Modules in Redis can bring their own data types and algorithms and don't need to resort to the same kind of hack you mentioned. The ecosystem inside Redis is designed for modularity and clear, minimalistic interfaces between components. RediSearch might not be perfect, but any problem will stem from a different set of causes, not because it tries to "emulate" a data model. The whole point of Redis is not having to emulate data types and algorithms.
    - derefr 6y ago
      > Modules in Redis can bring their own data types and algorithms and don't need to resort to the same kind of hack you mentioned. To emphasize this: One thing people might misunderstand about Redis is that Redis extension developers aren't expected to fit their data structures to the needs of a storage-engine. It's not like Cassandra, where your data structures must 'boil down' to key-value pairs; nor is it like an RDBMS, where your data structures must 'boil down' to tuple-sets; nor like a graph DBMS, where your data structures must 'boil down' to EAV triples. Redis data structures aren't "implemented in terms of" any other simpler 'canonical' data structure. Instead, when you look at something like Redis Streams or the Redis Graph module, the whole complex data structure for each stream/graph is a big opaque in-memory thing dangling off a single key. It doesn't need to be broken down into parts "legible" to Redis. It can just be what it is. "Objects" in the Redis keyspace (the things keys are holding) can't hold pointers to one-another, so the core Redis commands can blindly manipulate them (e.g. deallocate them.) For everything else, you go through the module's commands, which walk the internals of the data-structure as what it is: a plain-old in-memory C struct, defined in your module's header files. The canonical representation of a data structure in the Redis AOF (WAL log) is just the sequence of commands used to build it; not the data-structure itself. So, as a module developer, to get AOF persistence of your module's types, you don't need to do a thing, other than ensuring that your module's commands are deterministic. You do need to do a bit of work to get your module's types to serialize into Redis RDB snapshots. But it's completely up to you how to define your types' serializations. Redis just provides an API for writing and reading scalar types from the RDB file stream. How your module uses them to save/load a value of a type is up to you. (And you can just skip this if you like; RDB persistence is used far less often than AOF persistence, so support for RDB persistence it's not even a highly-demanded feature for modules. If you don't bother, then loading your module just disables RDB persistence.)
- derefr 6y ago
  I think the usual way to "level the playing field" between in-memory and disk-backed based storage engines for clearer comparison, is to back the disk-backed storage engine with a tmpfs volume. I believe that the combination of tmpfs + file access — and especially tmpfs + mmap(2) — results in a fast path that almost resembles direct memory access. (If the file or memory-region is opened read-only, the kernel can expose/share the relevant memory pages directly into the process.)
- jonstewart 6y ago
  The point is not that the benchmarks were setup incorrectly or disputing that RediSearch is faster. It’s thst it’s doing far far less because it’s not ranking, and full text search without ranking is... less than useful.
  - jiofih 6y ago
    That is just a wild assumption since results quality is not covered in the article.
dvirsky 6y ago
> RediSearch’s ranking is “user provided” That's completely untrue, it comes with a few scoring algorithms out of the box (https://oss.redislabs.com/redisearch/Scoring/ https://oss.redislabs.com/redisearch/Scoring/) and results can also be ranked by the value of arbitrary numeric or short text fields. On top of that, there is a plugin API to provide custom scorers written in C/C++. BTW It's funny though that in the end most of the use of ElasticSearch is for analyzing logs and all the IR scoring tricks are not even used.
Apofis 6y ago
Search is hard. When ElasticSearch isn't enough, I reach for Algolia. For basic search, this should be good, but ranking is very important... how's RediSearch's fuzzy search results? The second most important aspect of search.

jiofih 6y ago

It’s amazing that you reached the conclusion that it won’t do the job after reading a benchmark for 2 minutes! The benchmarks are quite realistic (5m docs, 25m products) for document search, and the settings reflect real usage. There is no “memory” storage type for ES, it has “mmapfs” which the documentation explicitly says is dangerous and might be removed. The default `hybridfs` already has the ability to use memory mapped files as an optimization. Forcing mmap would actually make it an unrealistic benchmark. > ElasticSearch could be better at scaling up on nodes to handle more query operations on a given cluster Based on? They are both running with 5 shards for the benchmark.

joking 6y ago
Well, you have MemoryIndex, which is different proposition (it's used for the percolate feature). I can get that redis search can be faster than Lucene based search engines, but actually running the search is the easiest part, there's a lot of thought and effort on making a decent search engine, and the article doesn't tell me if I can get facets, order by any field, make both fulltext and geospatial queries, and many more things important when you make a search engine. I rememeber long ago when SQL Server included full text, we thought that our time fiddling with lucene.net had become to an end, but when we tried it failed miserably because the full text engine and the relational one where like 2 different parts, and if you wanted to make a full text search and order by a numeric field, it would have to make a temporal table with all the results of the full text and order that. Those are the things tha lucene solves so well, that I'm reticent to think that redis search has managed to make them ok at the first try. So, not apples to oranges benchmark, but if you have already redis and the search capabilities that you want to add are fullfiled by redis search, it can be a good product.
- kristoff_it 6y ago
  > I'm reticent to think that redis search has managed to make them ok at the first try. Your example doesn't really match the reality of Redis. Modules in Redis can bring their own data types and algorithms and don't need to resort to the same kind of hack you mentioned. The ecosystem inside Redis is designed for modularity and clear, minimalistic interfaces between components. RediSearch might not be perfect, but any problem will stem from a different set of causes, not because it tries to "emulate" a data model. The whole point of Redis is not having to emulate data types and algorithms.
  - derefr 6y ago
    > Modules in Redis can bring their own data types and algorithms and don't need to resort to the same kind of hack you mentioned. To emphasize this: One thing people might misunderstand about Redis is that Redis extension developers aren't expected to fit their data structures to the needs of a storage-engine. It's not like Cassandra, where your data structures must 'boil down' to key-value pairs; nor is it like an RDBMS, where your data structures must 'boil down' to tuple-sets; nor like a graph DBMS, where your data structures must 'boil down' to EAV triples. Redis data structures aren't "implemented in terms of" any other simpler 'canonical' data structure. Instead, when you look at something like Redis Streams or the Redis Graph module, the whole complex data structure for each stream/graph is a big opaque in-memory thing dangling off a single key. It doesn't need to be broken down into parts "legible" to Redis. It can just be what it is. "Objects" in the Redis keyspace (the things keys are holding) can't hold pointers to one-another, so the core Redis commands can blindly manipulate them (e.g. deallocate them.) For everything else, you go through the module's commands, which walk the internals of the data-structure as what it is: a plain-old in-memory C struct, defined in your module's header files. The canonical representation of a data structure in the Redis AOF (WAL log) is just the sequence of commands used to build it; not the data-structure itself. So, as a module developer, to get AOF persistence of your module's types, you don't need to do a thing, other than ensuring that your module's commands are deterministic. You do need to do a bit of work to get your module's types to serialize into Redis RDB snapshots. But it's completely up to you how to define your types' serializations. Redis just provides an API for writing and reading scalar types from the RDB file stream. How your module uses them to save/load a value of a type is up to you. (And you can just skip this if you like; RDB persistence is used far less often than AOF persistence, so support for RDB persistence it's not even a highly-demanded feature for modules. If you don't bother, then loading your module just disables RDB persistence.)
derefr 6y ago
I think the usual way to "level the playing field" between in-memory and disk-backed based storage engines for clearer comparison, is to back the disk-backed storage engine with a tmpfs volume. I believe that the combination of tmpfs + file access — and especially tmpfs + mmap(2) — results in a fast path that almost resembles direct memory access. (If the file or memory-region is opened read-only, the kernel can expose/share the relevant memory pages directly into the process.)
jonstewart 6y ago
The point is not that the benchmarks were setup incorrectly or disputing that RediSearch is faster. It’s thst it’s doing far far less because it’s not ranking, and full text search without ranking is... less than useful.
- jiofih 6y ago
  That is just a wild assumption since results quality is not covered in the article.

dvirsky 6y ago

> RediSearch’s ranking is “user provided” That's completely untrue, it comes with a few scoring algorithms out of the box (https://oss.redislabs.com/redisearch/Scoring/ https://oss.redislabs.com/redisearch/Scoring/) and results can also be ranked by the value of arbitrary numeric or short text fields. On top of that, there is a plugin API to provide custom scorers written in C/C++. BTW It's funny though that in the end most of the use of ElasticSearch is for analyzing logs and all the IR scoring tricks are not even used.

Apofis 6y ago

Search is hard. When ElasticSearch isn't enough, I reach for Algolia. For basic search, this should be good, but ranking is very important... how's RediSearch's fuzzy search results? The second most important aspect of search.