5 ms·
* We imported ~1,000,000 messages of dynamic length (some very long, eg. emails); * Once imported, the search index weights 20MB (KV) + 1.4MB (FST) on disk; T
by marginalia_nu 4y ago
* We imported ~1,000,000 messages of dynamic length (some very long, eg. emails);
* Once imported, the search index weights 20MB (KV) + 1.4MB (FST) on disk;
This is almost unbelievably succinct! If you encode the document features into 8 bits per document, and thus completely forego the need to store the document ID by indexing them implicitly, that alone is 1 MB.
Getting meaningful search out of on average 21 bytes per document seriously impressive.
[For reference, this sentence is 42 bytes.]
- mattb314 4y agoWonder if this has anything to do with the sliding window: > Sonic only keeps the N most recently pushed results for a given word, in a sliding window way (the sliding window width can be configured) Default window looks like 1k documents. I read this as saying that super common words are basically dropped from the index (only 1k out of many thousands of docs retained), but I don’t know enough about the internals to be sure. Not sure if this actually hurts search results in practice, seems like an ok trade off for help docs at least.
- 411111111111111 4y agoIt's definitely a great trade-off to make for efficiently, but makes it inherently unusable for most of elastic searchs usecases. Looking at it from a practical example such as log search (almost everyone I know has used kibana/logstash/elasticsearch at some point): you'd be able to search for things like tracingId/requestId but adding more filters such as logLevel, requestType or serviceName would be impossible It has it's niche, but calling it an elasticsearch alternative really is a stretch
- rabuse 4y agoAlso the ability to weight fields when fetching results to boost relevancy, which is needed for a lot of my use cases.
- nightpool 4y agoI wonder how easy it would be to change "most recently pushed" to something like a redis sorted set where each document has a score and only the top N results are retained when sorted by their separate score value? That would allow you to sort by pageviews / popularity in a more useful way. But it fails entirely when looking for uncommon intersections of common words, which feels like it makes it useless for most actual full-text search use-cases :(
- deleted 4y ago[deleted]