3 ms·
I ran into some significant performance issues with Bleve during a weekend hackathon a few months ago. I was trying to index the stackoverflow data dump for fun
by shabbyrobe 9y ago
I ran into some significant performance issues with Bleve during a weekend hackathon a few months ago. I was trying to index the stackoverflow data dump for fun and I couldn't get it to successfully complete. I'm guessing i was running into some boltdb related limitations but I didn't have the time to dig deeper before the party ended and I had to get back to the day job.
SQLite's FTS5 allowed me to load the entire data set without breaking a sweat, but query performance was unexpectedly poor for some combinations of terms with no apparent consistency, and it became unusable due to a temp table I couldn't work out how to avoid when attempting to search in descending primary key order.
It was many months ago now and I have forgotten some of the particulars but thought it might make an interesting jumping-off point for discussion about any ideas anyone might have for indexing 60gb of data into a flat file setup like Bleve or SQLite uses.
Would the new scorch experiment perform better with an index of that size than the boltdb backend?
- mschoch 9y agoYeah, the old index format had many factors contributing to it taking up considerable space. As a single data point, we have a beer-search sample app, this includes a data directory with 29MB of JSON files. In the old index format, with mapping that did a realistic configuration of indexing many fields, and storing some of them, the bleve index size was over 200MB. With scorch the index size is 22MB. Query performance is comparable (and we haven't even gotten to really tuning this yet).
- shabbyrobe 9y agoNice one! I'll keep my eyes peeled for a release and give it another crack. I think all those creaky old scripts still work.