7 ms·

Bleve: full-text search and indexing for Go

141 points by porjo 9y ago

billconan 9y ago
is there a c++ library like this?
- latenightcoding 9y ago
  There is a C library: https://lucy.apache.org/ https://lucy.apache.org/
- hroman 9y ago
  Xapian https://xapian.org/ https://xapian.org/
- ddorian43 9y ago
  modern C++ https://github.com/phaistos-networks/Trinity https://github.com/phaistos-networks/Trinity
- leetcrew 9y ago
  if all you care about are relatively simple queries in english, clucene or lucy could probably work for you. AFAIK, there is no native C/C++ library that comes anywhere near modern lucene. for me the killer feature is that you get analyzers for many different languages out of the box with current lucene/elasticsearch.
lima 9y ago
I'm new to Go - but it looks like the example code completely ignores the error codes. Is that a common thing?
- yalue 9y ago
  It isn't common, and the code example wouldn't even compile because "err" is assigned but never used. I imagine that they included the "err" variables to show that they're available, but didn't want to clutter the example with error checks. You can ignore error codes in go by assigning them to the special variable "_", but, outside of very short toy examples, it is a huge warning sign for terrible code. It's certainly not common to ignore errors in Go.
- mschoch 9y ago
  You are correct. I think when I originally designed the homepage I wanted to show how you index and search in just a few lines. I thought omitting the error handling boilerplate was acceptable, but properly handling these errors is also key for a good initial experience. So I'm persuaded to at least revisit the decision. https://github.com/blevesearch/blevesearch.github.io-hugo/issues/12 https://github.com/blevesearch/blevesearch.github.io-hugo/is...
- roblaszczak 9y ago
  for examples like this it is much better to explicit ignore error: _ = index.Index(identifier, your_data) it will also pass https://github.com/kisielk/errcheck https://github.com/kisielk/errcheck check eventually we can just panic error if err := index.Index(identifier, your_data); err != nil { panic(err) }
shabbyrobe 9y ago
I ran into some significant performance issues with Bleve during a weekend hackathon a few months ago. I was trying to index the stackoverflow data dump for fun and I couldn't get it to successfully complete. I'm guessing i was running into some boltdb related limitations but I didn't have the time to dig deeper before the party ended and I had to get back to the day job. SQLite's FTS5 allowed me to load the entire data set without breaking a sweat, but query performance was unexpectedly poor for some combinations of terms with no apparent consistency, and it became unusable due to a temp table I couldn't work out how to avoid when attempting to search in descending primary key order. It was many months ago now and I have forgotten some of the particulars but thought it might make an interesting jumping-off point for discussion about any ideas anyone might have for indexing 60gb of data into a flat file setup like Bleve or SQLite uses. Would the new scorch experiment perform better with an index of that size than the boltdb backend?
- mschoch 9y ago
  Yeah, the old index format had many factors contributing to it taking up considerable space. As a single data point, we have a beer-search sample app, this includes a data directory with 29MB of JSON files. In the old index format, with mapping that did a realistic configuration of indexing many fields, and storing some of them, the bleve index size was over 200MB. With scorch the index size is 22MB. Query performance is comparable (and we haven't even gotten to really tuning this yet).
  - shabbyrobe 9y ago
    Nice one! I'll keep my eyes peeled for a release and give it another crack. I think all those creaky old scripts still work.
hauscraft 9y ago
Get cash immediately after issuing an invoice https://www.cityfinances.lv/faktorings/ https://www.cityfinances.lv/faktorings/
st3fan 9y ago
What kind of index size does Bleve work with well? Megabytes? Gigabytes? I’d love to understand at what scale people are using this engine.
- mschoch 9y ago
  With the current index scheme we would regularly do 10s of gigabytes (which is embarrassingly small in our opinion). We haven't done large scale testing with scorch yet, but in some smaller configurations the new index is 1/10th the size of the previous one, so we hope to have moved the bar considerably on data sizes that work.
mschoch 9y ago
Well since someone submitted us here with no apparent reason or context, allow me to provide something of interest. (primary contributor of bleve here) Just recently we merged support a new experimental index scheme called 'scorch'. This new index scheme is designed from the ground up to reduce index size and improve performance. It features: - a segment based approach, much like Lucene - vellum FTS for the term dictionary - https://github.com/couchbase/vellum https://github.com/couchbase/vellum - roaring bitmaps for the postings lists - https://github.com/RoaringBitmap/roaring https://github.com/RoaringBitmap/roaring - and compressed chunked integer storage for all the posting details It's still experimental at this point, but shows considerable indexing speedup, index size reduction, and similar query performance to the old index format used today. The code for this new index scheme can be found here: https://github.com/blevesearch/bleve/tree/master/index/scorch https://github.com/blevesearch/bleve/tree/master/index/scorc...
- oelmekki 9y ago
  Hi, There's a question I don't find an answer for by looking at homepage and documentation: does it handle concurrent queries? (I wonder, since I see a store is a single file) That is, is it something akin sqlite, meant to be used as an embedded engine for standalone applications, or is it fit to be used by a centralized api?
  - mschoch 9y ago
    Concurrent queries are supported (not sure what you mean by store being a single file, it is a directory of many files). Concurrent indexing is also possible, so long as you can arrange to not put duplicate document ids into batches executing concurrently. As for usage, it is just a library, so it is typically embedded in a single process (though this can serve multiple clients concurrently). Distributing the index across multiple nodes is done at the application level. At Couchbase we do this with bleve in a separate project called 'cbft'. https://github.com/couchbase/cbft https://github.com/couchbase/cbft
    - oelmekki 9y ago
est 9y ago
Some related work: https://github.com/go-ego/riot https://github.com/go-ego/riot https://github.com/huichen/wukong https://github.com/huichen/wukong
wejick 9y ago
I am using elasticsearch on production with golang as the both indexer and search service. Last holiday I played with blevesearch to make it works like Elasticsearch search, but the work is far far far from complete. https://github.com/wejick/balasticsearch https://github.com/wejick/balasticsearch
- bogomipz 9y ago
  >"I am using elasticsearch on production with golang as the both indexer and search service." Can you describe how this golang plus elasticsarch hybrid works? Is this is distinct from the link you provided?
  - wejick 9y ago
    Basically my applications are consuming elasticsearch REST API. One acts as indexer, the other one is using search API. I'm using this amazing package https://github.com/olivere/elastic https://github.com/olivere/elastic
  - wejick 9y ago
    Yes it is different from balasticsearch

141 points by porjo 9y ago

billconan 9y ago

is there a c++ library like this?

latenightcoding 9y ago
There is a C library: https://lucy.apache.org/ https://lucy.apache.org/
hroman 9y ago
Xapian https://xapian.org/ https://xapian.org/
ddorian43 9y ago
modern C++ https://github.com/phaistos-networks/Trinity https://github.com/phaistos-networks/Trinity
leetcrew 9y ago
if all you care about are relatively simple queries in english, clucene or lucy could probably work for you. AFAIK, there is no native C/C++ library that comes anywhere near modern lucene. for me the killer feature is that you get analyzers for many different languages out of the box with current lucene/elasticsearch.

lima 9y ago

I'm new to Go - but it looks like the example code completely ignores the error codes. Is that a common thing?

yalue 9y ago
It isn't common, and the code example wouldn't even compile because "err" is assigned but never used. I imagine that they included the "err" variables to show that they're available, but didn't want to clutter the example with error checks. You can ignore error codes in go by assigning them to the special variable "_", but, outside of very short toy examples, it is a huge warning sign for terrible code. It's certainly not common to ignore errors in Go.
mschoch 9y ago
You are correct. I think when I originally designed the homepage I wanted to show how you index and search in just a few lines. I thought omitting the error handling boilerplate was acceptable, but properly handling these errors is also key for a good initial experience. So I'm persuaded to at least revisit the decision. https://github.com/blevesearch/blevesearch.github.io-hugo/issues/12 https://github.com/blevesearch/blevesearch.github.io-hugo/is...
roblaszczak 9y ago
for examples like this it is much better to explicit ignore error: _ = index.Index(identifier, your_data) it will also pass https://github.com/kisielk/errcheck https://github.com/kisielk/errcheck check eventually we can just panic error if err := index.Index(identifier, your_data); err != nil { panic(err) }

shabbyrobe 9y ago

I ran into some significant performance issues with Bleve during a weekend hackathon a few months ago. I was trying to index the stackoverflow data dump for fun and I couldn't get it to successfully complete. I'm guessing i was running into some boltdb related limitations but I didn't have the time to dig deeper before the party ended and I had to get back to the day job. SQLite's FTS5 allowed me to load the entire data set without breaking a sweat, but query performance was unexpectedly poor for some combinations of terms with no apparent consistency, and it became unusable due to a temp table I couldn't work out how to avoid when attempting to search in descending primary key order. It was many months ago now and I have forgotten some of the particulars but thought it might make an interesting jumping-off point for discussion about any ideas anyone might have for indexing 60gb of data into a flat file setup like Bleve or SQLite uses. Would the new scorch experiment perform better with an index of that size than the boltdb backend?

mschoch 9y ago
Yeah, the old index format had many factors contributing to it taking up considerable space. As a single data point, we have a beer-search sample app, this includes a data directory with 29MB of JSON files. In the old index format, with mapping that did a realistic configuration of indexing many fields, and storing some of them, the bleve index size was over 200MB. With scorch the index size is 22MB. Query performance is comparable (and we haven't even gotten to really tuning this yet).
- shabbyrobe 9y ago
  Nice one! I'll keep my eyes peeled for a release and give it another crack. I think all those creaky old scripts still work.

hauscraft 9y ago

Get cash immediately after issuing an invoice https://www.cityfinances.lv/faktorings/ https://www.cityfinances.lv/faktorings/

st3fan 9y ago

What kind of index size does Bleve work with well? Megabytes? Gigabytes? I’d love to understand at what scale people are using this engine.

mschoch 9y ago
With the current index scheme we would regularly do 10s of gigabytes (which is embarrassingly small in our opinion). We haven't done large scale testing with scorch yet, but in some smaller configurations the new index is 1/10th the size of the previous one, so we hope to have moved the bar considerably on data sizes that work.

mschoch 9y ago

Well since someone submitted us here with no apparent reason or context, allow me to provide something of interest. (primary contributor of bleve here) Just recently we merged support a new experimental index scheme called 'scorch'. This new index scheme is designed from the ground up to reduce index size and improve performance. It features: - a segment based approach, much like Lucene - vellum FTS for the term dictionary - https://github.com/couchbase/vellum https://github.com/couchbase/vellum - roaring bitmaps for the postings lists - https://github.com/RoaringBitmap/roaring https://github.com/RoaringBitmap/roaring - and compressed chunked integer storage for all the posting details It's still experimental at this point, but shows considerable indexing speedup, index size reduction, and similar query performance to the old index format used today. The code for this new index scheme can be found here: https://github.com/blevesearch/bleve/tree/master/index/scorch https://github.com/blevesearch/bleve/tree/master/index/scorc...

oelmekki 9y ago
Hi, There's a question I don't find an answer for by looking at homepage and documentation: does it handle concurrent queries? (I wonder, since I see a store is a single file) That is, is it something akin sqlite, meant to be used as an embedded engine for standalone applications, or is it fit to be used by a centralized api?
- mschoch 9y ago
  Concurrent queries are supported (not sure what you mean by store being a single file, it is a directory of many files). Concurrent indexing is also possible, so long as you can arrange to not put duplicate document ids into batches executing concurrently. As for usage, it is just a library, so it is typically embedded in a single process (though this can serve multiple clients concurrently). Distributing the index across multiple nodes is done at the application level. At Couchbase we do this with bleve in a separate project called 'cbft'. https://github.com/couchbase/cbft https://github.com/couchbase/cbft
  - oelmekki 9y ago

est 9y ago

Some related work: https://github.com/go-ego/riot https://github.com/go-ego/riot https://github.com/huichen/wukong https://github.com/huichen/wukong

wejick 9y ago

I am using elasticsearch on production with golang as the both indexer and search service. Last holiday I played with blevesearch to make it works like Elasticsearch search, but the work is far far far from complete. https://github.com/wejick/balasticsearch https://github.com/wejick/balasticsearch

bogomipz 9y ago
>"I am using elasticsearch on production with golang as the both indexer and search service." Can you describe how this golang plus elasticsarch hybrid works? Is this is distinct from the link you provided?
- wejick 9y ago
  Basically my applications are consuming elasticsearch REST API. One acts as indexer, the other one is using search API. I'm using this amazing package https://github.com/olivere/elastic https://github.com/olivere/elastic
- wejick 9y ago
  Yes it is different from balasticsearch