3 ms·

Are there any plan to have sharding mechanism implemented on the blevesearch? IMHO it should be on the application level, but having this on blevesearch would b

by wejick 9y ago

Are there any plan to have sharding mechanism implemented on the blevesearch? IMHO it should be on the application level, but having this on blevesearch would be good. Regarding segments based approach like lucene, means we will need to do segments merging which in my experience quite resource intensive. I don't actually know how blevesearch handles this prior segment based approach.

millisecond 9y ago
Not directly sharding but I've been working on a fork of Bleve that uses Cassandra as the backing store which allows for horizontal scaling of a single index: https://github.com/wrble/flock https://github.com/wrble/flock Still very much a work in progress but the core is functional.
- ddorian43 9y ago
  see elassandra for something similar
mschoch 9y ago
Bleve has support for querying across multiple indexes (shards) but does not prescribe any mechanism to split the data. So, it's up the application to divide the data how it sees fit, but you can use Bleve functionality to execute the same query across multiple indexes and merge the results. Merging is required and is indeed somewhat resource intensive. Bleve's current indexing approach has no segments, instead all index data is serialized into a key/value store. This approach allowed us to experiment and plug-in a variety of implementations. Unfortunately, the key/value abstraction limits the way you interact with data, so there are a number of drawbacks. One key gain we get from the segmented approach vs the key/value store approach is that we no longer need to maintain a backindex to handle updates/deletes.
- wejick 9y ago
  Yes and if I'm not mistaken that's what alias on bleve for. Not yet looking into scorch. So scorch would replace other storage engine like rocksdb and leveldb?
  - mschoch 9y ago
    Yes, when you create a new index, you can choose the index implementation. The previous one was called 'upsidedown', and this one used key/value stores for the actual storage. So, you would also specify BoltDB/RocksDB/LevelDB/Moss/etc. The new index implementation is called 'scorch' and it writes directly to disk instead of going through a key/value store.

by wejick 9y ago

millisecond 9y ago

Not directly sharding but I've been working on a fork of Bleve that uses Cassandra as the backing store which allows for horizontal scaling of a single index: https://github.com/wrble/flock https://github.com/wrble/flock Still very much a work in progress but the core is functional.

ddorian43 9y ago
see elassandra for something similar

mschoch 9y ago

Bleve has support for querying across multiple indexes (shards) but does not prescribe any mechanism to split the data. So, it's up the application to divide the data how it sees fit, but you can use Bleve functionality to execute the same query across multiple indexes and merge the results. Merging is required and is indeed somewhat resource intensive. Bleve's current indexing approach has no segments, instead all index data is serialized into a key/value store. This approach allowed us to experiment and plug-in a variety of implementations. Unfortunately, the key/value abstraction limits the way you interact with data, so there are a number of drawbacks. One key gain we get from the segmented approach vs the key/value store approach is that we no longer need to maintain a backindex to handle updates/deletes.

wejick 9y ago
Yes and if I'm not mistaken that's what alias on bleve for. Not yet looking into scorch. So scorch would replace other storage engine like rocksdb and leveldb?
- mschoch 9y ago
  Yes, when you create a new index, you can choose the index implementation. The previous one was called 'upsidedown', and this one used key/value stores for the actual storage. So, you would also specify BoltDB/RocksDB/LevelDB/Moss/etc. The new index implementation is called 'scorch' and it writes directly to disk instead of going through a key/value store.