9 ms·

Bleve – A modern text indexing library for Go

107 points by tuneladora 12y ago

kapilvt 12y ago
Looks Great! Looks like a loosely equivalent structure to lucene, and is a feature i've wanted in the golang ecosystem for a while.
SaberTail 12y ago
Is the name meant to bring to mind a large explosion? http://en.wikipedia.org/wiki/Boiling_liquid_expanding_vapor_explosion http://en.wikipedia.org/wiki/Boiling_liquid_expanding_vapor_...
- mschoch 12y ago
  Author here, will try to answer questions as best I can from my phone. Yes, I think I was watching one of those amazing explosions shows on Netflix. I liked the way it sounded, and it suggested some simple logo ideas.
  - drsintoma 12y ago
    Have you run any benchmarks? Do you have any rough estimations on how does it compare to Lucene in terms of speed and memory usage?
    - mschoch 12y ago
      We haven't yet. Our philosophy has been to get the features and API right first. Though there is some low hanging fruit I'd like to tackle this week. First there will be some bleve only indexing and querying benchmarks to validate those improvements. Then we should be in better shape to do comparisons against lucene.
      - arafalov 12y ago
        Is the goal to compete with Lucene or with Solr/ElasticSearch layer as well? It was a little hard to tell from the samples if - for example - configuration is coded in (like in Lucene) or has externalized XML/REST interface (like in Solr/ES). Same, with whether you are planning to scale beyond single embedded instance and need to worry about scaling/cloud/etc.
        mschoch 12y ago
        Thanks for taking an interest. I would say that we're most comparable to Lucene, given that we're a library not a server. But, we're also very much not a port of Lucene to go. That already exists, and someone has posted that link elsewhere in this thread. I've worked a fair amount with Elasticsearch, so a lot of the higher level API is inspired by it. And thats one of the biggest differences between Bleve and Lucene today. Lucene is pretty low-level. The bleve top-level API works a lot more like Elasticsearch (specifically there is a mapping, its serialized and stored in the index). There is a lower level that looks more like Lucene (everything is just documents and fields), and the top-level API builds on top of that. That is I would say the primary goal of Bleve, to be a great text indexing and search library for Go. Next you asked about a REST interface. Bleve ships with HTTP handlers in a sub-package. These are completely optional, you can use Bleve without them, or you could write your own if you don't like these. There is a not-quite-done-yet example app that uses these to make it look a lot like a single-node Elasticsearch. But, its just an example app, I have no immediate plans to try to turn this into a production app. Regarding scaling beyond a single embedded instance... Obviously there is a lot of interest in this area. The way I'm looking at this right now is that this is simply a layer above Bleve, much as Elasticsearch/Solr provide this as a layer above Lucene. There are a few enhancements we could add to streamline this, like allowing the Searches to operate across multiple indexes. My hope is that lots of apps spring up around Bleve. Maybe someone wires up the HTTP handlers and Raft and makes an Elasticsearch clone. Maybe people will do something I don't expect. Will we go there? Not right now, we've got our hands full. But if we do, it will be in a separate project, built on top of Bleve.
kolev 12y ago
Main website: http://www.blevesearch.com/ http://www.blevesearch.com/
alrs 12y ago
Wishful thinking: I hope this prefaces an Elasticsearch rewrite in Go. I'm happy to be rid of the JVM wherever I find it. No hard feelings, but we just spent 20 years crowbaring Microsoft into irrelevance. It's time to get started on Larry.
- pjmlp 12y ago
  If only Go offered the same abstraction capabilities of any JVM language.
- justinsb 12y ago
  Presumably you just want to replace the _Oracle_ JVM? Does OpenJDK meet your needs? How about Dalvik / ART or RoboVM? I'd love to hear what you're trying to achieve here ... I have often bemoaned the JVM, but have also found that it's often better than many alternatives.
  - threeseed 12y ago
    There is a whole range of JVMs: http://en.wikipedia.org/wiki/List_of_Java_virtual_machines http://en.wikipedia.org/wiki/List_of_Java_virtual_machines
  - alrs 12y ago
    Whenever you get in trouble with Elasticsearch, the first question a consultant will ask is, "why aren't you running the Oracle JVM?" Technically it supports OpenJDK. In practice, circumstance usually drags you to Oracle.
- swah 12y ago
  Aren't we going to fight Google as well?
- deleted 12y ago
  [deleted]
- threeseed 12y ago
  Why on earth would you rewrite ElasticSearch in Go ? You would have woeful monitoring support compared to the huge array of tooling for JVM, an average GC implementation and almost zero integration with most enterprise applications e.g. Hadoop. All so what, you can go from one multinational (Oracle) to another (Google).
kawera 12y ago
Looks very interesting. I've been playing with golucene[1], another alternative for those looking for a jvm-free indexing/search engine. [1] https://github.com/balzaczyy/golucene https://github.com/balzaczyy/golucene
sciurus 12y ago
Any plans to support the lucene query parser syntax?
- mschoch 12y ago
  We have support for limited version of it right now. The syntax we support is documented here: https://github.com/blevesearch/bleve/wiki/Query-String-Query https://github.com/blevesearch/bleve/wiki/Query-String-Query You can see an example query using it executed here: http://wikisearch.blevesearch.com/search/?q=name:query%20%2Bbody:%22query%20string%22%20human%5E2&p=1 http://wikisearch.blevesearch.com/search/?q=name:query%20%2B... A few things that are missing from the syntax. Numeric ranges only support simple one-sided syntax (age:>25), not a two-sided range (age:[25 to 30]). Date ranges aren't supported yet. Complex boolean expressions using parens, AND and OR. Those are things that we support in structured queries, just haven't wired up the grammar for it yet. There are handful of others we don't support at all yet, like wildcard, and proximity searches. The grammar is also still pretty simple, with an emphasis on making simple things work, not on things like escaping special characters within expressions.

107 points by tuneladora 12y ago

Looks Great! Looks like a loosely equivalent structure to lucene, and is a feature i've wanted in the golang ecosystem for a while.

SaberTail 12y ago

Is the name meant to bring to mind a large explosion? http://en.wikipedia.org/wiki/Boiling_liquid_expanding_vapor_explosion http://en.wikipedia.org/wiki/Boiling_liquid_expanding_vapor_...

mschoch 12y ago
Author here, will try to answer questions as best I can from my phone. Yes, I think I was watching one of those amazing explosions shows on Netflix. I liked the way it sounded, and it suggested some simple logo ideas.
- drsintoma 12y ago
  Have you run any benchmarks? Do you have any rough estimations on how does it compare to Lucene in terms of speed and memory usage?
  - mschoch 12y ago
    We haven't yet. Our philosophy has been to get the features and API right first. Though there is some low hanging fruit I'd like to tackle this week. First there will be some bleve only indexing and querying benchmarks to validate those improvements. Then we should be in better shape to do comparisons against lucene.
    - arafalov 12y ago
      Is the goal to compete with Lucene or with Solr/ElasticSearch layer as well? It was a little hard to tell from the samples if - for example - configuration is coded in (like in Lucene) or has externalized XML/REST interface (like in Solr/ES). Same, with whether you are planning to scale beyond single embedded instance and need to worry about scaling/cloud/etc.
      - mschoch 12y ago
        Thanks for taking an interest. I would say that we're most comparable to Lucene, given that we're a library not a server. But, we're also very much not a port of Lucene to go. That already exists, and someone has posted that link elsewhere in this thread. I've worked a fair amount with Elasticsearch, so a lot of the higher level API is inspired by it. And thats one of the biggest differences between Bleve and Lucene today. Lucene is pretty low-level. The bleve top-level API works a lot more like Elasticsearch (specifically there is a mapping, its serialized and stored in the index). There is a lower level that looks more like Lucene (everything is just documents and fields), and the top-level API builds on top of that. That is I would say the primary goal of Bleve, to be a great text indexing and search library for Go. Next you asked about a REST interface. Bleve ships with HTTP handlers in a sub-package. These are completely optional, you can use Bleve without them, or you could write your own if you don't like these. There is a not-quite-done-yet example app that uses these to make it look a lot like a single-node Elasticsearch. But, its just an example app, I have no immediate plans to try to turn this into a production app. Regarding scaling beyond a single embedded instance... Obviously there is a lot of interest in this area. The way I'm looking at this right now is that this is simply a layer above Bleve, much as Elasticsearch/Solr provide this as a layer above Lucene. There are a few enhancements we could add to streamline this, like allowing the Searches to operate across multiple indexes. My hope is that lots of apps spring up around Bleve. Maybe someone wires up the HTTP handlers and Raft and makes an Elasticsearch clone. Maybe people will do something I don't expect. Will we go there? Not right now, we've got our hands full. But if we do, it will be in a separate project, built on top of Bleve.

kolev 12y ago

Main website: http://www.blevesearch.com/ http://www.blevesearch.com/

alrs 12y ago

Wishful thinking: I hope this prefaces an Elasticsearch rewrite in Go. I'm happy to be rid of the JVM wherever I find it. No hard feelings, but we just spent 20 years crowbaring Microsoft into irrelevance. It's time to get started on Larry.

pjmlp 12y ago
If only Go offered the same abstraction capabilities of any JVM language.
justinsb 12y ago
Presumably you just want to replace the _Oracle_ JVM? Does OpenJDK meet your needs? How about Dalvik / ART or RoboVM? I'd love to hear what you're trying to achieve here ... I have often bemoaned the JVM, but have also found that it's often better than many alternatives.
- threeseed 12y ago
  There is a whole range of JVMs: http://en.wikipedia.org/wiki/List_of_Java_virtual_machines http://en.wikipedia.org/wiki/List_of_Java_virtual_machines
- alrs 12y ago
  Whenever you get in trouble with Elasticsearch, the first question a consultant will ask is, "why aren't you running the Oracle JVM?" Technically it supports OpenJDK. In practice, circumstance usually drags you to Oracle.
swah 12y ago
Aren't we going to fight Google as well?
deleted 12y ago
[deleted]
threeseed 12y ago
Why on earth would you rewrite ElasticSearch in Go ? You would have woeful monitoring support compared to the huge array of tooling for JVM, an average GC implementation and almost zero integration with most enterprise applications e.g. Hadoop. All so what, you can go from one multinational (Oracle) to another (Google).

kawera 12y ago

Looks very interesting. I've been playing with golucene[1], another alternative for those looking for a jvm-free indexing/search engine. [1] https://github.com/balzaczyy/golucene https://github.com/balzaczyy/golucene

sciurus 12y ago

Any plans to support the lucene query parser syntax?

mschoch 12y ago
We have support for limited version of it right now. The syntax we support is documented here: https://github.com/blevesearch/bleve/wiki/Query-String-Query https://github.com/blevesearch/bleve/wiki/Query-String-Query You can see an example query using it executed here: http://wikisearch.blevesearch.com/search/?q=name:query%20%2Bbody:%22query%20string%22%20human%5E2&p=1 http://wikisearch.blevesearch.com/search/?q=name:query%20%2B... A few things that are missing from the syntax. Numeric ranges only support simple one-sided syntax (age:>25), not a two-sided range (age:[25 to 30]). Date ranges aren't supported yet. Complex boolean expressions using parens, AND and OR. Those are things that we support in structured queries, just haven't wired up the grammar for it yet. There are handful of others we don't support at all yet, like wildcard, and proximity searches. The grammar is also still pretty simple, with an emphasis on making simple things work, not on things like escaping special characters within expressions.