24 ms·

Bayard: a full-text search and indexing server written in Rust

283 points by jinqueeny 7y ago

jinqueeny 7y ago
It's built on top of Tantivy (https://github.com/tantivy-search/tantivy https://github.com/tantivy-search/tantivy) that implements Tha Raft Consensus Algorithm (https://raft.github.io/ https://raft.github.io/) by raft-rs (https://github.com/tikv/grpc-rs https://github.com/tikv/grpc-rs) and The gRPC (HTTP/2 + Protocol Buffers) by grpc-rs (https://github.com/tikv/grpc-rs https://github.com/tikv/grpc-rs) and rust-protobuf (https://github.com/stepancheg/rust-protobuf https://github.com/stepancheg/rust-protobuf).
- clemParis 7y ago
  typo: raft-rs is at https://github.com/tikv/raft-rs https://github.com/tikv/raft-rs
  - jinqueeny 7y ago
    thanks
- mrec 7y ago
  So would it be roughly accurate to say that Bayard is to Tantivy what Elasticsearch is to Lucene?
  - fulmicoton 7y ago
    Precisely yes.
  - g4nt1 7y ago
    There is toshi-search https://github.com/toshi-search/Toshi https://github.com/toshi-search/Toshi who is trying to be a drop-in replacement for Elasticsearch. My understanding is that Bayard is trying to achieve the same use cases as Elasticsearch but with a different API
mmoez 7y ago
"X written in Rust" is becoming a tiring clickbait pattern on tech boards. Why not simply announcing "X" in the title?
- giancarlostoro 7y ago
  I like knowing what language something is coded in. It makes me more likely to look into the project. If it's written in something I'm not interested in I may click through, but not be as thorough, and some languages I save the link for later because I have no interest in them professionally or on my time off. I like looking at all projects eventually because some people come up with amazing pieces of software in all types of languages, but others might not care to look at a Ruby, PHP, NodeJS, Python, C, C++, Rust etc project.
  - mmoez 7y ago
    So per your rationale, why not listing in the title other pertinent information about the project? I am saying that because the programming language is not what defines a project. It could be a pile of junk even if it written in the greatest language ever made.
    - rapsey 7y ago
      It can be a pile of junk because of the language it is written in.
      - mmoez 7y ago
        > It can be a pile of junk because of the language it is written in. So many wonderful things were written in assembly or PHP (assuming you rate PHP and assembly on the other end of the spectrum of awesomeness.)
        rapsey 7y ago
        The ratio of garbage to awesome is relevant not individual examples
fnord123 7y ago
1 commit. Is this an opening of a mature project that has been coded in private somewhere? Is this just a code drop on the community? Note: this comes from a developer in Japan. Tantivy's main developer is also based in Japan. @fulmicoton, is there any interaction between the projects?
- chopraaa 7y ago
  The creator of Bayard is apparently the co-created of Solr. Bayard looks like a search-in-rust PoC.
- jsd1982 7y ago
  Not all projects are birthed in public. It may have been extracted from a larger private project which may have issues sharing its exact history of development.
- wil421 7y ago
  I was looking at Raspberry Pi projects for Rust. There were similar complaints on the Pi forum. Looks I’ll be using Python for my projects.
  - coldtea 7y ago
    Because...
    - wil421 7y ago
      Libraries weren’t maintained or in a state of development for months without regular commits. Statements like “sometimes this doesn’t work” or commits that totally redesigned an api. Meaning you have to rewrite your code. I need stability and some maturity in personal projects due to time constraints. Not a jab at Rust, I’m still in wait and watch mode. Looking to learn a low level language in the next year and Rust is my top choice.
  - wil421 7y ago
    Lots of downvotes but no replies. Rust is the language de jour on HN at the moment. Lots of comments about it being a young language so you’ll just have to wait. I invite a healthy debate.
rapsey 7y ago
Raft storage in-memory only. Not exactly safe replication.
lolive 7y ago
Is the query langage less oscure than the query langage of ElasticSearch?
NPMaxwell 7y ago
FYI: Who was Bayard Rustin? https://en.wikipedia.org/wiki/Bayard_Rustin https://en.wikipedia.org/wiki/Bayard_Rustin It's a silly play on words celebrating one of the very great heroes of 20th Century America
- MS90 7y ago
  I wonder if he was named after Le Bon Chevalier? https://en.wikipedia.org/wiki/Pierre_Terrail,_seigneur_de_Bayard https://en.wikipedia.org/wiki/Pierre_Terrail,_seigneur_de_Ba...
  - fulmicoton 7y ago
    https://twitter.com/minoru_osuka/status/1190236816277790720?s=19 https://twitter.com/minoru_osuka/status/1190236816277790720?...
- seanc 7y ago
  I assumed the same thing. Quite the co-incidence. And yes, Bayard Rustin is absolutely one of the great heroes of the 20th Century.
- devy 7y ago
  Nice try. But no. https://twitter.com/minoru_osuka/status/1190236816277790720?s=19 https://twitter.com/minoru_osuka/status/1190236816277790720?...
Kinnard 7y ago
Named after Bayard Rustin??
- fulmicoton 7y ago
  No.
atheiste 7y ago
I see the author did the same search engine in Go a while ago. So I suppose the project being a side project to learn a new language. Or is there a different reason?
- mark_l_watson 7y ago
  That is a good observation. The author might also need flexible search options at work. In any case, I have some interest in Rust but don’t actively use it. I found reading through the main server.rs file interesting as example code.
- devy 7y ago
  > So I suppose the project being a side project to learn a new language. Perhaps. But the author Minoru Osuka ain't nobody[1]. He is - Engineer at Mercari, Inc. - Committer at Apache Software Foundation - Co-author of a Apache Solr book in Japanese - Ex-Yahoo! JAPAN - Ex-Rakuten So yeah, I think he knows what he's doing. [1] https://twitter.com/minoru_osuka/ https://twitter.com/minoru_osuka/
deleted 7y ago
[deleted]
jinqueeny 7y ago
FYI: this is just a PoC and is very early in the stage :)
coleifer 7y ago
The proof that rust is a meme language is evidenced by the need to include "written in rust" every time a rust project is mentioned.
- pengstrom 7y ago
  How do you define a "meme language" and why does that follow for rust?
- bishala 7y ago
  "meme language". lol thats hilarious.
- coldtea 7y ago
  That makes no sense. Devs write "written in rust" because: - it's interesting to other developers (which are HN's main audience) to see. They don't sell some shrink wrapped software, where the language doesns't matter. Nor some already established package you just download and use as is like Postgres or Bash, or whatever. - it matters for those looking for compatible stuff for their own projects (for libraries, reusable packages, etc.) - it offers certain guarantees other languages do no (e.g. memory safety, native binaries) which can be an important criterium for those looking for a project - it's important for possible collaborators to know the language (the project being Open Source and everything). - in a field where a Java based project (Lucene/Elastic Search) dominates, it is important to advertise that you offer a non-Java alternative for people who want to avoid Java/Oracle/etc. - Rust is also currently on the rise (!= meme), and thus gets new programmers, and new greenfield projects. And since those people are trying the language, they want to advertise their involvement to the community, talk about how they found the experience, etc.
  - Gene_Parmesan 7y ago
    > it offers certain guarantees... Does it really, though? Unsafe rust exists, and while the language is certainly built to strongly encourage certain safer programming practices, I don't really see it as offering any guarantees at all. If a project is open source I can go and investigate for myself, but who has the time for that? The guarantees that safe rust provides are very good for me as a developer, because it kills a large class of potential errors and will therefore theoretically make the dev process easier. But I don't really feel any trust in these 'guarantees' when I switch roles to a user of someone else's libraries or products. Most of the rest of your points I agree with, but I also agree with the original comment that it starts to make me feel just a little bit eye-rolly every time I see "...written in Rust." (And I do like the language.)
xvilka 7y ago
It would be nice to integrate all Rust alternatives to ELK stack: 1. Toshi[1] - alternative to Elasticsearch 2. Sonic[2] - alternative to Elasticsearch 3. Vector[3][4] - alternative to Logstash 4. native_spark[5] - alternative to Apache Spark [1] https://github.com/toshi-search/Toshi https://github.com/toshi-search/Toshi [2] https://github.com/valeriansaliou/sonic https://github.com/valeriansaliou/sonic [3] https://vector.dev/ https://vector.dev/ [4] https://github.com/timberio/vector https://github.com/timberio/vector [5] https://github.com/rajasekarv/native_spark https://github.com/rajasekarv/native_spark
- MuffinFlavored 7y ago
  none of these are actual drop in replacements as far as I can tell for Elasticsearch aka you can't point your Kibana instance to these "nodes" and have it speak the Elasticsearch API
- CameronNemo 7y ago
  One of logstash's main draws is as a data transformation pipeline. You can do lookups via dns or a json or csv file, for example. From what I can tell vector is just a simple log shipper.
  - manigandham 7y ago
    Vector is a source -> transform -> sink pipeline as well. There are no transforms that do lookups or joins available now but the functionality is supported if someone writes a custom transformer middleware.
- snikolaev 7y ago
  + Manticore Search - alternative to Elasticsearch (https://github.com/manticoresoftware/manticoresearch https://github.com/manticoresoftware/manticoresearch)
karterk 7y ago
Elasticseach is notoriously hard to roll out and develop against (for smaller companies especially), and so I am happy to see smaller projects in this space. I've also been working on a light, fast, typo-tolerant search engine: https://github.com/typesense/typesense https://github.com/typesense/typesense It's been around for a couple of years now, and have a few happy customers who have had great success in replacing $X0,000/year popular hosted search with Typesense!
- elephantum 7y ago
  Postgresql is quite good easy to integrate alternative to Elastic. It has built-in fulltext search: https://www.postgresql.org/docs/12/textsearch.html https://www.postgresql.org/docs/12/textsearch.html
  - mfrye0 7y ago
    Agreed. We've been using this successfully for awhile now. I'm curious though at what point something like this or ES itself would make sense for primarily text search. Is speed the biggest thing, or is it more flexibility to tweak and get better search results?
    - maxmcd 7y ago
      Search is somewhat embarrassingly parallel right? So Postgres is great until you want to shard all of your queries. Which is (of course) possible, but then you're using attributes of a tool that aren't specifically tailored to your problem space? Postgres full text search is fast and easy, but it doesn't seem much more scaleable/reliable/resilient than a clustered search solution that is shaped to the problem at hand.
      - fulmicoton 7y ago
        Is it fast though !?
    - rpedela 7y ago
      Postgres is fine if your search problem is mostly a recall problem. If N is large enough or you have small N with enough overlapping keywords (long documents) then precision becomes important. That is when you need things like BM25, PageRank, machine learning, etc and Postgres just doesn't cut it anymore. Additionally spell check, high-quality autocomplete, multiple languages are better supported and much easier to implement in ES/Solr.
wiradikusuma 7y ago
Similar space, Vespa from Yahoo: https://vespa.ai/ https://vespa.ai/
- atombender 7y ago
  Vespa is great, but is much harder to use than Elasticsearch. It's also very much geared towards ranking and not filtering. For example, you can't do exact string matches (!). All string matches are case insensitive. You also cannot index nested fields (e.g. a map or array of maps) for search. In the end, you have to munge your data considerably to make it fit Vespa's data model. It also feels odd and antiquated in many ways, with XML configurations all over the place. But it's fast!
atombender 7y ago
Speaking of, has anyone tried working with Rucene [1], the Lucene port to Rust? [1] https://github.com/zhihu/rucene https://github.com/zhihu/rucene
manigandham 7y ago
Awesome to see more competitors for elasticsearch. Added to the list: https://gist.github.com/manigandham/58320ddb24fed654b57b4ba22aceae25 https://gist.github.com/manigandham/58320ddb24fed654b57b4ba2...
jhancock 7y ago
I'm looking for an easy to use typeahead/autocomplete search solution. javascript lib for frontend paired with easy to manage, lightweight server. something modern. The dataset isn't huge. e.g. 1 million strings of no more than 512 utf-8 chars each and not reindexed more than once a day or week. clusters, sharding etc unnecessary. I keep hoping to stumble on a fully baked solution...any ideas?
- FridgeSeal 7y ago
  Would this meet your needs? https://github.com/valeriansaliou/sonic https://github.com/valeriansaliou/sonic
snitch182 7y ago
Interesting. Since the underlying engine(Tantivy) is faster than lucene - at least in their benchmarks - it should be faster that solr. Seems like the author is exploring a faster alternative to solr. I never got around to explore elasticsearch since our solr instances are running so smoothly.

283 points by jinqueeny 7y ago

It's built on top of Tantivy (https://github.com/tantivy-search/tantivy https://github.com/tantivy-search/tantivy) that implements Tha Raft Consensus Algorithm (https://raft.github.io/ https://raft.github.io/) by raft-rs (https://github.com/tikv/grpc-rs https://github.com/tikv/grpc-rs) and The gRPC (HTTP/2 + Protocol Buffers) by grpc-rs (https://github.com/tikv/grpc-rs https://github.com/tikv/grpc-rs) and rust-protobuf (https://github.com/stepancheg/rust-protobuf https://github.com/stepancheg/rust-protobuf).

clemParis 7y ago
typo: raft-rs is at https://github.com/tikv/raft-rs https://github.com/tikv/raft-rs
- jinqueeny 7y ago
  thanks
mrec 7y ago
So would it be roughly accurate to say that Bayard is to Tantivy what Elasticsearch is to Lucene?
- fulmicoton 7y ago
  Precisely yes.
- g4nt1 7y ago
  There is toshi-search https://github.com/toshi-search/Toshi https://github.com/toshi-search/Toshi who is trying to be a drop-in replacement for Elasticsearch. My understanding is that Bayard is trying to achieve the same use cases as Elasticsearch but with a different API

mmoez 7y ago

"X written in Rust" is becoming a tiring clickbait pattern on tech boards. Why not simply announcing "X" in the title?

giancarlostoro 7y ago
I like knowing what language something is coded in. It makes me more likely to look into the project. If it's written in something I'm not interested in I may click through, but not be as thorough, and some languages I save the link for later because I have no interest in them professionally or on my time off. I like looking at all projects eventually because some people come up with amazing pieces of software in all types of languages, but others might not care to look at a Ruby, PHP, NodeJS, Python, C, C++, Rust etc project.
- mmoez 7y ago
  So per your rationale, why not listing in the title other pertinent information about the project? I am saying that because the programming language is not what defines a project. It could be a pile of junk even if it written in the greatest language ever made.
  - rapsey 7y ago
    It can be a pile of junk because of the language it is written in.
    - mmoez 7y ago
      > It can be a pile of junk because of the language it is written in. So many wonderful things were written in assembly or PHP (assuming you rate PHP and assembly on the other end of the spectrum of awesomeness.)
      - rapsey 7y ago
        The ratio of garbage to awesome is relevant not individual examples

fnord123 7y ago

1 commit. Is this an opening of a mature project that has been coded in private somewhere? Is this just a code drop on the community? Note: this comes from a developer in Japan. Tantivy's main developer is also based in Japan. @fulmicoton, is there any interaction between the projects?

chopraaa 7y ago
The creator of Bayard is apparently the co-created of Solr. Bayard looks like a search-in-rust PoC.
jsd1982 7y ago
Not all projects are birthed in public. It may have been extracted from a larger private project which may have issues sharing its exact history of development.
wil421 7y ago
I was looking at Raspberry Pi projects for Rust. There were similar complaints on the Pi forum. Looks I’ll be using Python for my projects.
- coldtea 7y ago
  Because...
  - wil421 7y ago
    Libraries weren’t maintained or in a state of development for months without regular commits. Statements like “sometimes this doesn’t work” or commits that totally redesigned an api. Meaning you have to rewrite your code. I need stability and some maturity in personal projects due to time constraints. Not a jab at Rust, I’m still in wait and watch mode. Looking to learn a low level language in the next year and Rust is my top choice.
- wil421 7y ago
  Lots of downvotes but no replies. Rust is the language de jour on HN at the moment. Lots of comments about it being a young language so you’ll just have to wait. I invite a healthy debate.

rapsey 7y ago

Raft storage in-memory only. Not exactly safe replication.

lolive 7y ago

Is the query langage less oscure than the query langage of ElasticSearch?

NPMaxwell 7y ago

FYI: Who was Bayard Rustin? https://en.wikipedia.org/wiki/Bayard_Rustin https://en.wikipedia.org/wiki/Bayard_Rustin It's a silly play on words celebrating one of the very great heroes of 20th Century America

MS90 7y ago
I wonder if he was named after Le Bon Chevalier? https://en.wikipedia.org/wiki/Pierre_Terrail,_seigneur_de_Bayard https://en.wikipedia.org/wiki/Pierre_Terrail,_seigneur_de_Ba...
- fulmicoton 7y ago
  https://twitter.com/minoru_osuka/status/1190236816277790720?s=19 https://twitter.com/minoru_osuka/status/1190236816277790720?...
seanc 7y ago
I assumed the same thing. Quite the co-incidence. And yes, Bayard Rustin is absolutely one of the great heroes of the 20th Century.
devy 7y ago
Nice try. But no. https://twitter.com/minoru_osuka/status/1190236816277790720?s=19 https://twitter.com/minoru_osuka/status/1190236816277790720?...

Kinnard 7y ago

Named after Bayard Rustin??

fulmicoton 7y ago
No.

atheiste 7y ago

I see the author did the same search engine in Go a while ago. So I suppose the project being a side project to learn a new language. Or is there a different reason?

mark_l_watson 7y ago
That is a good observation. The author might also need flexible search options at work. In any case, I have some interest in Rust but don’t actively use it. I found reading through the main server.rs file interesting as example code.
devy 7y ago
> So I suppose the project being a side project to learn a new language. Perhaps. But the author Minoru Osuka ain't nobody[1]. He is - Engineer at Mercari, Inc. - Committer at Apache Software Foundation - Co-author of a Apache Solr book in Japanese - Ex-Yahoo! JAPAN - Ex-Rakuten So yeah, I think he knows what he's doing. [1] https://twitter.com/minoru_osuka/ https://twitter.com/minoru_osuka/

deleted 7y ago

[deleted]

jinqueeny 7y ago

FYI: this is just a PoC and is very early in the stage :)

coleifer 7y ago

The proof that rust is a meme language is evidenced by the need to include "written in rust" every time a rust project is mentioned.

pengstrom 7y ago
How do you define a "meme language" and why does that follow for rust?
bishala 7y ago
"meme language". lol thats hilarious.
coldtea 7y ago
That makes no sense. Devs write "written in rust" because: - it's interesting to other developers (which are HN's main audience) to see. They don't sell some shrink wrapped software, where the language doesns't matter. Nor some already established package you just download and use as is like Postgres or Bash, or whatever. - it matters for those looking for compatible stuff for their own projects (for libraries, reusable packages, etc.) - it offers certain guarantees other languages do no (e.g. memory safety, native binaries) which can be an important criterium for those looking for a project - it's important for possible collaborators to know the language (the project being Open Source and everything). - in a field where a Java based project (Lucene/Elastic Search) dominates, it is important to advertise that you offer a non-Java alternative for people who want to avoid Java/Oracle/etc. - Rust is also currently on the rise (!= meme), and thus gets new programmers, and new greenfield projects. And since those people are trying the language, they want to advertise their involvement to the community, talk about how they found the experience, etc.
- Gene_Parmesan 7y ago
  > it offers certain guarantees... Does it really, though? Unsafe rust exists, and while the language is certainly built to strongly encourage certain safer programming practices, I don't really see it as offering any guarantees at all. If a project is open source I can go and investigate for myself, but who has the time for that? The guarantees that safe rust provides are very good for me as a developer, because it kills a large class of potential errors and will therefore theoretically make the dev process easier. But I don't really feel any trust in these 'guarantees' when I switch roles to a user of someone else's libraries or products. Most of the rest of your points I agree with, but I also agree with the original comment that it starts to make me feel just a little bit eye-rolly every time I see "...written in Rust." (And I do like the language.)

xvilka 7y ago

It would be nice to integrate all Rust alternatives to ELK stack: 1. Toshi[1] - alternative to Elasticsearch 2. Sonic[2] - alternative to Elasticsearch 3. Vector[3][4] - alternative to Logstash 4. native_spark[5] - alternative to Apache Spark [1] https://github.com/toshi-search/Toshi https://github.com/toshi-search/Toshi [2] https://github.com/valeriansaliou/sonic https://github.com/valeriansaliou/sonic [3] https://vector.dev/ https://vector.dev/ [4] https://github.com/timberio/vector https://github.com/timberio/vector [5] https://github.com/rajasekarv/native_spark https://github.com/rajasekarv/native_spark

MuffinFlavored 7y ago
none of these are actual drop in replacements as far as I can tell for Elasticsearch aka you can't point your Kibana instance to these "nodes" and have it speak the Elasticsearch API
CameronNemo 7y ago
One of logstash's main draws is as a data transformation pipeline. You can do lookups via dns or a json or csv file, for example. From what I can tell vector is just a simple log shipper.
- manigandham 7y ago
  Vector is a source -> transform -> sink pipeline as well. There are no transforms that do lookups or joins available now but the functionality is supported if someone writes a custom transformer middleware.
snikolaev 7y ago
+ Manticore Search - alternative to Elasticsearch (https://github.com/manticoresoftware/manticoresearch https://github.com/manticoresoftware/manticoresearch)

karterk 7y ago

Elasticseach is notoriously hard to roll out and develop against (for smaller companies especially), and so I am happy to see smaller projects in this space. I've also been working on a light, fast, typo-tolerant search engine: https://github.com/typesense/typesense https://github.com/typesense/typesense It's been around for a couple of years now, and have a few happy customers who have had great success in replacing $X0,000/year popular hosted search with Typesense!

elephantum 7y ago
Postgresql is quite good easy to integrate alternative to Elastic. It has built-in fulltext search: https://www.postgresql.org/docs/12/textsearch.html https://www.postgresql.org/docs/12/textsearch.html
- mfrye0 7y ago
  Agreed. We've been using this successfully for awhile now. I'm curious though at what point something like this or ES itself would make sense for primarily text search. Is speed the biggest thing, or is it more flexibility to tweak and get better search results?
  - maxmcd 7y ago
    Search is somewhat embarrassingly parallel right? So Postgres is great until you want to shard all of your queries. Which is (of course) possible, but then you're using attributes of a tool that aren't specifically tailored to your problem space? Postgres full text search is fast and easy, but it doesn't seem much more scaleable/reliable/resilient than a clustered search solution that is shaped to the problem at hand.
    - fulmicoton 7y ago
      Is it fast though !?
  - rpedela 7y ago
    Postgres is fine if your search problem is mostly a recall problem. If N is large enough or you have small N with enough overlapping keywords (long documents) then precision becomes important. That is when you need things like BM25, PageRank, machine learning, etc and Postgres just doesn't cut it anymore. Additionally spell check, high-quality autocomplete, multiple languages are better supported and much easier to implement in ES/Solr.

wiradikusuma 7y ago

Similar space, Vespa from Yahoo: https://vespa.ai/ https://vespa.ai/

atombender 7y ago
Vespa is great, but is much harder to use than Elasticsearch. It's also very much geared towards ranking and not filtering. For example, you can't do exact string matches (!). All string matches are case insensitive. You also cannot index nested fields (e.g. a map or array of maps) for search. In the end, you have to munge your data considerably to make it fit Vespa's data model. It also feels odd and antiquated in many ways, with XML configurations all over the place. But it's fast!

atombender 7y ago

Speaking of, has anyone tried working with Rucene [1], the Lucene port to Rust? [1] https://github.com/zhihu/rucene https://github.com/zhihu/rucene

manigandham 7y ago

Awesome to see more competitors for elasticsearch. Added to the list: https://gist.github.com/manigandham/58320ddb24fed654b57b4ba22aceae25 https://gist.github.com/manigandham/58320ddb24fed654b57b4ba2...

jhancock 7y ago

I'm looking for an easy to use typeahead/autocomplete search solution. javascript lib for frontend paired with easy to manage, lightweight server. something modern. The dataset isn't huge. e.g. 1 million strings of no more than 512 utf-8 chars each and not reindexed more than once a day or week. clusters, sharding etc unnecessary. I keep hoping to stumble on a fully baked solution...any ideas?

FridgeSeal 7y ago
Would this meet your needs? https://github.com/valeriansaliou/sonic https://github.com/valeriansaliou/sonic

snitch182 7y ago

Interesting. Since the underlying engine(Tantivy) is faster than lucene - at least in their benchmarks - it should be faster that solr. Seems like the author is exploring a faster alternative to solr. I never got around to explore elasticsearch since our solr instances are running so smoothly.