10 ms·
Toshi: An Elasticsearch competitor written in Rust
- nathcd 8y agoNeat! I hope this goes far, it'd be great to have a faster/lighterweight Elastcsearch. Something similar I'm really hoping to see is Tantivy in a Postgres extension, so I can stop playing the game of trying to keep my search engine and database in sync. Seeing pg-extend-rs (https://github.com/bluejekyll/pg-extend-rs https://github.com/bluejekyll/pg-extend-rs) on HN the other week got me thinking about it again. Does anyone know whether this is feasible or if anyone is working on something in this vein?
- aldoushuxley001 8y agoI know you can go pretty far just within postgresql these days without any extensions, even for production grade search. For my projects thats all I've been doing, in order to avoid the overhead of ensuring ES or Solr are synced with db.
- GordonS 8y agoJust released a system that does the same - Postgres FTS can get you a long way. I've used Elasticsearch before, and it's a lot of complexity to add, and isn't always necessary.
- profquail 8y agoOut of curiosity —- have you looked at using Postgresql’s full text search functionality to implement your search engine (e.g. [1])? If so, what do you get out of the combination of Postgres + Elasticsearch that you chose it over just the Postgres full text search? [1] http://rachbelaid.com/postgres-full-text-search-is-good-enough/ http://rachbelaid.com/postgres-full-text-search-is-good-enou...
- Ralfp 8y agoMajor problem with Postgres full-text search that those articles don't dwell into too much is that unless your documents are in one of the "chosen languages", you are more likely to find support for your language in search engine (like ElasticSearch) than get it on PostgreSQL. You can convert existing dictionaries available to format Postgres understand, but this is annoying pain point if you happen to be an open source project like CMS or communication platform.
- hashhar 8y agoI don't get the hype about elasticsearch at all. Elasticsearch is more suited to searching logs. It doesn't have powerful sort functions, doesn't allow you to use multiple sort parameters etc. Apache Solr is more suited to search. Lots of document filters, query filters, the index itself is highly configurable and the ability to sort on multiple parameters is great. LTR is also something too good to miss out on.
- rpedela 8y agoCan you explain what you mean by "multiple sort parameters" because it looks like you can to me in ES [1]. There is a well maintained LTR plugin for ES. Honestly Solr and ES are more similar than different. There are a few things Solr has which ES doesn't and the reverse is true too. 1. https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-sort.html https://www.elastic.co/guide/en/elasticsearch/reference/curr...
- hashhar 8y agoThanks for letting me know about Elasticsearch LTR.
- bryanrasmussen 8y agoBoth solr and elasticsearch use lucene to index.
- hashhar 8y ago
- burntsushi 8y agoLast time I checked, PostgreSQL doesn't support any kind of tfidf ranking scheme because it doesn't track corpus frequencies of terms in the index. This impacts how well relevance ranking works for some workloads.
- lykr0n 8y agoZoomDB (https://github.com/zombodb/zombodb https://github.com/zombodb/zombodb) offers something like this.
- FridgeSeal 8y agoThere’s a super neat plugin for Postgres called “ZomboDB” that hooks up ElasticSearch to Postgres, so when you make full text queries, it gets performed on ES. https://www.zombodb.com/ https://www.zombodb.com/ I had a brief play with it for a project at work and it was super straightforward to get running.
- jedisct1 8y agoPGroonga https://pgroonga.github.io/ https://pgroonga.github.io/
- markhenderson 8y agoIt would be great if this achieved API parity with ES. Being able to swap out parts of the ELK stack would make tools like kibana even more powerful.
- mdaniel 8y agoI hear you, and I know why you'd say that, but wow the API surface-area of ES is ginormous. Maybe the 80-20 rule goes a long way here, but I wouldn't expect API parity to be a simple matter of exposing the same REST endpoints -- it's the payload that'll be the headache I actually strongly considered that just with Solr, which has the extreme benefit of using the same query language under the hood, but the more I scratched the more I found it would be a horrific amount of work
- jazoom 8y agoPlus the Elasticsearch API isn't especially nice to use. I haven't tried their new SQL, since it requires an enterprise licence or something.
- mdaniel 8y agoDo you mean encoding the lucene queries as JSON objects into the ES endpoints, or do you mean the actual lucene syntax (as would be surfaced by kibana et al)?
- jazoom 8y agoI mean the Elasticsearch API. Kinda what you were referring to in the first part of your sentence, but I don't know why you'd say it like that, especially since the Elasticsearch API covers other things, such as mapping indexes and other cluster administration.
- Proven 8y ago> auto_commit_duration = 10 That's interval, not duration.
- latenightcoding 8y agoI love these type of projects. I am a big fan of Elasticsearch, but sometimes it feels overly complex, bloated and memory-hungry. I hope someday a decent Rust/C++ alternative will take over. I was following the development of Apache Lucy (https://lucy.apache.org/ https://lucy.apache.org/) but the project has been retired now.
- bluejekyll 8y ago> I hope someday a decent Rust/C++ alternative will take over. I found this part of your comment interesting, given that Rust is in someways being offered as an alternative to C++ for similar use cases. What would you like to see different in a language that you see as the common issue with C++ and Rust? edit: I misread the parent comment, the question should be disregarded.
- jazoom 8y agoI'm pretty sure he/she was saying they want to see something like Elasticsearch written in Rust/C++ rather than Java and made to be less bloated and less complex. I would also be very interested in that. Elasticsearch usually works very well for me, especially the latest version, but it feels very heavy, and seems to only be getting worse in that regard.
- bluejekyll 8y agoOh! I completely misread that. Thanks for clarifying.
- aditirex 8y agoThere is Manticore Search (https://github.com/manticoresoftware/manticoresearch https://github.com/manticoresoftware/manticoresearch) - it's a fork of Sphinx - written in C++, not based on Lucene library. (disclaimer: I was/am involved in both projects)
- AcentoProsaico 8y agoHave you tried Xapiand? (https://kronuz.io/Xapiand/ https://kronuz.io/Xapiand/). It's a Xapian-based search and storage engine written in C++! We've been using it for a while in a number of projects and it's quite simple and reliable.
- CSDude 8y agoTantivy does not allow schema evolution, but Lucene does, this is a major blocker for dynamic indices
- fulmicoton 8y agoSo technically, it is supposed to allow you to add field... but it is not well tested so I try to keep that a secret. I should probably work on a proper scenario for schema evolution.
- Svenstaro 8y agoSo what's the modus for that case? Rebuild the whole DB?
- georgecalm 8y ago> What is a Toshi? > Toshi is a three year old Shiba Inu. He is a very good boy and is the official mascot of this project. Toshi personally reviews all code before it is commited to this repository and is dedicated to only accepting the highest quality contributions from his human. He will though accept treats for easier code reviews.
- pornel 8y agoThere are some real gems among Rust crates. I'm using tantivy[1]. It has been super easy to set up and it's faaast. [1]: https://crates.rs/crates/tantivy https://crates.rs/crates/tantivy
- coldtea 8y agoToshi is built on top of tantivy if I'm not mistaken...
- kevsim 8y agoYep. Very first line of the README
- fulmicoton 8y agoThanks! If you can share your use case, I'm interested.
- badrabbit 8y agoLove the idea, saw another one in 2018(https://www.gravwell.io/ https://www.gravwell.io/) At least in my experience the web interface is a huge gap. Kibana is ok but Splunk and it's query language and visuals are much better. Anything that competes with Splunk is great imo.
- remasis 8y agoHey, Corey from Gravwell here. Huuuge updates coming to the web interface this month! You can see teaser screenshots on our website and a blog post coming this week using the new UI. Subscribe to the blog (we are non-spammy) to get the email announcement.
- fulmicoton 8y agoHi! -tantivy main dev here-. Do you have a engineering dev blog that disclose information about your backend?
- cthuen 8y agoWe haven't published a lot on the backend architecture, but there's some info in the docs. I think it would be interesting to chat. Can you hit me up at info@gravwell.io?
- ivoras 8y agoCool project! But there's an attitude attached which amuses me: > Toshi will always target stable Rust and will try our best to never make any use of unsafe Rust. While underlying libraries may make some use of unsafe, Toshi will make a concerted effort to vet these libraries in an effort to be completely free of unsafe Rust usage. The reason I chose this was because I felt that for this to actually become an attractive option for people to consider it would have to have be safe, stable and consistent. This was why stable Rust was chosen because of the guarantees and safety it provides. It's an admirable goal, though the fact that it's stated prominently as one of the first things on the front page gives off somewhat of a "doth protest too much" vibe. It's not like "safety" is rare these days. Any project written in Go, Python, Java, C#, Erlang, JS, and a myriad of others will be "safe" as far as memory access is concerned, and in many cases this safety will be easier to achieve than in Rust. As far as error handling safety, so far exceptions seem to be more expressive, though the jury is out there. Basically, if a project stays away from C and C++ and libraries written in them, it's more likely it will be hit by a hardware problem than an inherent language safety / security issue. Luckily, "safety" is for the largest part the default for modern projects.
- troutwine 8y agoThere's an inside baseball discussion happening here with the larger Rust community that I think, maybe, you're missing. You are absolutely correct in saying that Go, Python et al offer memory safety in a way that low-level languages do not. These languages use techniques that incure some kind of runtime penalty. Erlang, for instance, copies memory like it's going out of style. Now, with Rust you get a similar kind of memory safety but, somewhat uniquely, without the same category of runtime penalty. There's still some, sometimes, and that's where the community discussion around 'unsafe' happens in Rust. So, you're writing in Rust and you'd like to write code which has the absolute most aggressive performance possible. In Rust parlance this _maybe_ means you need to do unsafe things: turn off bounds checks, fiddle with the raw memory of a structure, allow multiple threads to access the same memory without synchronization, interact with mutable globals and so on. That's great, but, you've potentially opened up your program to crashes or security issues. If you're a solo author that's a trade-off you can make based on your needs. Here's the rub: if you use similar techniques in a crate -- a shared library for all to use -- then you've opted everyone using your crate into the same trade-off, one which they might not have otherwise chosen for themselves. What the Toshi project is saying here is that it's design preference is to avoid opting into this trade-off, preserving all the guarantees that Rust can provide at _possibly_ the expense of absolute performance. There's a safety-focused subset of the Rust community that takes the presence of an 'unsafe' in a body of Rust code very seriously and this project is participating in that conversation.
- zozbot123 8y agoYet another great piece of work by the Rewrite-It-In-Rust Task Force (a sister organization to the Rust Evangelism Strike Force)!
- supernintendo 8y agoCute!
- fxfan 8y agodang?
- arminiusreturns 8y agoDoes anyone know of any other prod ready elasticsearch alternatives? I'm working on a logging infrastructure project for shipping syslog, and it seems no one these days just uses plain central syslog and ES is the standard, but it seems bloated. I've been tempted to just ship straight to a dB and skip all these crazy shippers and parsers and all the other middle men in the equation. Also, why has no product unified monitoring and logging? AFAIK that's why Splunk is worth it is you have the budget (I dont)
- FBISurveillance 8y agoIf you're looking for SaaS solution, check out DataDog. They offer monitoring and logging (and a bit more) as a packaged service.
- noahl 8y agoBack when I worked at Google, the standard log processing tool was Dremel. You could get exactly the same thing by shipping your logs to BigQuery. I haven't checked, but I bet it's cheaper than ES for data that's mostly cold, like logs. You will need a separate monitoring solution though.
- dominotw 8y agowe use spark on hdfs.
- mentat 8y agoIf you want streaming inserts to BQ, that becomes the biggest cost. Dataflow could be used to turn inserts into batch and gather interesting metrics that you don't want to hit BQ for, but I don't think anyone's open sourced anything in this space. I've implemented streaming inserts to BQ for logs "at scale" and it was at least an order of magnitude cheaper than splunk still. Happy to talk via email.
- dominotw 8y agowhat was the monitoring solution at the end, where were BQ results going to?
- lettergram 8y agoNice! I’m going to have to check it out. At the same time, I’m curious on the performance differences with postgres. I’ve been able to get very quick queries from Postgres: https://austingwalters.com/fast-full-text-search-in-postgresql/ https://austingwalters.com/fast-full-text-search-in-postgres... The only time it’s slower to the point of using elasticsearch (for my usecases) is something like log search (so far).
- ilikehurdles 8y agoWell that’s a big use case that ES is meant for. There’s also the fact that ES is easy to shard. At my previous gig we easily had over terabyte of data written into ES indices every day, with hundreds of TBs worth of documents searchable in various indices, and none of this is even counting the logging use case. Used it extensively for calculating various aggregations (for reporting/analytics) of events.
- amelius 8y agoAlso check out the roadmap: > https://github.com/toshi-search/Toshi/blob/master/roadmap.md https://github.com/toshi-search/Toshi/blob/master/roadmap.md It seems that the main added functionality of ES, namely clustering, will still take a while to be implemented.