7 ms·
Tikv: Distributed key value store powered by Rust
- biokoda 11y agoDistributed KV store based on rocksdb. Isn't that just reinventing the wheel? Is the fact that it is written in Rust that important?
- Zikes 11y agoLanguage is one of the more important features for attracting other contributors to open source projects. As to reinventing the wheel, given the number of competing tire companies that exist apparently even the wheel still has room to improve.
- biokoda 11y agoThat's a silly and un-applicable example. How many tire startups are there?
- Zikes 11y agoAt least one: http://www.timesofisrael.com/with-innovative-tire-israeli-start-up-reinvents-the-wheel/ http://www.timesofisrael.com/with-innovative-tire-israeli-st...
- andremendes 11y agoThis is offtopic but as I do not have the time to investigate what this website tries to do, I must alert that it completely froze my system, twice. Had to actually do a physical reset.
- arthursilva 11y agoIt's transactional and mvcc, you can't say the same for most KVS around.
- stephen_mcd 11y agoCurioDB too: http://curiodb.jupo.org http://curiodb.jupo.org
- dikaiosune 11y agoIt looks like a parallel implementation of TiDB, which they had built in Go. I would assume that avoiding garbage collection and reducing heap usage were valid motivations for the project. But it doesn't really have enough information on the repo to tell what their motivations are.
- bpicolo 11y agoNot quite. TiDB can use it as a data layer, as an alternative to Rocks, Bolt, etc. Different part of the system
- dv_dt 11y agoSo when the Readme says "Distributed KV Database", I'm imagining that it refers maybe to being able to be used by TiDB which provides the distributed logic, while TiKV concentrates on a compatible interface to TiDB and local non-distributed storage?
- bpicolo 11y agoRight. It should be roughly equivalent to InnoDB/MyISAM alternatives in Mysql (I believe)?
- lrem 11y agoTo be fair, I was considering learning Rust and implementing a distributed hash table just as an exercise. With a like-minded team, reimplementing a toy Spanner sounds like a fun project.
- alexnewman 11y agoI appreciate the sentiment, but this project is very difficult to understand what's connected to what. That's not a good sign for a database
- dikaiosune 11y agoThe first PR was merged ~3 months ago, I'm not surprised that they don't have more info in the README.
- andrewvijay 11y agoI'm pretty much sure this software wouldn't get a lot of attention if it was not built in rust. Darn I need to build something in rust and get some attention!
- ddorian43 11y agoYou're always welcome to build an opensource transactional bigtable clone!
- mindcrime 11y agoAnd do it in INTERCAL or Befunge to boot!
- dang 11y agoSure, trends come in trends—in programming as in other things. But please don't post comments that put down other people's work. If you think other stories should get attention on HN, please submit some.
- andrewvijay 11y agoIt's a joke man. Why would I put down someone I don't even know!
- dang 11y ago> Why would I put down someone I don't even know I share your question, yet it seems to be what most internet forums are for.
- arthursilva 11y agoI think everyone could benefit from extracting that raft implementation to a crate. I'll see if I can get started on that.
- viperscape 11y agoI think hoverbear has a raft crate: https://github.com/Hoverbear/raft-rs https://github.com/Hoverbear/raft-rs
- bluejekyll 11y agoYes, I was just wondering why they didn't use that? I reviewed that raft implementation. Looks decent and uses mio for non-blocking io, which was quite clean (I'm jealous).
- justinsb 11y agoMost raft implementation don't implement dynamic reconfiguration or snapshots/compaction. AFAICT hoverbear implements neither, and Tikv implements at least snapshots.
- bluejekyll 11y agoYou're right. And thanks for the pointer on this implementing snapshots.
- hoverbear 11y agoHi! Thanks for linking this! We're hoping to add more robust features like dynamic membership soon! I'm graduating and will have more time. If you feel like hacking on our code please just let us know, we'd love to help.
- ideal0227 11y agotikv's raft is a port of etcd/raft, which is used by a few projects in production. So we can expect rust raft to be mature soon.
- endymi0n 11y agoWith "inspired by" meaning the very same thing as in the opening credits of a Hollywood movie...
- xuejie 11y agoI know the authors of this project also read Hacker News, so it probably is better if we wait till them to state their motives. But I suppose I could give a brief first, since most of the confusing comments are due to language barriers(sorry if you don't know enough Chinese, you might not have read enough about this project). This is actually affiliated to https://github.com/pingcap/tidb https://github.com/pingcap/tidb, which is another implementation of Google Spanner/F1, similar to CockroachDB. So think TiKV as spanner, and TiDB as F1. They actually started out using Go. And the reason they are changing to Rust is not trying to be a Rust cool kid, but due to real constraints: Go's GC introduces too much of latency time (and no, Go 1.6 is not enough despite being better), they are also not satisfied with cgo's costs. That's why they are changing to Rust. Note that according to their words, they still love Go better, and TiDB is also happily using Go. But for TiKV's case, Rust is perfect. Disclaimer: I'm not working on TiKV or TiDB, I'm just a fan of their work(they are also the founders behind Codis: https://github.com/CodisLabs/codis https://github.com/CodisLabs/codis, tho not working on that anymore). I'm happy to list references on where I got all the information in case anyone are interested but they are all in Chinese, which I think might not be that polite since not everyone here reads Chinese.
- zmanian 11y agoI'm super excited to see a Mandarin speaking team building something in Spanner class of databases.
- collyw 11y agoCurious why the language that the team speaks matters to you at all?
- spullara 11y agoThis looks like it could be a drop in replacement for FoundationDB but I don't see enough documentation on the guarantees they give or limitations. Any more information? Maybe even translated from Chinese?
- xuejie 11y ago
- skyde 11y agoI could not find a description of the algorithm used for distributed transaction. I can only assume it's similar to CockroachDB but looking at the code I see a lot of reference to "LOCK" in the protocol buffer definition which mean they might be using something else. Anybody know where I can find more documentation ?
- hmottestad 11y agoSays they are using Raft: https://raft.github.io/ https://raft.github.io/
- rystsov 11y agoRaft is a replication protocol which is responsible for the consistency of multiple copies of the same data but it isn't applicable when the data is sharded.
- mtanski 11y agoYou can use raft as to synchronizing other data structures. If this a KV store then that's as easy as key level locks / ownership / placement.
- skyde 11y agoRaft is used to Replicate write in a shard but distributed transaction is used to make transaction that span several shard atomic (ACID). I could find the implementation at : https://github.com/pingcap/tikv/blob/master/src/storage/mvcc/txn.rs https://github.com/pingcap/tikv/blob/master/src/storage/mvcc... But a lot of the detail are not clear.
- c4pt0r 11y agoThe transaction model is similar to Google's Percolator, and with some performance improvements.
- skyde 11y agoThanks after seeing its based on themis code that is itself based on Percolator the distributed transaction logic make sense. Any idea how it compare to CockroachDB ?
- llogiq 11y agoFrom the README: > Thanks rust-clippy. We do love the great project. This just made my day. :-)
- brokenmachine 11y agoCan someone ELI5 what a distributed key value store is useful for? Is it like a DHT for torrents?