4 ms·

Distributed SQL database (TiDB) source code explained

102 points by simonz05 10y ago

spullara 10y ago
I am watching this project very closely. So far I think it has the potential to be a full featured replacement for FoundationDB. Note that they rewrote the underlying distributed, sorted, transactional KV store from Go to Rust which is the piece I am most interested in.
- shenli3514 10y ago
  Thanks for your attention.
Hayvok 10y ago
I'd be more interested to see some articles and posts on how to use TiDB and/or TiKV in interesting ways. There's been plenty of explanation on how its built, their process, the architecture, etc. Some practical application would be helpful now to help contextualize those decisions.
- c4pt0r 10y ago
  Hi from TiDB developer, we have many users in China, some of them have already used TiDB in production. Most of them treat TiDB as a drop-in replacement of MySQL sharding solution, and some of them use TiDB as an Ad-Hoc OLAP database with transactional insert/update/delete support.
marknadal 10y ago
This takes a lot of work to do, kudos! Don't listen to any haters, this is the type of document you can use to reply to them. Great job, keep it up. Cheers! ~ A friendly distributed DB competitor
- ngaut 10y ago
  Thank you,We are still working on the next RC release.
- c4pt0r 10y ago
  Thanks, that means a lot to us, btw happy Chinese new year! 新年快乐! :)
elvinyung 10y ago
Within the spectrum of so-called "NewSQL" databases, there are basically two kinds: * ones built on top of a mature relational database (e.g. Vitess, CitusDB) * ones built on top of some custom or unproven KV store (e.g. TiDB, CockroachDB, NuoDB) Ultimately, despite claims of benchmarks/elegance/MySQL not being web scale enough, I'm unsure if (at least in the immediate future) it's a wise move to do the latter. Solutions that use a proven storage engine seem like an easier pill to swallow.
- biokoda 10y ago
  TiDB and CockroachDB both use Facebook's rocksdb. I would not call it an unproved KV store. Both also use etcd's raft implementation which is also quite well proven.
  - elvinyung 10y ago
    Sure, but I wouldn't exactly call them mature yet. I think MySQL is still much easier to trust than RocksDB. (I don't actually know much about RocksDB, but I remember that LevelDB had a bunch of data corruption issues. Hopefully RocksDB doesn't.) To quote the famous video[1]: "Relational databases have been around since the fucking 70s and are some of the most mature technology you can find." 1: https://www.youtube.com/watch?v=b2F-DItXtZs https://www.youtube.com/watch?v=b2F-DItXtZs
    - ngaut 10y ago
      RocksDB is widely used in many of projects for years, i think it's quite mature. Not just TiDB and CockroachDB, MyRocks and MongoRocks are using RocksDB as storage engine too. Also there are plenty of other projects are using RocksDB. If you are worry about the adding new feature of RocksDB, yes, you are right, but all of the storage engine are changing rapidly(even InnoDB which is used by MySQL as default engine), Anyway, happy hacking :)
- assface 10y ago
  > Within the spectrum of so-called "NewSQL" databases, there are basically two kinds: The guy that invented the term "NewSQL" seems to disagree with you: https://sigmodrecord.org/publications/sigmodRecord/1606/pdfs/07_forum_Pavlo.pdf https://sigmodrecord.org/publications/sigmodRecord/1606/pdfs...
lobster_johnson 10y ago
I think TiKV on its own looks quite interesting. I see that the readme considers it to be part of TiDB, but I can see uses for TiKV on its own, without TiDB. Do you have anyone doing this in production?
- shenli3514 10y ago
  Here is two examples about how to use TiKV's KV API: https://github.com/pingcap/tidb/blob/master/cmd/benchkv/main.go https://github.com/pingcap/tidb/blob/master/cmd/benchkv/main... https://github.com/pingcap/tidb/blob/master/cmd/benchraw/main.go https://github.com/pingcap/tidb/blob/master/cmd/benchraw/mai...
deleted 10y ago
[deleted]

102 points by simonz05 10y ago

I am watching this project very closely. So far I think it has the potential to be a full featured replacement for FoundationDB. Note that they rewrote the underlying distributed, sorted, transactional KV store from Go to Rust which is the piece I am most interested in.

shenli3514 10y ago
Thanks for your attention.

Hayvok 10y ago

I'd be more interested to see some articles and posts on how to use TiDB and/or TiKV in interesting ways. There's been plenty of explanation on how its built, their process, the architecture, etc. Some practical application would be helpful now to help contextualize those decisions.

c4pt0r 10y ago
Hi from TiDB developer, we have many users in China, some of them have already used TiDB in production. Most of them treat TiDB as a drop-in replacement of MySQL sharding solution, and some of them use TiDB as an Ad-Hoc OLAP database with transactional insert/update/delete support.

marknadal 10y ago

This takes a lot of work to do, kudos! Don't listen to any haters, this is the type of document you can use to reply to them. Great job, keep it up. Cheers! ~ A friendly distributed DB competitor

ngaut 10y ago
Thank you,We are still working on the next RC release.
c4pt0r 10y ago
Thanks, that means a lot to us, btw happy Chinese new year! 新年快乐! :)

elvinyung 10y ago

Within the spectrum of so-called "NewSQL" databases, there are basically two kinds: * ones built on top of a mature relational database (e.g. Vitess, CitusDB) * ones built on top of some custom or unproven KV store (e.g. TiDB, CockroachDB, NuoDB) Ultimately, despite claims of benchmarks/elegance/MySQL not being web scale enough, I'm unsure if (at least in the immediate future) it's a wise move to do the latter. Solutions that use a proven storage engine seem like an easier pill to swallow.

biokoda 10y ago
TiDB and CockroachDB both use Facebook's rocksdb. I would not call it an unproved KV store. Both also use etcd's raft implementation which is also quite well proven.
- elvinyung 10y ago
  Sure, but I wouldn't exactly call them mature yet. I think MySQL is still much easier to trust than RocksDB. (I don't actually know much about RocksDB, but I remember that LevelDB had a bunch of data corruption issues. Hopefully RocksDB doesn't.) To quote the famous video[1]: "Relational databases have been around since the fucking 70s and are some of the most mature technology you can find." 1: https://www.youtube.com/watch?v=b2F-DItXtZs https://www.youtube.com/watch?v=b2F-DItXtZs
  - ngaut 10y ago
    RocksDB is widely used in many of projects for years, i think it's quite mature. Not just TiDB and CockroachDB, MyRocks and MongoRocks are using RocksDB as storage engine too. Also there are plenty of other projects are using RocksDB. If you are worry about the adding new feature of RocksDB, yes, you are right, but all of the storage engine are changing rapidly(even InnoDB which is used by MySQL as default engine), Anyway, happy hacking :)
assface 10y ago
> Within the spectrum of so-called "NewSQL" databases, there are basically two kinds: The guy that invented the term "NewSQL" seems to disagree with you: https://sigmodrecord.org/publications/sigmodRecord/1606/pdfs/07_forum_Pavlo.pdf https://sigmodrecord.org/publications/sigmodRecord/1606/pdfs...

lobster_johnson 10y ago

I think TiKV on its own looks quite interesting. I see that the readme considers it to be part of TiDB, but I can see uses for TiKV on its own, without TiDB. Do you have anyone doing this in production?

shenli3514 10y ago
Here is two examples about how to use TiKV's KV API: https://github.com/pingcap/tidb/blob/master/cmd/benchkv/main.go https://github.com/pingcap/tidb/blob/master/cmd/benchkv/main... https://github.com/pingcap/tidb/blob/master/cmd/benchraw/main.go https://github.com/pingcap/tidb/blob/master/cmd/benchraw/mai...

deleted 10y ago

[deleted]