4 ms·

This very similar to cockroachdb, only mysql compatible instead of postgres compatible.

ngaut 10y ago
There are lots of other differences. The default distributed storage engine of TiDB is TiKV, and TiKV is written is Rust. The transaction mode is different, and so on. The are something similar are they are both NewSQL, both use Raft to replicate data.
- simonw 10y ago
  Any idea why the same people appear to have used Rust for their KV store but Go for their SQL frontend to it? They are both great languages, I'm just surprised to see one team with major projects spread across the two.
  - shenli3514 10y ago
    Go is good at concurrent concurrency and has great development efficiency. We could easily develop complex SQL logical and parallelize many operators. But its GC and cgo overhead is not suitable for developing storage engine. So we choose Rust to develop TiKV.
    - devty 10y ago
      Did you (or your team) find rust's development efficiency to be less than that of go? (was it lack of libraries? was rust too low-level?)
      - shenli3514 10y ago
        Yes, Go has higher development efficiency than Rust, especially for new comers. For a storage engine, the most important thing is stable. We focus on write right and stable codes. Rust's compiler is quite strict. Which is good for write right codes. We have many good programmers who is very good at writing Rust. So we do not suffer from the low development efficiency. About the libraries, we could get most of we need. But some were missing when we begin to develop TiKV (GRPC/HTTP2 for example). But thanks to Rust Team/Community, they give us great supports. GRPC for Rust is available now. We also contribute back to the community. We develop prometheus client for Rust. The Rust community is active and helpful. So we do not worry about the lack of libraries.
    - lobster_johnson 10y ago
      Isn't the overhead of calling Rust from Go quite high? Also, I'd assume that Rust would be better at expressing and pattern-matching against the sort of complex execution plan trees and query expressions you need for this sort of thing. I was looking at Apache Spark the other day, which is written in Scala, and which has a planner that goes SQL -> AST -> logical plan -> physical plan. The planner/optimizer relies extensively on pattern matching to apply rules to the logical plan (pushdown and so on), and it manages a lot of this with "match" statements and not a lot of recursion. But that stuff is murder in Go, which doesn't have pattern matching, or generics for that matter. I know it's bad because I'm in the middle of something similar right now, in Go. The Spark code is exactly how I wanted to organize it (minus the awful class inheritance they do), but that's not possible in Go.
      - ngaut 10y ago
        (TiDB developer here) Actually we are using RPC to call Rust from go, and it has nothing to do with Go or Rust, because either way we have to do RPC call(by encoding and decoding data), currently we depend on protocol buffer and customized RPC , and we are trying to migrate to gPRC. We know Go doesn't have pattern matching and generics, and it's kind of pain sometimes. But still we like Go very much because of simplicity and concurrency.
      - shenli3514 10y ago
        Go code communicate with Rust code through network. So there is no overhead. Go has interface, which could be used for expressing the AST, plan tree. We define some interface for AST node and plan tree node. It is convenient to apply some rules (predict-pushdown/column-pruning/cost-computing) on the tree. You could refer to the code in https://github.com/pingcap/tidb/blob/master/plan/plan.go https://github.com/pingcap/tidb/blob/master/plan/plan.go
        biokoda 10y ago
        > So there is no overhead. Other than going through the kernel to talk to a library..
dongxu 10y ago
TiDB's transaction model is different from Cockroach, and it's highly layered, you can use tikv(the underlying storage layer of tidb) as a key-value database without sql for better performance.
- placeybordeaux 10y ago
  Couldn't you also just create a K/V style table in cockroach?
  - ngaut 10y ago
    Hi, TiDB maps SQL table to K/V table and store the SQL table to K/V store(TiDB store these K/V to TiKV), Please see https://pingcap.github.io/blog/2016/10/17/how-we-build-tidb/ https://pingcap.github.io/blog/2016/10/17/how-we-build-tidb/

ngaut 10y ago

There are lots of other differences. The default distributed storage engine of TiDB is TiKV, and TiKV is written is Rust. The transaction mode is different, and so on. The are something similar are they are both NewSQL, both use Raft to replicate data.

simonw 10y ago
Any idea why the same people appear to have used Rust for their KV store but Go for their SQL frontend to it? They are both great languages, I'm just surprised to see one team with major projects spread across the two.
- shenli3514 10y ago
  Go is good at concurrent concurrency and has great development efficiency. We could easily develop complex SQL logical and parallelize many operators. But its GC and cgo overhead is not suitable for developing storage engine. So we choose Rust to develop TiKV.
  - devty 10y ago
    Did you (or your team) find rust's development efficiency to be less than that of go? (was it lack of libraries? was rust too low-level?)
    - shenli3514 10y ago
      Yes, Go has higher development efficiency than Rust, especially for new comers. For a storage engine, the most important thing is stable. We focus on write right and stable codes. Rust's compiler is quite strict. Which is good for write right codes. We have many good programmers who is very good at writing Rust. So we do not suffer from the low development efficiency. About the libraries, we could get most of we need. But some were missing when we begin to develop TiKV (GRPC/HTTP2 for example). But thanks to Rust Team/Community, they give us great supports. GRPC for Rust is available now. We also contribute back to the community. We develop prometheus client for Rust. The Rust community is active and helpful. So we do not worry about the lack of libraries.
  - lobster_johnson 10y ago
    Isn't the overhead of calling Rust from Go quite high? Also, I'd assume that Rust would be better at expressing and pattern-matching against the sort of complex execution plan trees and query expressions you need for this sort of thing. I was looking at Apache Spark the other day, which is written in Scala, and which has a planner that goes SQL -> AST -> logical plan -> physical plan. The planner/optimizer relies extensively on pattern matching to apply rules to the logical plan (pushdown and so on), and it manages a lot of this with "match" statements and not a lot of recursion. But that stuff is murder in Go, which doesn't have pattern matching, or generics for that matter. I know it's bad because I'm in the middle of something similar right now, in Go. The Spark code is exactly how I wanted to organize it (minus the awful class inheritance they do), but that's not possible in Go.
    - ngaut 10y ago
      (TiDB developer here) Actually we are using RPC to call Rust from go, and it has nothing to do with Go or Rust, because either way we have to do RPC call(by encoding and decoding data), currently we depend on protocol buffer and customized RPC , and we are trying to migrate to gPRC. We know Go doesn't have pattern matching and generics, and it's kind of pain sometimes. But still we like Go very much because of simplicity and concurrency.
    - shenli3514 10y ago
      Go code communicate with Rust code through network. So there is no overhead. Go has interface, which could be used for expressing the AST, plan tree. We define some interface for AST node and plan tree node. It is convenient to apply some rules (predict-pushdown/column-pruning/cost-computing) on the tree. You could refer to the code in https://github.com/pingcap/tidb/blob/master/plan/plan.go https://github.com/pingcap/tidb/blob/master/plan/plan.go
      - biokoda 10y ago
        > So there is no overhead. Other than going through the kernel to talk to a library..

dongxu 10y ago

TiDB's transaction model is different from Cockroach, and it's highly layered, you can use tikv(the underlying storage layer of tidb) as a key-value database without sql for better performance.

placeybordeaux 10y ago
Couldn't you also just create a K/V style table in cockroach?
- ngaut 10y ago
  Hi, TiDB maps SQL table to K/V table and store the SQL table to K/V store(TiDB store these K/V to TiKV), Please see https://pingcap.github.io/blog/2016/10/17/how-we-build-tidb/ https://pingcap.github.io/blog/2016/10/17/how-we-build-tidb/