5 ms·
TiDB – cloud-native, distributed SQL database written in Go
- peterson_lock 2y agoI was able to get a small TiDB cluster running in my homelab last night! Looking forward to using it :) Reference: https://docs.pingcap.com/tidb-in-kubernetes/stable/get-started https://docs.pingcap.com/tidb-in-kubernetes/stable/get-start...
- otterley 2y agoTiDB is 8 years old now; what's new?
- alpb 2y agoYeah that and the actual core storage engine is a key-value store library called https://github.com/tikv/tikv https://github.com/tikv/tikv which is written in Rust. I suspect the Go part only handles the replication component, MVCC, query planner, server etc.
- tapirl 2y agoIs tikv (still) a wrapper of RocksDB?
- willvarfar 2y agoHardly hear about TiDB. How is it faring and how does it stack up to more well know competition?
- xmichael909 2y agoSame as before, excellent when you little little data and limited transactions. Once things get big and you lose sync, everything goes to shit.
- sgarland 2y agoThey claimed they successfully onboarded a customer with 1 PB of data at KubeCon NA. I have strong doubts about distributed DBs in general, but I also can’t see them blatantly lying in a talk.
- dangoodmanUT 2y agohow do you have doubts when nearly every significant company is running one, and distributed DBs are consistently proven to be the most correct (e.g. FDB, TigerBeetle)?
- sgarland 2y agoSorry, should have clarified: I have doubts about their necessity (as I stated in another comment). Most tech companies have poor knowledge of proper data modeling and SQL, leading to poor schema design, and suboptimal queries. Combine that with the fact that networked storage (e.g. EBS) is the norm, and it’s no wonder that people think they need another solution. The amount of QPS you can get out of a single DB is staggering when it’s correctly designed, and on fast hardware with local NVMe disks (or has a faster distributed storage solution). Consider that a modern NVMe drive can quite easily deliver 1,000,000+ IOPS.
- riku_iki 2y agodistributed DBs provide seamless fault tolerance story usually. > Consider that a modern NVMe drive can quite easily deliver 1,000,000+ IOPS. There could be other bottlenecks, for example I am consistently experiencing that linux kernel doesn't handle memory pages allocations fast enough once disk traffic hits few GB/s, because it does it in single thread.
- PeterCorless 2y agoIt all depends on what kind of queries you're running. I came from the OLTP market, where you're generally doing single-row operations. Basic CRUD. Single table work on denormalized data. Now go to OLAP, and a single query might be doing multiple table joins. It might be scouring billions of records. It might need to do aggregations. Suddenly "millions of ops" might be reduced to 100 QPS. If you're lucky. And yes, that's even using fast local NVMe. It's just a different kind of query, with a different kind of result set. YMMV.
- jitl 2y agoI hear through the grapevine they’ve got a bunch of companies using it. I think Airbnb migrated to it.
- travem 2y agoThere are quite a few. Pinterest, LinkedIn, Plaid, Mercari, and Rakuten shared their experiences with TiDB at PingCAP's annual event last year (2024) - https://www.pingcap.com/htap-summit/ https://www.pingcap.com/htap-summit/. In the previous event (2023) there were speakers from Airbnb, Databricks, Flipkart, PayPay, and others sharing their experiences as well - https://www.pingcap.com/htap-summit/sept-2023/ https://www.pingcap.com/htap-summit/sept-2023/ Disclosure: Employee of PingCAP the company behind TiDB
- normie3000 2y ago> PingCAP's annual event last year (2014) Wasn't last year 2024?
- travem 2y agoYes, I am still getting up to speed after a long break for Christmas and New Year!
- lambda 2y agoI think you mean 2024 and 2023...
- travem 2y agoYes, corrected!
- rpaik 2y agoNot only are these companies using TiDB, they're also contributing to it as you can see in this recent blog post: https://medium.com/@ray.paik/tidb-8-5-is-here-b2ca09e972d5 https://medium.com/@ray.paik/tidb-8-5-is-here-b2ca09e972d5
- deleted 2y ago[deleted]
- srameshc 2y agoI love these new distributed DBs. CockroachDB is one of them. Still I think a managed Postgres/MySQL is a better choice. My primary concern is how challenging will it be if you have to eventually move your data out to a RDBMS for cost or other reasons. Does anyone have any experience ? I am not talking enterprise scale but data about size of 50 - 100GB scale.
- convolvatron 2y agowhy do you think that would be harder? assume for the moment that the reader here is going to run at the effective rate of a single node and not that we're going to try to parallelize that. Assuming we have transaction isolation, that reader is going to get a consistent snapshot. a distributed database is potentially more complicated to operate, and optimize, and because its new and potentially has more sharp edges maybe less reliable (?) but the extraction of moderate sized datasets doesn't really seem to be an obvious failing.
- std_reply 2y agoI would argue the opposite, distributed databases are much easier to operate at large scale. Truly online distributed DDL, at least in TIDB, strong consistency etc. People who bang on about Postgres replication have rarely setup replication in Postgres themselves and that too in the 100a of Pb scale. MySQL replication works well and can be scaled more easily (relative to Postgres) but has its own problems. eg., DDL is still a nightmare, lag is a real problem, usually masked by async replication. But then eventual consistency makes the application developers life more complicated.
- sgarland 2y agoDistributed DBs and traditional RDBMS serve different purposes. Most – by which I mean the overwhelming majority – companies do not need a distributed DB, they need better schema and queries. My fear is companies without in-house RDBMS expertise see these products as a way to continue to avoid getting that expertise.
- debarshri 2y agoI think it is mostly three folds. Availability, to be cloud agnostic and scale of few tbs to few hundred TBs. I have been seeing cockroach and yugabyte in an on prem setup where you don't get the benefits of a cloud provider but you can get the availability guarantees and fault tolerance.
- eikenberry 2y agoVC backed database company with a CLA on their "open source" project. Red flag. Is there a community fork yet?
- zhouzhao 2y agoThank you for the warning. Whats the source on the VC part?
- russell_h 2y agoI don't share OP's hostility to VC-funded databases, but PingCAP is definitely VC-funded: https://www.pingcap.com/press-release/pingcap-the-company-behind-tidb-raises-270-million-in-series-d-funding/ https://www.pingcap.com/press-release/pingcap-the-company-be...
- eikenberry 2y agoThe VC part isn't 100% negative but it generally comes with the strings attached that highly incentivizes fast growth. Fast growth kills any company culture you have and converts you into yet another company that views all employees as cost-centers. IE. it destroys companies as good places to work. If you could get VCs to fund companies to grow slowly into stable, small to mid-sized companies I would have no problem with them. A side aspect of this is that they also destroy any motivation for developing free software other than as a marketing tool, a way to drive growth. While that is perfectly fine for companies to do this, it has long term consequences that lead to bad behavior that mean you should be extra wary of relying on them.
- hbbio 2y agoHow would you fund the open source development of a database? TiDB is Apache licensed, that should be enough no? Because if we ban all projects from VC-backed startups, you're going to clear your tech stack pretty quickly.
- foota 2y ago
- nuphonenuacc 2y agoWhat is that first paragraph? "This implementation proves the CAP theorem as wrong"? Doesn't tell me what your drawbacks are. This is mongodb all over again. Will only ever consider if the customer really wants it. (My guess performance) (Also, bold move to write a db without fully being able to manage memory).
- travem 2y agoI do want to clarify a few points, on the project page it does provide the following information: > Distributed Transactions: TiDB uses a two-phase commit protocol to ensure ACID compliance, providing strong consistency. Transactions span multiple nodes, and TiDB's distributed nature ensures data correctness even in the presence of network partitions or node failures. > High Availability: Built-in Raft consensus protocol ensures reliability and automated failover. Data is stored in multiple replicas, and transactions are committed only after writing to the majority of replicas, guaranteeing strong consistency and availability, even if some replicas fail. Geographic placement of replicas can be configured for different disaster tolerance levels. See https://github.com/pingcap/tidb?tab=readme-ov-file#key-features https://github.com/pingcap/tidb?tab=readme-ov-file#key-featu... Correctness has been a focus for a long time for TiDB, including working on passing Jepsen Tests back in 2019, see https://www.pingcap.com/blog/tidb-passes-jepsen-test-for-snapshot-isolation-and-single-key-linearizability/ https://www.pingcap.com/blog/tidb-passes-jepsen-test-for-sna... and https://jepsen.io/analyses/tidb-2.1.7 https://jepsen.io/analyses/tidb-2.1.7 Disclosure: Employee of PingCAP the company behind TiDB
- fipar 2y agoI can't find that first paragraph, and it seems that readme file was last modified 3 months ago, though you're using quotes, so perhaps you are paraphrasing? The first paragraph I find under Key Features is: Distributed Transactions: TiDB uses a two-phase commit protocol to ensure ACID compliance, providing strong consistency. Transactions span multiple nodes, and TiDB's distributed nature ensures data correctness even in the presence of network partitions or node failures. It claims to ensure correctness in the presence of partitions, not availability, so I don't think that's a claim that CAP is wrong? I would expect it to be unavailable if it can't provide a consistent response. I don't know about memory management. It seems go may not be used for the storage layer, but in any case, it wouldn't be the first attempt at a DB on a garbage collected runtime. I can think of at least Cassandra on the jvm and CockroachDB also on Go. I do prefer that DBs do their own memory management (as well as storage) but this is not as unique as it used to be though.
- std_reply 2y agoTiDB has four main components: 1. SQL front end nodes 2. Distributed shared nothing storage (TiKV) 3. Meta data server (PS) 4. TiFlash column store 1 and 3 are written in Go 2 is written in Rust and uses RocksDB 4 is written in C++ 2 & 3 are graduated CNCF projects maintained by PingCAP. Disclaimer: I work for PingCAP
- oa335 2y agoInterview with TiDB developer explaining some of the architecture of the DB : https://youtu.be/mmzoSkEhYrA?si=5eIppYzqKvOxWDtv https://youtu.be/mmzoSkEhYrA?si=5eIppYzqKvOxWDtv
- tonymet 2y agoDoes "cloud native" mean "has kubernetes config" or is there something else to it?
- geenat 2y agoNot having a single binary distribution like CockroachDB is hurting adoption of TiDB.
- std_reply 2y agoHave you looked at tiup? https://tiup.io https://tiup.io