5 ms·

How We Achieved 40K QPS and 20B Rows of Data in a Single Scale-Out Cluster

40 points by gesaint 7y ago

rubyn00bie 7y ago
Nice article, I just wish they went into more detail as to why TiKV was chosen over other distributed stores like: Cockroach, Yugabyte, or Foundation. It'd also be nice to know the size of those 40bn rows is it 10TB or 100TB?
- Caitin_Chen 7y ago
  For database selection, there is a blog post about why choose TiDB over CockroachDB and other MySQL-based solutions: https://pingcap.com/success-stories/why-we-chose-a-distributed-sql-database-alongside-mysql-to-break-down-data-silos/ https://pingcap.com/success-stories/why-we-chose-a-distribut...
  - tyingq 7y ago
    For the impatient, they were using MySQL, and so Tikv was the easiest migration path while meeting most of their other requirements since it uses the MySQL protocol. The other databases mentioned are either Postgres protocol or NoSQL.
  - jiggawatts 7y ago
    It's interesting to see how crazy complex some of these scale-out solutions end up being, especially when they start off with something like MySQL and are forced to maintain compatibility while scaling far beyond what it was designed to do. You've got to wonder if simply sharding by either player or game would have worked well enough, even with MySQL. Worked well enough for Blizzard for over a decade! Similarly, I wonder if a single active write server paired with a handful of readable secondaries could have taken this query volume in its stride if using something like MS SQL Server on high-spec bare metal kit. Think In-Memory processing on dual AMD EPYC 2 servers with 128 cores per box. They mention billions of rows and terabytes of data, but I've heard of similar scale systems as far back as over decade ago! The Australian phone company Telstra did all their billing in a single IBM DB2 database, for example, and the incoming data in that system is just as real time. I'm probably wrong, but this smells like the company missed some basic optimisation opportunities somewhere...
    - adev_ 7y ago
      > You've got to wonder if simply sharding by either player or game would have worked well enough, even with MySQL. Worked well enough for Blizzard for over a decade! With all the respect I have for blizzard, I would not take old Blizzard as an example of scalability handling. I remember a time where there authentication system was well known to give up under load after every update.
  - PeterZaitsev 7y ago
    The headline Performance numbers better speak for "when not to use sharding" use case. 40K QPS is trivial for even medium size MySQL Deployments with high end servers being able to handle 1M QPS (simple queries) 20Bil rows is also quite trivial - with 100 Bytes rows this will be just 2TB of data. We probably look at long rows in this case and complicated queries but as they read numbers are rather unimpressive :)
chucky_z 7y ago
Calling cockroachdb "not open source" is a stretch. https://github.com/cockroachdb/cockroach https://github.com/cockroachdb/cockroach (The title is also wildly inaccurate to what's in the article, and should just have the articles title)
- jinqueeny 7y ago
  Thanks for the comment! CockroachDB is now under BSL and CCL licenses. Different users have different opinions but you are right, we need to be accurate about it. Please give me some time to consult our user and see whether we can update this case study.
  - jinqueeny 7y ago
    This case study has been updated: https://pingcap.com/success-stories/why-we-chose-tidb-over-other-mysql-based-and-newsql-storage-solutions/ https://pingcap.com/success-stories/why-we-chose-tidb-over-o...
_tkzm 7y ago
Pingcap is loosing hundreds and thousands of potential customers and millions of dollars by not having a single-binary product. The current setup is over-complicated and confusing and I would not touch it with a 10ft pole, as they say. That's why it's worth rewriting the app from mysql syntax to postgre syntax and use cockroachdb more than spending dev, ops, cloud* hours on making legacy code work with this thing. But they'll never learn. It's not like this thing came out just yesterday. I have been keeping my eye on this since it came out and I think it is maybe even superior product to cdb and possibly the best in the category, but again, i am getting head-ache just looking at the setup docs.
- est 7y ago
  Seconded. I mean what's the advantage of several confusing small binaries than lets say one single fat binaries with some switches? It's not like we are running out of disk space storing the binary file or too fat to load it into memory.
- deleted 7y ago
  [deleted]
- dongxu 7y ago
  PingCAP CTO here, thanks for these comments! We highly appreciate all the feedback! First, it’s true that the current setup/deployment of TiDB is not easy. This is something that we're making serious moves to improve. For example, A. We provide Ansible playbooks to simplify the deployment and rolling upgrade for on-prem users; B. We built and open-sourced TiDB Operator (https://github.com/pingcap/tidb-operator https://github.com/pingcap/tidb-operator) to enable TiDB on Kubernetes. We are working on a fully managed service in the public cloud (coming soon). Whether it is one binary or multiple binaries, it’ll be all transparent at the user level; C. We are improving the default or self-adaptative parameters and are continuously refining the configuration process; D. We are also trying to reduce the number of components. For example, the new version of CDC is implemented directly inside TiKV. E. We are developing TiOps tools in a single binary to improve the operating and maintaining experience of the cluster. A fair amount of customers around the world are using TiDB in their production environments and we are making sure they get our help when needed in the setup so it would not be a deal-breaker. Second, TiDB’s multiple-component or highly-layered architecture is challenging for deployment but the benefits are also obvious: A. The separation of the storage and computing layers makes it flexible and agile to scale/upgrade each layer as needed. Different layers need different types or different number of hardware resources. If the computing resources become the bottleneck, users can scale the SQL layer by adding more TiDB instances in real-time; if the bottleneck is the storage layer, they can easily add more TiKV instances to increase the storage capacity. B. As is known to many that we have donated TiKV to CNCF last year. We are fully committed to the open-source community and would like to see TiKV be the building block and foundation of the next-generation infrastructure. For example, we are happy to see some community users sit Redis on top of TiKV, and we ourselves built the TiSpark (https://github.com/pingcap/tispark https://github.com/pingcap/tispark) connector to run Apache Spark on TiKV. For more thoughts about this, please take a look at my blog: https://pingcap.com/blog/9-whys-to-ask-when-evaluating-a-distributed-database/#2-why-separate-sql-from-storage https://pingcap.com/blog/9-whys-to-ask-when-evaluating-a-dis... Feel free to give more feedback on https://github.com/pingcap/tidb https://github.com/pingcap/tidb and our community Slack channel https://pingcap.com/tidbslack https://pingcap.com/tidbslack. We're glad to discuss more with you on this issue!

40 points by gesaint 7y ago

Nice article, I just wish they went into more detail as to why TiKV was chosen over other distributed stores like: Cockroach, Yugabyte, or Foundation. It'd also be nice to know the size of those 40bn rows is it 10TB or 100TB?

Caitin_Chen 7y ago
For database selection, there is a blog post about why choose TiDB over CockroachDB and other MySQL-based solutions: https://pingcap.com/success-stories/why-we-chose-a-distributed-sql-database-alongside-mysql-to-break-down-data-silos/ https://pingcap.com/success-stories/why-we-chose-a-distribut...
- tyingq 7y ago
  For the impatient, they were using MySQL, and so Tikv was the easiest migration path while meeting most of their other requirements since it uses the MySQL protocol. The other databases mentioned are either Postgres protocol or NoSQL.
- jiggawatts 7y ago
  It's interesting to see how crazy complex some of these scale-out solutions end up being, especially when they start off with something like MySQL and are forced to maintain compatibility while scaling far beyond what it was designed to do. You've got to wonder if simply sharding by either player or game would have worked well enough, even with MySQL. Worked well enough for Blizzard for over a decade! Similarly, I wonder if a single active write server paired with a handful of readable secondaries could have taken this query volume in its stride if using something like MS SQL Server on high-spec bare metal kit. Think In-Memory processing on dual AMD EPYC 2 servers with 128 cores per box. They mention billions of rows and terabytes of data, but I've heard of similar scale systems as far back as over decade ago! The Australian phone company Telstra did all their billing in a single IBM DB2 database, for example, and the incoming data in that system is just as real time. I'm probably wrong, but this smells like the company missed some basic optimisation opportunities somewhere...
  - adev_ 7y ago
    > You've got to wonder if simply sharding by either player or game would have worked well enough, even with MySQL. Worked well enough for Blizzard for over a decade! With all the respect I have for blizzard, I would not take old Blizzard as an example of scalability handling. I remember a time where there authentication system was well known to give up under load after every update.
- PeterZaitsev 7y ago
  The headline Performance numbers better speak for "when not to use sharding" use case. 40K QPS is trivial for even medium size MySQL Deployments with high end servers being able to handle 1M QPS (simple queries) 20Bil rows is also quite trivial - with 100 Bytes rows this will be just 2TB of data. We probably look at long rows in this case and complicated queries but as they read numbers are rather unimpressive :)

chucky_z 7y ago

Calling cockroachdb "not open source" is a stretch. https://github.com/cockroachdb/cockroach https://github.com/cockroachdb/cockroach (The title is also wildly inaccurate to what's in the article, and should just have the articles title)

jinqueeny 7y ago
Thanks for the comment! CockroachDB is now under BSL and CCL licenses. Different users have different opinions but you are right, we need to be accurate about it. Please give me some time to consult our user and see whether we can update this case study.
- jinqueeny 7y ago
  This case study has been updated: https://pingcap.com/success-stories/why-we-chose-tidb-over-other-mysql-based-and-newsql-storage-solutions/ https://pingcap.com/success-stories/why-we-chose-tidb-over-o...

_tkzm 7y ago

Pingcap is loosing hundreds and thousands of potential customers and millions of dollars by not having a single-binary product. The current setup is over-complicated and confusing and I would not touch it with a 10ft pole, as they say. That's why it's worth rewriting the app from mysql syntax to postgre syntax and use cockroachdb more than spending dev, ops, cloud* hours on making legacy code work with this thing. But they'll never learn. It's not like this thing came out just yesterday. I have been keeping my eye on this since it came out and I think it is maybe even superior product to cdb and possibly the best in the category, but again, i am getting head-ache just looking at the setup docs.

est 7y ago
Seconded. I mean what's the advantage of several confusing small binaries than lets say one single fat binaries with some switches? It's not like we are running out of disk space storing the binary file or too fat to load it into memory.
deleted 7y ago
[deleted]
dongxu 7y ago
PingCAP CTO here, thanks for these comments! We highly appreciate all the feedback! First, it’s true that the current setup/deployment of TiDB is not easy. This is something that we're making serious moves to improve. For example, A. We provide Ansible playbooks to simplify the deployment and rolling upgrade for on-prem users; B. We built and open-sourced TiDB Operator (https://github.com/pingcap/tidb-operator https://github.com/pingcap/tidb-operator) to enable TiDB on Kubernetes. We are working on a fully managed service in the public cloud (coming soon). Whether it is one binary or multiple binaries, it’ll be all transparent at the user level; C. We are improving the default or self-adaptative parameters and are continuously refining the configuration process; D. We are also trying to reduce the number of components. For example, the new version of CDC is implemented directly inside TiKV. E. We are developing TiOps tools in a single binary to improve the operating and maintaining experience of the cluster. A fair amount of customers around the world are using TiDB in their production environments and we are making sure they get our help when needed in the setup so it would not be a deal-breaker. Second, TiDB’s multiple-component or highly-layered architecture is challenging for deployment but the benefits are also obvious: A. The separation of the storage and computing layers makes it flexible and agile to scale/upgrade each layer as needed. Different layers need different types or different number of hardware resources. If the computing resources become the bottleneck, users can scale the SQL layer by adding more TiDB instances in real-time; if the bottleneck is the storage layer, they can easily add more TiKV instances to increase the storage capacity. B. As is known to many that we have donated TiKV to CNCF last year. We are fully committed to the open-source community and would like to see TiKV be the building block and foundation of the next-generation infrastructure. For example, we are happy to see some community users sit Redis on top of TiKV, and we ourselves built the TiSpark (https://github.com/pingcap/tispark https://github.com/pingcap/tispark) connector to run Apache Spark on TiKV. For more thoughts about this, please take a look at my blog: https://pingcap.com/blog/9-whys-to-ask-when-evaluating-a-distributed-database/#2-why-separate-sql-from-storage https://pingcap.com/blog/9-whys-to-ask-when-evaluating-a-dis... Feel free to give more feedback on https://github.com/pingcap/tidb https://github.com/pingcap/tidb and our community Slack channel https://pingcap.com/tidbslack https://pingcap.com/tidbslack. We're glad to discuss more with you on this issue!