5 ms·

Same as before, excellent when you little little data and limited transactions. Once things get big and you lose sync, everything goes to shit.

sgarland 2y ago
They claimed they successfully onboarded a customer with 1 PB of data at KubeCon NA. I have strong doubts about distributed DBs in general, but I also can’t see them blatantly lying in a talk.
- dangoodmanUT 2y ago
  how do you have doubts when nearly every significant company is running one, and distributed DBs are consistently proven to be the most correct (e.g. FDB, TigerBeetle)?
  - sgarland 2y ago
    Sorry, should have clarified: I have doubts about their necessity (as I stated in another comment). Most tech companies have poor knowledge of proper data modeling and SQL, leading to poor schema design, and suboptimal queries. Combine that with the fact that networked storage (e.g. EBS) is the norm, and it’s no wonder that people think they need another solution. The amount of QPS you can get out of a single DB is staggering when it’s correctly designed, and on fast hardware with local NVMe disks (or has a faster distributed storage solution). Consider that a modern NVMe drive can quite easily deliver 1,000,000+ IOPS.
    - riku_iki 2y ago
      distributed DBs provide seamless fault tolerance story usually. > Consider that a modern NVMe drive can quite easily deliver 1,000,000+ IOPS. There could be other bottlenecks, for example I am consistently experiencing that linux kernel doesn't handle memory pages allocations fast enough once disk traffic hits few GB/s, because it does it in single thread.
    - PeterCorless 2y ago
      It all depends on what kind of queries you're running. I came from the OLTP market, where you're generally doing single-row operations. Basic CRUD. Single table work on denormalized data. Now go to OLAP, and a single query might be doing multiple table joins. It might be scouring billions of records. It might need to do aggregations. Suddenly "millions of ops" might be reduced to 100 QPS. If you're lucky. And yes, that's even using fast local NVMe. It's just a different kind of query, with a different kind of result set. YMMV.
      - sgarland 2y ago
        Not sure why you think OLTP doesn’t also do complex joins. In a properly normalized schema, you’ll likely have many. But yes, OLAP is of course its own beast, and most DBs are suited for one or the other.
        PeterCorless 2y ago
        I think it's a matter of use case. Doing ad hoc data exploration on an OLTP system generally sucks the wind out of the performance. Even if you have some type of workload prioritization, isolation, and limitation, allowing data scientists and business analysts freely wandering through your production OLTP database sounds like a Bad Time. The organization might say "Okay. Maybe you should do your ad hoc exploration on an OLAP system. Preferably our data warehouse where you can let your report run for hours and we won't see a production brownout while it's running." So complexity of ad hoc joins in the warehouse generally can get more complex.
  - arccy 2y ago
    "every significant company"? i'd say most of them just use postgres or mysql and scale fine.
    - PeterCorless 2y ago
      If you look at DB-engines.com/ranking and look at all of the collective interest in all of the databases listed, you will see that the aggregate "score" of all databases combined 7105.84. Postgres is indeed popular; but it is only ranked 4th on the list, with its own score of 648.96. MySQL currently is still 50% larger in terms of interest, with a score of 998.15. Which means interest in Postgres (specifically) is only 9.13% of overall interest in databases; MySQL another 14.04%. Combined 23.27%. Is that a significant percentage of interest? Yes. Many others are a fraction of 1% of mindshare in the market. Yet the reason there are 423 systems ranked in DB-Engines is because no one size fits all data, or data query patterns, or workloads, or SLAs, or use cases. PostgreSQL and MySQL are, at the end of the day, oriented towards OLTP workloads. While you can stretch them to be used for OLAP, these are "unnatural acts." They were both designed in days long ago for far smaller datasets than typical for modern-day petabyte-scale, real-time (streaming) ingestion, cloud-native deployments. While many engineering teams have cobbled together PostgreSQL and MySQL frankenservers designed for petabyte-scale workloads, YMMV for your data ingest, and for p99s and QPS. The dynamic at play here is that there are some projects that lend themselves to "general services" databases, where MySQL or PostgreSQL or anything else to hand is useful for them. And then there are specialized databases designed for purpose for certain types of workloads, data models, query patterns, use cases, and so on. So long as "chaos" fights against "law" in the universe, you will see this desire to have "one" database standard rule them all, versus a Cambrian explosion of options for users and use cases.
      - sgarland 2y ago
        While you’re not wrong re: Postgres and MySQL not being necessarily designed for PB-scale, IME many shops with huge datasets are just doing it wrong. Storing numbers as strings, not using lookup tables for low-cardinality data, massive JSON blobs everywhere, etc. I’m not saying it fixes everything, but knowing how a DB works, and applying proper data modeling and normalization could severely reduce the size of many datasets.
- std_reply 2y ago
  TiDB is running core banking services too. I think people’s idea of scale and operating at scale is limited to their experience. You can get MySQL to run at any scale, look at Meta and Shopify. Operational complexity at that scale is a different story. Distributed databases reduce a lot of the operational complexity. To take one example: Try a DDL on a 5 TB table in any replicated MySQL topology of choice and compare it with TiDB’s Distributed execution framework.
  - sgarland 2y ago
    > Distributed databases reduce a lot of the operational complexity That’s a new one to me; never have I ever heard someone claim that making a system distributed reduces complexity. I’ve operated EDB’s Distributed Postgres at a decent scale (< 100 TB of unique data); in no way did it enable reduced complexity. For the devs, maybe – “chuck whatever data you want wherever you want, it’ll show up.” But for Ops? Absolutely not, it was a nightmare.
    - std_reply 2y ago
      I’m contrasting a single node database with bolted on replication vs a distributed SQL database that gives strong consistency out of the box.
    - std_reply 2y ago
      EDB Distributed Postgres under the hood is a single node database with bolted on replication.
PeterCorless 2y ago
Note that TiDB did subject itself to Jepsen testing (relatively) early. Here's their 2019 results: https://jepsen.io/analyses/tidb-2.1.7 https://jepsen.io/analyses/tidb-2.1.7 The devil is in the details, and anyone who is looking to implement TiDB for data correctness should read through not just this but other currently-open correctness-related Github issues: e.g., https://github.com/pingcap/tidb/issues?q=is%3Aissue%20state%3Aopen%20correctness https://github.com/pingcap/tidb/issues?q=is%3Aissue%20state%... TiDB currently has 74 open issues related to "correctness," and has closed 163. https://github.com/pingcap/tidb https://github.com/pingcap/tidb
std_reply 2y ago
This is quite an ignorant comment. TiDB routinely handles 100s of TB of data. Go watch LinkedIn’s presentation on why they choose TiDB as their strategic db going forward.
- peterson_lock 2y ago
  Do you have a link? Edit: https://www.youtube.com/watch?v=DzZ-I8WL2jM https://www.youtube.com/watch?v=DzZ-I8WL2jM

sgarland 2y ago

They claimed they successfully onboarded a customer with 1 PB of data at KubeCon NA. I have strong doubts about distributed DBs in general, but I also can’t see them blatantly lying in a talk.

dangoodmanUT 2y ago
how do you have doubts when nearly every significant company is running one, and distributed DBs are consistently proven to be the most correct (e.g. FDB, TigerBeetle)?
- sgarland 2y ago
  Sorry, should have clarified: I have doubts about their necessity (as I stated in another comment). Most tech companies have poor knowledge of proper data modeling and SQL, leading to poor schema design, and suboptimal queries. Combine that with the fact that networked storage (e.g. EBS) is the norm, and it’s no wonder that people think they need another solution. The amount of QPS you can get out of a single DB is staggering when it’s correctly designed, and on fast hardware with local NVMe disks (or has a faster distributed storage solution). Consider that a modern NVMe drive can quite easily deliver 1,000,000+ IOPS.
  - riku_iki 2y ago
    distributed DBs provide seamless fault tolerance story usually. > Consider that a modern NVMe drive can quite easily deliver 1,000,000+ IOPS. There could be other bottlenecks, for example I am consistently experiencing that linux kernel doesn't handle memory pages allocations fast enough once disk traffic hits few GB/s, because it does it in single thread.
  - PeterCorless 2y ago
    It all depends on what kind of queries you're running. I came from the OLTP market, where you're generally doing single-row operations. Basic CRUD. Single table work on denormalized data. Now go to OLAP, and a single query might be doing multiple table joins. It might be scouring billions of records. It might need to do aggregations. Suddenly "millions of ops" might be reduced to 100 QPS. If you're lucky. And yes, that's even using fast local NVMe. It's just a different kind of query, with a different kind of result set. YMMV.
    - sgarland 2y ago
      Not sure why you think OLTP doesn’t also do complex joins. In a properly normalized schema, you’ll likely have many. But yes, OLAP is of course its own beast, and most DBs are suited for one or the other.
      - PeterCorless 2y ago
        I think it's a matter of use case. Doing ad hoc data exploration on an OLTP system generally sucks the wind out of the performance. Even if you have some type of workload prioritization, isolation, and limitation, allowing data scientists and business analysts freely wandering through your production OLTP database sounds like a Bad Time. The organization might say "Okay. Maybe you should do your ad hoc exploration on an OLAP system. Preferably our data warehouse where you can let your report run for hours and we won't see a production brownout while it's running." So complexity of ad hoc joins in the warehouse generally can get more complex.
- arccy 2y ago
  "every significant company"? i'd say most of them just use postgres or mysql and scale fine.
  - PeterCorless 2y ago
    If you look at DB-engines.com/ranking and look at all of the collective interest in all of the databases listed, you will see that the aggregate "score" of all databases combined 7105.84. Postgres is indeed popular; but it is only ranked 4th on the list, with its own score of 648.96. MySQL currently is still 50% larger in terms of interest, with a score of 998.15. Which means interest in Postgres (specifically) is only 9.13% of overall interest in databases; MySQL another 14.04%. Combined 23.27%. Is that a significant percentage of interest? Yes. Many others are a fraction of 1% of mindshare in the market. Yet the reason there are 423 systems ranked in DB-Engines is because no one size fits all data, or data query patterns, or workloads, or SLAs, or use cases. PostgreSQL and MySQL are, at the end of the day, oriented towards OLTP workloads. While you can stretch them to be used for OLAP, these are "unnatural acts." They were both designed in days long ago for far smaller datasets than typical for modern-day petabyte-scale, real-time (streaming) ingestion, cloud-native deployments. While many engineering teams have cobbled together PostgreSQL and MySQL frankenservers designed for petabyte-scale workloads, YMMV for your data ingest, and for p99s and QPS. The dynamic at play here is that there are some projects that lend themselves to "general services" databases, where MySQL or PostgreSQL or anything else to hand is useful for them. And then there are specialized databases designed for purpose for certain types of workloads, data models, query patterns, use cases, and so on. So long as "chaos" fights against "law" in the universe, you will see this desire to have "one" database standard rule them all, versus a Cambrian explosion of options for users and use cases.
    - sgarland 2y ago
      While you’re not wrong re: Postgres and MySQL not being necessarily designed for PB-scale, IME many shops with huge datasets are just doing it wrong. Storing numbers as strings, not using lookup tables for low-cardinality data, massive JSON blobs everywhere, etc. I’m not saying it fixes everything, but knowing how a DB works, and applying proper data modeling and normalization could severely reduce the size of many datasets.
std_reply 2y ago
TiDB is running core banking services too. I think people’s idea of scale and operating at scale is limited to their experience. You can get MySQL to run at any scale, look at Meta and Shopify. Operational complexity at that scale is a different story. Distributed databases reduce a lot of the operational complexity. To take one example: Try a DDL on a 5 TB table in any replicated MySQL topology of choice and compare it with TiDB’s Distributed execution framework.
- sgarland 2y ago
  > Distributed databases reduce a lot of the operational complexity That’s a new one to me; never have I ever heard someone claim that making a system distributed reduces complexity. I’ve operated EDB’s Distributed Postgres at a decent scale (< 100 TB of unique data); in no way did it enable reduced complexity. For the devs, maybe – “chuck whatever data you want wherever you want, it’ll show up.” But for Ops? Absolutely not, it was a nightmare.
  - std_reply 2y ago
    I’m contrasting a single node database with bolted on replication vs a distributed SQL database that gives strong consistency out of the box.
  - std_reply 2y ago
    EDB Distributed Postgres under the hood is a single node database with bolted on replication.

PeterCorless 2y ago

Note that TiDB did subject itself to Jepsen testing (relatively) early. Here's their 2019 results: https://jepsen.io/analyses/tidb-2.1.7 https://jepsen.io/analyses/tidb-2.1.7 The devil is in the details, and anyone who is looking to implement TiDB for data correctness should read through not just this but other currently-open correctness-related Github issues: e.g., https://github.com/pingcap/tidb/issues?q=is%3Aissue%20state%3Aopen%20correctness https://github.com/pingcap/tidb/issues?q=is%3Aissue%20state%... TiDB currently has 74 open issues related to "correctness," and has closed 163. https://github.com/pingcap/tidb https://github.com/pingcap/tidb

std_reply 2y ago

This is quite an ignorant comment. TiDB routinely handles 100s of TB of data. Go watch LinkedIn’s presentation on why they choose TiDB as their strategic db going forward.

peterson_lock 2y ago
Do you have a link? Edit: https://www.youtube.com/watch?v=DzZ-I8WL2jM https://www.youtube.com/watch?v=DzZ-I8WL2jM