5 ms·

How is this meaningful without detailed setup description? http://www.tpc.org/tpcc/results/tpcc_results.asp?print=false&orderby=tpm&sortby=desc http://www.tpc.

by qaq 9y ago

How is this meaningful without detailed setup description? http://www.tpc.org/tpcc/results/tpcc_results.asp?print=false&orderby=tpm&sortby=desc http://www.tpc.org/tpcc/results/tpcc_results.asp?print=false... Looking at this list of results one wonders what those results actually mean?

ElijahLynn 9y ago
At the bottom of the article it says that information is coming: "Note: We have not filed for official certification of our TPC-C results. However, we will post full reproduction steps in a forthcoming whitepaper."
- qaq 9y ago
  I guess will have to wait. The progress they made is obviously impressive but would really help if one could understand the overhead vs conventional RDBMS 5X might be OK 20X not so much.
  - espadrine 9y ago
    The minimum latency will still be at least ~5ms (worse if nodes are very far apart) and especially bad for contended reads (because of the way they use clocks for serializable operations). A traditional RDBMS does not have to worry about split brain decisions, but it can hardly do multi-master in the intuitive way.
pinars 9y ago
I think you can still drive some insights. I clicked on the TPC-C results you shared and read their executive summaries. The Oracle on SPARC cluster (at the top, 2010) performs 30.2M qualified tx/min vs the 16K tx/min in this blog post. The Oracle cluster also costs $30M, which is clearly higher than the Cockroach cluster's cost. That said, the TPC-C benchmark is new to me. Happy to update this comment if I'm misreading the numbers. (Edited to incorporate the reply below.)
- arjunnarayan 9y ago
  A short note that the total cost of that SPARC cluster was $30 million. You're not misreading those numbers, but it requires a little context. We're focusing today on our improvements over CockroachDB 1.1, using a small-ish cluster. We'll be showing some more scalability with larger clusters in the coming weeks. If you've found CockroachDB performance slow in the past, you will be pleasantly surprised with this release!
  - pinars 9y ago
    Sure thing. I was primarily answering the question above - in terms of how the numbers in the TPC-C benchmark fit in. I updated my comment to reflect the cost. I think what's interesting with TPC-C is that you can sort the results based on performance or price/performance. On the price/performance metric, SPARC looks expensive. Dell has a $20K SQL Anywhere cluster that can do 113K tx/min. I wonder if anyone tried to run these benchmarks on the cloud and how one would calculate total cost of ownership there now.
    - qaq 9y ago
      you do realize it's ancient hardware thats $300-400 USD on ebay now.
      - tyingq 9y ago
        http://www.tpc.org/tpcc/results/tpcc_result_detail.asp?id=110120201 http://www.tpc.org/tpcc/results/tpcc_result_detail.asp?id=11... Yeah, but 1700 cores worth. That's still a lot of $300 boxes. Like qty 53 Sparc T3-2's for example. Which seem to be $1200 to $2k on eBay. And unsupported, end of life, etc. I'd compare CockroachDB's number to some more recent result with a similar number of cores. (If you can find one)
        qaq 9y ago
        I meant dell boxes Sparc based boxes retain some value

by qaq 9y ago

ElijahLynn 9y ago

At the bottom of the article it says that information is coming: "Note: We have not filed for official certification of our TPC-C results. However, we will post full reproduction steps in a forthcoming whitepaper."

qaq 9y ago
I guess will have to wait. The progress they made is obviously impressive but would really help if one could understand the overhead vs conventional RDBMS 5X might be OK 20X not so much.
- espadrine 9y ago
  The minimum latency will still be at least ~5ms (worse if nodes are very far apart) and especially bad for contended reads (because of the way they use clocks for serializable operations). A traditional RDBMS does not have to worry about split brain decisions, but it can hardly do multi-master in the intuitive way.

pinars 9y ago

I think you can still drive some insights. I clicked on the TPC-C results you shared and read their executive summaries. The Oracle on SPARC cluster (at the top, 2010) performs 30.2M qualified tx/min vs the 16K tx/min in this blog post. The Oracle cluster also costs $30M, which is clearly higher than the Cockroach cluster's cost. That said, the TPC-C benchmark is new to me. Happy to update this comment if I'm misreading the numbers. (Edited to incorporate the reply below.)

arjunnarayan 9y ago
A short note that the total cost of that SPARC cluster was $30 million. You're not misreading those numbers, but it requires a little context. We're focusing today on our improvements over CockroachDB 1.1, using a small-ish cluster. We'll be showing some more scalability with larger clusters in the coming weeks. If you've found CockroachDB performance slow in the past, you will be pleasantly surprised with this release!
- pinars 9y ago
  Sure thing. I was primarily answering the question above - in terms of how the numbers in the TPC-C benchmark fit in. I updated my comment to reflect the cost. I think what's interesting with TPC-C is that you can sort the results based on performance or price/performance. On the price/performance metric, SPARC looks expensive. Dell has a $20K SQL Anywhere cluster that can do 113K tx/min. I wonder if anyone tried to run these benchmarks on the cloud and how one would calculate total cost of ownership there now.
  - qaq 9y ago
    you do realize it's ancient hardware thats $300-400 USD on ebay now.
    - tyingq 9y ago
      http://www.tpc.org/tpcc/results/tpcc_result_detail.asp?id=110120201 http://www.tpc.org/tpcc/results/tpcc_result_detail.asp?id=11... Yeah, but 1700 cores worth. That's still a lot of $300 boxes. Like qty 53 Sparc T3-2's for example. Which seem to be $1200 to $2k on eBay. And unsupported, end of life, etc. I'd compare CockroachDB's number to some more recent result with a similar number of cores. (If you can find one)
      - qaq 9y ago
        I meant dell boxes Sparc based boxes retain some value