4 ms·

Hey jiggawatts - TPC is the official way to audit benchmarks in the database industry. They’ve been around for a bit, but let me know if you want more info, I’m

by maslam 5y ago

Hey jiggawatts - TPC is the official way to audit benchmarks in the database industry. They’ve been around for a bit, but let me know if you want more info, I’m happy to share more about them.

lmeyerov 5y ago
It sounds fundamentally busted if a competitor can submit benchmarks for someone else. TPC is great in general, but I didn't realize it had such a gaping flaw. TPC submissions take real time/$/energy/expertise, so I don't know anyone who has ever done it casually. Ex: It was a multi-company effort for the RAPIDS community to get enough API coverage & edge case optimization for an end-to-end GPU submission on the big data one (SQL, ...), and even there the TPC folks made them resubmit if I remember right. Also, note how the parent's response did not actually answer 'audited how'. Pushing the work to the questioner is on the shortlist of techniques studied by misinformation researchers. I'm a fan of both companies, so disappointing to see from a company rep.
- rxin 5y ago
  Check my reply, Leo.
  - lmeyerov 5y ago
    The audit question is on Databricks marketing unaudited Snowflake TPC numbers. I do think Snowflake is big enough to run TPC, but how you guys choose to market is on you. But: I think it's cool both companies got it to $200-300. Way better than years ago. Next stop: GPUs :)
    - rxin 5y ago
      Ah ok. Wasn't clear. I think some repro scripts will be available soon.
  - deleted 5y ago
    [deleted]
redis_mlc 5y ago
> TPC is the official way to audit benchmarks in the database industry. TPC is a benchmark suite for a certain problem class. It says nothing about how the databases are configured or managed. In case you're thick, the above is a polite way of calling you a liar. This is why Oracle and other database vendors don't allow publishing of benchmarks. It's to protect them from incompetent or lazy authors primarily. Source: DBA.

by maslam 5y ago

Hey jiggawatts - TPC is the official way to audit benchmarks in the database industry. They’ve been around for a bit, but let me know if you want more info, I’m happy to share more about them.

lmeyerov 5y ago

It sounds fundamentally busted if a competitor can submit benchmarks for someone else. TPC is great in general, but I didn't realize it had such a gaping flaw. TPC submissions take real time/$/energy/expertise, so I don't know anyone who has ever done it casually. Ex: It was a multi-company effort for the RAPIDS community to get enough API coverage & edge case optimization for an end-to-end GPU submission on the big data one (SQL, ...), and even there the TPC folks made them resubmit if I remember right. Also, note how the parent's response did not actually answer 'audited how'. Pushing the work to the questioner is on the shortlist of techniques studied by misinformation researchers. I'm a fan of both companies, so disappointing to see from a company rep.

rxin 5y ago
Check my reply, Leo.
- lmeyerov 5y ago
  The audit question is on Databricks marketing unaudited Snowflake TPC numbers. I do think Snowflake is big enough to run TPC, but how you guys choose to market is on you. But: I think it's cool both companies got it to $200-300. Way better than years ago. Next stop: GPUs :)
  - rxin 5y ago
    Ah ok. Wasn't clear. I think some repro scripts will be available soon.
- deleted 5y ago
  [deleted]

redis_mlc 5y ago

> TPC is the official way to audit benchmarks in the database industry. TPC is a benchmark suite for a certain problem class. It says nothing about how the databases are configured or managed. In case you're thick, the above is a polite way of calling you a liar. This is why Oracle and other database vendors don't allow publishing of benchmarks. It's to protect them from incompetent or lazy authors primarily. Source: DBA.