4 ms·

Anything running a full database server is not micro.

brendangregg 3y ago
If I call the same "get statistics" command over and over in a loop (with zero queries), or 100% the same invalid query (to test the error path performance), I believe we'd call that a micro-benchmark, despite involving a full database. It's a completely unrealistic artificial workload to test a particular type of operation. The pgbench docs make it sound microbenchmark-y by describing making the same call over and over. If people find that this simulates actual production workloads, then yes, it can be considered a macro-benchmark.
- menaerus 3y ago
  "get statistics" is not what TPC-B does. Nor the invalid queries nor ... From https://www.tpc.org/tpcb/ https://www.tpc.org/tpcb/, a TPC-B workload that pgbench runs by default: In August 1990, the TPC approved its second benchmark, TPC-B. In contrast to TPC-A, TPC-B is not an OLTP benchmark. Rather, TPC-B can be looked at as a database stress test, characterized by: Significant disk input/output Moderate system and application execution time Transaction integrity TPC-B measures throughput in terms of how many transactions per second a system can perform. Because there are substantial differences between the two benchmarks (OLTP vs. database stress test), TPC-B results cannot be compared to TPC-A. ... Transactions are submitted by programs all executing concurrently.
  - brendangregg 3y ago
    I think you missed the context of what I was responding to, which was about whether databases could even have micro-benchmarks. You also missed the word "Obsolete" splattered all over the website you sent me, and the text that TPC-B was "Obsolete as of 6/6/95".
    - menaerus 3y ago
      I don't think I have. I was only responding to the factually incorrect statement of yours that pgbench is a microbenchmark. > which was about whether databases could even have micro-benchmarks. No, this was an argument of yours that you pulled out out of nowhere. The topic very specifically was about the pgbench and not whether or not databases can have micro-benchmarks. Obvious answer is, yes, they can as any other software out there. I think that you kinda tried to imply that pgbench was one of such micro-benchmarks in disguise and which is why I c/p the description which proves that it is not. > You also missed the word "Obsolete" I did not since that was not the topic being discussed at all. And in a technical sense, it doesn't matter at all. pgbench still runs so it is very much "not obsolete".
      - brendangregg 3y ago
        I didn't pull this argument out of nowhere, please read the direct comment I was replying to. Your position is also completely untenable: this benchmark was obsoleted by its creators 29 years ago, who very clearly say it is obsolete, and you're arguing that it isn't because it "still runs." I'm guessing that this discussion would be more productive if you would please say who you are and the company you work for. I'm Brendan Gregg, I work for Intel, and I'm well known in the performance space. Who are you?
        menaerus 3y ago
        > I'm guessing that this discussion would be more productive if you would please say who you are and the company you work for. I'm Brendan Gregg, I work for Intel, and I'm well known in the performance space. Who are you? Wow, just wow. How ridiculous this is? > Your position is also completely untenable: this benchmark was obsoleted by its creators 29 years ago, who very clearly say it is obsolete, and you're arguing that it isn't because it "still runs." My only position in the whole thread was "1% of overhead cannot be universally claimed" and I still stand by it 100%. pgbench experiment from Linux kernel folks was just one of the counter-examples that can be found in the wild that goes against your claim. And which you first disputed by saying that it is a micro-benchmark (meaning that you have no idea what it is) and now you're disputing it by saying it's obsolete (yes, but still very relevant in database development, used in the Linux kernel and not the slightest technical reasoning after all). Personally, I couldn't care less about this but if names is what you're after, you're not disagreeing with me but with the methodology and results presented by Mel Gorman, Linux kernel developer.

If I call the same "get statistics" command over and over in a loop (with zero queries), or 100% the same invalid query (to test the error path performance), I believe we'd call that a micro-benchmark, despite involving a full database. It's a completely unrealistic artificial workload to test a particular type of operation. The pgbench docs make it sound microbenchmark-y by describing making the same call over and over. If people find that this simulates actual production workloads, then yes, it can be considered a macro-benchmark.

menaerus 3y ago
"get statistics" is not what TPC-B does. Nor the invalid queries nor ... From https://www.tpc.org/tpcb/ https://www.tpc.org/tpcb/, a TPC-B workload that pgbench runs by default: In August 1990, the TPC approved its second benchmark, TPC-B. In contrast to TPC-A, TPC-B is not an OLTP benchmark. Rather, TPC-B can be looked at as a database stress test, characterized by: Significant disk input/output Moderate system and application execution time Transaction integrity TPC-B measures throughput in terms of how many transactions per second a system can perform. Because there are substantial differences between the two benchmarks (OLTP vs. database stress test), TPC-B results cannot be compared to TPC-A. ... Transactions are submitted by programs all executing concurrently.
- brendangregg 3y ago
  I think you missed the context of what I was responding to, which was about whether databases could even have micro-benchmarks. You also missed the word "Obsolete" splattered all over the website you sent me, and the text that TPC-B was "Obsolete as of 6/6/95".
  - menaerus 3y ago
    I don't think I have. I was only responding to the factually incorrect statement of yours that pgbench is a microbenchmark. > which was about whether databases could even have micro-benchmarks. No, this was an argument of yours that you pulled out out of nowhere. The topic very specifically was about the pgbench and not whether or not databases can have micro-benchmarks. Obvious answer is, yes, they can as any other software out there. I think that you kinda tried to imply that pgbench was one of such micro-benchmarks in disguise and which is why I c/p the description which proves that it is not. > You also missed the word "Obsolete" I did not since that was not the topic being discussed at all. And in a technical sense, it doesn't matter at all. pgbench still runs so it is very much "not obsolete".
    - brendangregg 3y ago
      I didn't pull this argument out of nowhere, please read the direct comment I was replying to. Your position is also completely untenable: this benchmark was obsoleted by its creators 29 years ago, who very clearly say it is obsolete, and you're arguing that it isn't because it "still runs." I'm guessing that this discussion would be more productive if you would please say who you are and the company you work for. I'm Brendan Gregg, I work for Intel, and I'm well known in the performance space. Who are you?
      - menaerus 3y ago
        > I'm guessing that this discussion would be more productive if you would please say who you are and the company you work for. I'm Brendan Gregg, I work for Intel, and I'm well known in the performance space. Who are you? Wow, just wow. How ridiculous this is? > Your position is also completely untenable: this benchmark was obsoleted by its creators 29 years ago, who very clearly say it is obsolete, and you're arguing that it isn't because it "still runs." My only position in the whole thread was "1% of overhead cannot be universally claimed" and I still stand by it 100%. pgbench experiment from Linux kernel folks was just one of the counter-examples that can be found in the wild that goes against your claim. And which you first disputed by saying that it is a micro-benchmark (meaning that you have no idea what it is) and now you're disputing it by saying it's obsolete (yes, but still very relevant in database development, used in the Linux kernel and not the slightest technical reasoning after all). Personally, I couldn't care less about this but if names is what you're after, you're not disagreeing with me but with the methodology and results presented by Mel Gorman, Linux kernel developer.