4 ms·

pgbench is not a microbenchmark.

brendangregg 3y ago
From the docs: "pgbench is a simple program for running benchmark tests on PostgreSQL. It runs the same sequence of SQL commands over and over" While it might call itself a benchmark, it behaves very microbenchmark-y. The other numbers I and others have shared have been from actual production workloads. Not a simple program that tests same sequence of commands over and over.
- menaerus 3y ago
  While pgbench might be "simple" program, as in a test-runner, workloads that are run by it are far from it. It runs TPC-B by default but can also run your own arbitrary script that defines whatever the workload is? It also allows to run queries concurrently so I fail to understand the reasoning of it "being simple" or "microbenchmarkey". It's far from the truth I think.
- weebull 3y ago
  Anything running a full database server is not micro.
  - brendangregg 3y ago
    If I call the same "get statistics" command over and over in a loop (with zero queries), or 100% the same invalid query (to test the error path performance), I believe we'd call that a micro-benchmark, despite involving a full database. It's a completely unrealistic artificial workload to test a particular type of operation. The pgbench docs make it sound microbenchmark-y by describing making the same call over and over. If people find that this simulates actual production workloads, then yes, it can be considered a macro-benchmark.
    - menaerus 3y ago
      "get statistics" is not what TPC-B does. Nor the invalid queries nor ... From https://www.tpc.org/tpcb/ https://www.tpc.org/tpcb/, a TPC-B workload that pgbench runs by default: In August 1990, the TPC approved its second benchmark, TPC-B. In contrast to TPC-A, TPC-B is not an OLTP benchmark. Rather, TPC-B can be looked at as a database stress test, characterized by: Significant disk input/output Moderate system and application execution time Transaction integrity TPC-B measures throughput in terms of how many transactions per second a system can perform. Because there are substantial differences between the two benchmarks (OLTP vs. database stress test), TPC-B results cannot be compared to TPC-A. ... Transactions are submitted by programs all executing concurrently.
      - brendangregg 3y ago
        I think you missed the context of what I was responding to, which was about whether databases could even have micro-benchmarks. You also missed the word "Obsolete" splattered all over the website you sent me, and the text that TPC-B was "Obsolete as of 6/6/95".
        menaerus 3y ago
        I don't think I have. I was only responding to the factually incorrect statement of yours that pgbench is a microbenchmark. > which was about whether databases could even have micro-benchmarks. No, this was an argument of yours that you pulled out out of nowhere. The topic very specifically was about the pgbench and not whether or not databases can have micro-benchmarks. Obvious answer is, yes, they can as any other software out there. I think that you kinda tried to imply that pgbench was one of such micro-benchmarks in disguise and which is why I c/p the description which proves that it is not. > You also missed the word "Obsolete" I did not since that was not the topic being discussed at all. And in a technical sense, it doesn't matter at all. pgbench still runs so it is very much "not obsolete".
        brendangregg 3y ago
        I didn't pull this argument out of nowhere, please read the direct comment I was replying to. Your position is also completely untenable: this benchmark was obsoleted by its creators 29 years ago, who very clearly say it is obsolete, and you're arguing that it isn't because it "still runs." I'm guessing that this discussion would be more productive if you would please say who you are and the company you work for. I'm Brendan Gregg, I work for Intel, and I'm well known in the performance space. Who are you?
- anarazel 3y ago
  The are loads of real world workloads that have similar patterns to pgbench, particularly read only pgbench.

From the docs: "pgbench is a simple program for running benchmark tests on PostgreSQL. It runs the same sequence of SQL commands over and over" While it might call itself a benchmark, it behaves very microbenchmark-y. The other numbers I and others have shared have been from actual production workloads. Not a simple program that tests same sequence of commands over and over.

menaerus 3y ago
While pgbench might be "simple" program, as in a test-runner, workloads that are run by it are far from it. It runs TPC-B by default but can also run your own arbitrary script that defines whatever the workload is? It also allows to run queries concurrently so I fail to understand the reasoning of it "being simple" or "microbenchmarkey". It's far from the truth I think.
weebull 3y ago
Anything running a full database server is not micro.
- brendangregg 3y ago
  If I call the same "get statistics" command over and over in a loop (with zero queries), or 100% the same invalid query (to test the error path performance), I believe we'd call that a micro-benchmark, despite involving a full database. It's a completely unrealistic artificial workload to test a particular type of operation. The pgbench docs make it sound microbenchmark-y by describing making the same call over and over. If people find that this simulates actual production workloads, then yes, it can be considered a macro-benchmark.
  - menaerus 3y ago
    "get statistics" is not what TPC-B does. Nor the invalid queries nor ... From https://www.tpc.org/tpcb/ https://www.tpc.org/tpcb/, a TPC-B workload that pgbench runs by default: In August 1990, the TPC approved its second benchmark, TPC-B. In contrast to TPC-A, TPC-B is not an OLTP benchmark. Rather, TPC-B can be looked at as a database stress test, characterized by: Significant disk input/output Moderate system and application execution time Transaction integrity TPC-B measures throughput in terms of how many transactions per second a system can perform. Because there are substantial differences between the two benchmarks (OLTP vs. database stress test), TPC-B results cannot be compared to TPC-A. ... Transactions are submitted by programs all executing concurrently.
    - brendangregg 3y ago
      I think you missed the context of what I was responding to, which was about whether databases could even have micro-benchmarks. You also missed the word "Obsolete" splattered all over the website you sent me, and the text that TPC-B was "Obsolete as of 6/6/95".
      - menaerus 3y ago
        I don't think I have. I was only responding to the factually incorrect statement of yours that pgbench is a microbenchmark. > which was about whether databases could even have micro-benchmarks. No, this was an argument of yours that you pulled out out of nowhere. The topic very specifically was about the pgbench and not whether or not databases can have micro-benchmarks. Obvious answer is, yes, they can as any other software out there. I think that you kinda tried to imply that pgbench was one of such micro-benchmarks in disguise and which is why I c/p the description which proves that it is not. > You also missed the word "Obsolete" I did not since that was not the topic being discussed at all. And in a technical sense, it doesn't matter at all. pgbench still runs so it is very much "not obsolete".
        brendangregg 3y ago
        I didn't pull this argument out of nowhere, please read the direct comment I was replying to. Your position is also completely untenable: this benchmark was obsoleted by its creators 29 years ago, who very clearly say it is obsolete, and you're arguing that it isn't because it "still runs." I'm guessing that this discussion would be more productive if you would please say who you are and the company you work for. I'm Brendan Gregg, I work for Intel, and I'm well known in the performance space. Who are you?
anarazel 3y ago
The are loads of real world workloads that have similar patterns to pgbench, particularly read only pgbench.