3 ms·

Those are microbenchmarks.

by brendangregg 3y ago

menaerus 3y ago
pgbench is not a microbenchmark.
- brendangregg 3y ago
  From the docs: "pgbench is a simple program for running benchmark tests on PostgreSQL. It runs the same sequence of SQL commands over and over" While it might call itself a benchmark, it behaves very microbenchmark-y. The other numbers I and others have shared have been from actual production workloads. Not a simple program that tests same sequence of commands over and over.
  - menaerus 3y ago
    While pgbench might be "simple" program, as in a test-runner, workloads that are run by it are far from it. It runs TPC-B by default but can also run your own arbitrary script that defines whatever the workload is? It also allows to run queries concurrently so I fail to understand the reasoning of it "being simple" or "microbenchmarkey". It's far from the truth I think.
  - weebull 3y ago
    Anything running a full database server is not micro.
    - brendangregg 3y ago
      If I call the same "get statistics" command over and over in a loop (with zero queries), or 100% the same invalid query (to test the error path performance), I believe we'd call that a micro-benchmark, despite involving a full database. It's a completely unrealistic artificial workload to test a particular type of operation. The pgbench docs make it sound microbenchmark-y by describing making the same call over and over. If people find that this simulates actual production workloads, then yes, it can be considered a macro-benchmark.
      - menaerus 3y ago
        "get statistics" is not what TPC-B does. Nor the invalid queries nor ... From https://www.tpc.org/tpcb/ https://www.tpc.org/tpcb/, a TPC-B workload that pgbench runs by default: In August 1990, the TPC approved its second benchmark, TPC-B. In contrast to TPC-A, TPC-B is not an OLTP benchmark. Rather, TPC-B can be looked at as a database stress test, characterized by: Significant disk input/output Moderate system and application execution time Transaction integrity TPC-B measures throughput in terms of how many transactions per second a system can perform. Because there are substantial differences between the two benchmarks (OLTP vs. database stress test), TPC-B results cannot be compared to TPC-A. ... Transactions are submitted by programs all executing concurrently.
        brendangregg 3y ago
        I think you missed the context of what I was responding to, which was about whether databases could even have micro-benchmarks. You also missed the word "Obsolete" splattered all over the website you sent me, and the text that TPC-B was "Obsolete as of 6/6/95".
        menaerus 3y ago
        I don't think I have. I was only responding to the factually incorrect statement of yours that pgbench is a microbenchmark. > which was about whether databases could even have micro-benchmarks. No, this was an argument of yours that you pulled out out of nowhere. The topic very specifically was about the pgbench and not whether or not databases can have micro-benchmarks. Obvious answer is, yes, they can as any other software out there. I think that you kinda tried to imply that pgbench was one of such micro-benchmarks in disguise and which is why I c/p the description which proves that it is not. > You also missed the word "Obsolete" I did not since that was not the topic being discussed at all. And in a technical sense, it doesn't matter at all. pgbench still runs so it is very much "not obsolete".
        brendangregg 3y ago
  - anarazel 3y ago
    The are loads of real world workloads that have similar patterns to pgbench, particularly read only pgbench.

Those are microbenchmarks.

menaerus 3y ago

pgbench is not a microbenchmark.

brendangregg 3y ago
From the docs: "pgbench is a simple program for running benchmark tests on PostgreSQL. It runs the same sequence of SQL commands over and over" While it might call itself a benchmark, it behaves very microbenchmark-y. The other numbers I and others have shared have been from actual production workloads. Not a simple program that tests same sequence of commands over and over.
- menaerus 3y ago
  While pgbench might be "simple" program, as in a test-runner, workloads that are run by it are far from it. It runs TPC-B by default but can also run your own arbitrary script that defines whatever the workload is? It also allows to run queries concurrently so I fail to understand the reasoning of it "being simple" or "microbenchmarkey". It's far from the truth I think.
- weebull 3y ago
  Anything running a full database server is not micro.
  - brendangregg 3y ago
    If I call the same "get statistics" command over and over in a loop (with zero queries), or 100% the same invalid query (to test the error path performance), I believe we'd call that a micro-benchmark, despite involving a full database. It's a completely unrealistic artificial workload to test a particular type of operation. The pgbench docs make it sound microbenchmark-y by describing making the same call over and over. If people find that this simulates actual production workloads, then yes, it can be considered a macro-benchmark.
    - menaerus 3y ago
      "get statistics" is not what TPC-B does. Nor the invalid queries nor ... From https://www.tpc.org/tpcb/ https://www.tpc.org/tpcb/, a TPC-B workload that pgbench runs by default: In August 1990, the TPC approved its second benchmark, TPC-B. In contrast to TPC-A, TPC-B is not an OLTP benchmark. Rather, TPC-B can be looked at as a database stress test, characterized by: Significant disk input/output Moderate system and application execution time Transaction integrity TPC-B measures throughput in terms of how many transactions per second a system can perform. Because there are substantial differences between the two benchmarks (OLTP vs. database stress test), TPC-B results cannot be compared to TPC-A. ... Transactions are submitted by programs all executing concurrently.
      - brendangregg 3y ago
        I think you missed the context of what I was responding to, which was about whether databases could even have micro-benchmarks. You also missed the word "Obsolete" splattered all over the website you sent me, and the text that TPC-B was "Obsolete as of 6/6/95".
        menaerus 3y ago
        I don't think I have. I was only responding to the factually incorrect statement of yours that pgbench is a microbenchmark. > which was about whether databases could even have micro-benchmarks. No, this was an argument of yours that you pulled out out of nowhere. The topic very specifically was about the pgbench and not whether or not databases can have micro-benchmarks. Obvious answer is, yes, they can as any other software out there. I think that you kinda tried to imply that pgbench was one of such micro-benchmarks in disguise and which is why I c/p the description which proves that it is not. > You also missed the word "Obsolete" I did not since that was not the topic being discussed at all. And in a technical sense, it doesn't matter at all. pgbench still runs so it is very much "not obsolete".
        brendangregg 3y ago
- anarazel 3y ago
  The are loads of real world workloads that have similar patterns to pgbench, particularly read only pgbench.