7 ms·

I believe it's a misrepresentation to say that "actual measured overhead is under 1%". I don't think such a claim can be universally applied because this depend

I believe it's a misrepresentation to say that "actual measured overhead is under 1%". I don't think such a claim can be universally applied because this depends on the very workload you're measuring the overhead with. FWIW your results don't quite match the measurements from Linux kernel folks who claim that the overhead is anywhere between 5-10%. Source: https://lore.kernel.org/lkml/20170602104048.jkkzssljsompjdwy@suse.de/T/#u https://lore.kernel.org/lkml/20170602104048.jkkzssljsompjdwy... I didn't preserve the data involved but in a variety of workloads including netperf, page allocator microbenchmark, pgbench and sqlite, enabling framepointer introduced overhead of around the 5-10% mark. Significance in their results IMO is in the fact that they measured the impact by using PostgreSQL and SQLite. If anything, DBMS are one of the best ways to really stress out the system.

brendangregg 3y ago
Those are microbenchmarks.
- menaerus 3y ago
  pgbench is not a microbenchmark.
  - brendangregg 3y ago
    From the docs: "pgbench is a simple program for running benchmark tests on PostgreSQL. It runs the same sequence of SQL commands over and over" While it might call itself a benchmark, it behaves very microbenchmark-y. The other numbers I and others have shared have been from actual production workloads. Not a simple program that tests same sequence of commands over and over.
    - menaerus 3y ago
      While pgbench might be "simple" program, as in a test-runner, workloads that are run by it are far from it. It runs TPC-B by default but can also run your own arbitrary script that defines whatever the workload is? It also allows to run queries concurrently so I fail to understand the reasoning of it "being simple" or "microbenchmarkey". It's far from the truth I think.
    - weebull 3y ago
      Anything running a full database server is not micro.
      - brendangregg 3y ago
        If I call the same "get statistics" command over and over in a loop (with zero queries), or 100% the same invalid query (to test the error path performance), I believe we'd call that a micro-benchmark, despite involving a full database. It's a completely unrealistic artificial workload to test a particular type of operation. The pgbench docs make it sound microbenchmark-y by describing making the same call over and over. If people find that this simulates actual production workloads, then yes, it can be considered a macro-benchmark.
        menaerus 3y ago
        "get statistics" is not what TPC-B does. Nor the invalid queries nor ... From https://www.tpc.org/tpcb/ https://www.tpc.org/tpcb/, a TPC-B workload that pgbench runs by default: In August 1990, the TPC approved its second benchmark, TPC-B. In contrast to TPC-A, TPC-B is not an OLTP benchmark. Rather, TPC-B can be looked at as a database stress test, characterized by: Significant disk input/output Moderate system and application execution time Transaction integrity TPC-B measures throughput in terms of how many transactions per second a system can perform. Because there are substantial differences between the two benchmarks (OLTP vs. database stress test), TPC-B results cannot be compared to TPC-A. ... Transactions are submitted by programs all executing concurrently.
        brendangregg 3y ago
        I think you missed the context of what I was responding to, which was about whether databases could even have micro-benchmarks. You also missed the word "Obsolete" splattered all over the website you sent me, and the text that TPC-B was "Obsolete as of 6/6/95".
        menaerus 3y ago
    - anarazel 3y ago
      The are loads of real world workloads that have similar patterns to pgbench, particularly read only pgbench.
babel_ 3y ago
Those are numbers from 7 years ago, so they're beginning to get a bit stale as people start to put more weight behind having frame pointers and make upstream contributions to their compilers to improve their output. People put it at <1% from much more recent testing by the very R.W.M. Jones you're replying to [0] and separate testing by others like Brendan Gregg [1b], whose post this is commenting on (and included [1b] in the Appendix as well), with similar accounts by others in the last couple years. Oh, and if you use flamegraph, you might want to check the repo for a familiar name. Some programs, like Python, have reported worse, 2-7% [2], but there is traction on tackling that [1a] (see both rwmj's and brendangregg's replies to sibling comments, they've both done a lot of upstreamed work wrt. frame pointers, performance, and profiling). As has been frequently pointed out, the benefits from improved profiling cannot be understated, even a 10% cost to having frame pointers can be well worth it when you leverage that information to target the actual bottlenecks that are eating up your cycles. Plus, you can always disable it in specific hotspots later when needed, which is much easier than the reverse. Something, something, premature optimisation -- though in seriousness, this information benefits actual optimisation, exactly because we don't have the information and understanding that would allow truly universal claims, precisely because things like this haven't been available, and so haven't been widely used. We know frame pointers, from additional register pressure and extended function prologue/epilogue, can be a detriment in certain hotspots; that's why we have granular control. But without them, we often don't know which hotspots are actually affected, so I'm sure even the databases would benefit... though the "my database is the fastest database" problem has always been the result of endless micro-benchmarking, rather than actual end-to-end program performance and latency, so even a claimed "10%" drop there probably doesn't impact actual real-world usage, but that's a reason why some of the most interesting profiling work lately has been from ideas like causal profilers and continuous profilers, which answer exactly that. [0]: https://rwmj.wordpress.com/2023/02/14/frame-pointers-vs-dwarf-my-verdict/ https://rwmj.wordpress.com/2023/02/14/frame-pointers-vs-dwar... [1a]: https://pagure.io/fesco/issue/2817#comment-826636 https://pagure.io/fesco/issue/2817#comment-826636 [1b]: https://pagure.io/fesco/issue/2817#comment-826805 https://pagure.io/fesco/issue/2817#comment-826805 [2]: https://discuss.python.org/t/the-performance-of-python-with-perf-support-is-not-great-and-is-going-to-get-a-lot-worse/25280/1 https://discuss.python.org/t/the-performance-of-python-with-...
barrkel 3y ago
This isn't an argument for a default.
- menaerus 3y ago
  I was not even trying to make one. I was questioning the validity of "1% overhead" claim by providing the counter-example from respectable source.
deleted 3y ago
[deleted]

brendangregg 3y ago

Those are microbenchmarks.

menaerus 3y ago
pgbench is not a microbenchmark.
- brendangregg 3y ago
  From the docs: "pgbench is a simple program for running benchmark tests on PostgreSQL. It runs the same sequence of SQL commands over and over" While it might call itself a benchmark, it behaves very microbenchmark-y. The other numbers I and others have shared have been from actual production workloads. Not a simple program that tests same sequence of commands over and over.
  - menaerus 3y ago
    While pgbench might be "simple" program, as in a test-runner, workloads that are run by it are far from it. It runs TPC-B by default but can also run your own arbitrary script that defines whatever the workload is? It also allows to run queries concurrently so I fail to understand the reasoning of it "being simple" or "microbenchmarkey". It's far from the truth I think.
  - weebull 3y ago
    Anything running a full database server is not micro.
    - brendangregg 3y ago
      If I call the same "get statistics" command over and over in a loop (with zero queries), or 100% the same invalid query (to test the error path performance), I believe we'd call that a micro-benchmark, despite involving a full database. It's a completely unrealistic artificial workload to test a particular type of operation. The pgbench docs make it sound microbenchmark-y by describing making the same call over and over. If people find that this simulates actual production workloads, then yes, it can be considered a macro-benchmark.
      - menaerus 3y ago
        "get statistics" is not what TPC-B does. Nor the invalid queries nor ... From https://www.tpc.org/tpcb/ https://www.tpc.org/tpcb/, a TPC-B workload that pgbench runs by default: In August 1990, the TPC approved its second benchmark, TPC-B. In contrast to TPC-A, TPC-B is not an OLTP benchmark. Rather, TPC-B can be looked at as a database stress test, characterized by: Significant disk input/output Moderate system and application execution time Transaction integrity TPC-B measures throughput in terms of how many transactions per second a system can perform. Because there are substantial differences between the two benchmarks (OLTP vs. database stress test), TPC-B results cannot be compared to TPC-A. ... Transactions are submitted by programs all executing concurrently.
        brendangregg 3y ago
        I think you missed the context of what I was responding to, which was about whether databases could even have micro-benchmarks. You also missed the word "Obsolete" splattered all over the website you sent me, and the text that TPC-B was "Obsolete as of 6/6/95".
        menaerus 3y ago
  - anarazel 3y ago
    The are loads of real world workloads that have similar patterns to pgbench, particularly read only pgbench.

babel_ 3y ago

Those are numbers from 7 years ago, so they're beginning to get a bit stale as people start to put more weight behind having frame pointers and make upstream contributions to their compilers to improve their output. People put it at <1% from much more recent testing by the very R.W.M. Jones you're replying to [0] and separate testing by others like Brendan Gregg [1b], whose post this is commenting on (and included [1b] in the Appendix as well), with similar accounts by others in the last couple years. Oh, and if you use flamegraph, you might want to check the repo for a familiar name. Some programs, like Python, have reported worse, 2-7% [2], but there is traction on tackling that [1a] (see both rwmj's and brendangregg's replies to sibling comments, they've both done a lot of upstreamed work wrt. frame pointers, performance, and profiling). As has been frequently pointed out, the benefits from improved profiling cannot be understated, even a 10% cost to having frame pointers can be well worth it when you leverage that information to target the actual bottlenecks that are eating up your cycles. Plus, you can always disable it in specific hotspots later when needed, which is much easier than the reverse. Something, something, premature optimisation -- though in seriousness, this information benefits actual optimisation, exactly because we don't have the information and understanding that would allow truly universal claims, precisely because things like this haven't been available, and so haven't been widely used. We know frame pointers, from additional register pressure and extended function prologue/epilogue, can be a detriment in certain hotspots; that's why we have granular control. But without them, we often don't know which hotspots are actually affected, so I'm sure even the databases would benefit... though the "my database is the fastest database" problem has always been the result of endless micro-benchmarking, rather than actual end-to-end program performance and latency, so even a claimed "10%" drop there probably doesn't impact actual real-world usage, but that's a reason why some of the most interesting profiling work lately has been from ideas like causal profilers and continuous profilers, which answer exactly that. [0]: https://rwmj.wordpress.com/2023/02/14/frame-pointers-vs-dwarf-my-verdict/ https://rwmj.wordpress.com/2023/02/14/frame-pointers-vs-dwar... [1a]: https://pagure.io/fesco/issue/2817#comment-826636 https://pagure.io/fesco/issue/2817#comment-826636 [1b]: https://pagure.io/fesco/issue/2817#comment-826805 https://pagure.io/fesco/issue/2817#comment-826805 [2]: https://discuss.python.org/t/the-performance-of-python-with-perf-support-is-not-great-and-is-going-to-get-a-lot-worse/25280/1 https://discuss.python.org/t/the-performance-of-python-with-...

barrkel 3y ago

This isn't an argument for a default.

menaerus 3y ago
I was not even trying to make one. I was questioning the validity of "1% overhead" claim by providing the counter-example from respectable source.

deleted 3y ago

[deleted]