6 ms·

I'm glad he mentioned Fedora because it's been a tiresome battle to keep frame pointers enabled in the whole distribution (eg https://pagure.io/fesco/issue/3084

by rwmj 3y ago

I'm glad he mentioned Fedora because it's been a tiresome battle to keep frame pointers enabled in the whole distribution (eg https://pagure.io/fesco/issue/3084 https://pagure.io/fesco/issue/3084). There's a persistent myth that frame pointers have a huge overhead, because there was a single Python case that had a +10% slow down (now fixed). The actual measured overhead is under 1%, which is far outweighed by the benefits we've been able to make in certain applications.

sidewndr46 3y ago
I'm still baffled by this attitude. That "under 1%" overhead is why computers are measurably slower than 30 years ago to use. All those "under 1%" overhead add up
brendangregg 3y ago
Thanks; what was the Python fix?
- rwmj 3y ago
  This was the investigation: https://discuss.python.org/t/python-3-11-performance-with-frame-pointers/22507 https://discuss.python.org/t/python-3-11-performance-with-fr... Initially we just turned off frame pointers for the Python 3.9 interpreter in Fedora. They are back on in Python 3.12 where it seems the upstream bug has been fixed, although I can't find the actual fix right now. Fedora tracking bug: https://bugzilla.redhat.com/2158729 https://bugzilla.redhat.com/2158729 Fedora change in Python 3.9 to disable frame pointers: https://src.fedoraproject.org/rpms/python3.9/c/9b71f8369141c1840c8d87db1655fcefc576ddff?branch=rawhide https://src.fedoraproject.org/rpms/python3.9/c/9b71f8369141c...
  - brendangregg 3y ago
    Ah right, thanks, I remember I saw Andrii's analysis in the other thread. https://pagure.io/fesco/issue/2817#comment-826636 https://pagure.io/fesco/issue/2817#comment-826636
menaerus 3y ago
I believe it's a misrepresentation to say that "actual measured overhead is under 1%". I don't think such a claim can be universally applied because this depends on the very workload you're measuring the overhead with. FWIW your results don't quite match the measurements from Linux kernel folks who claim that the overhead is anywhere between 5-10%. Source: https://lore.kernel.org/lkml/20170602104048.jkkzssljsompjdwy@suse.de/T/#u https://lore.kernel.org/lkml/20170602104048.jkkzssljsompjdwy... I didn't preserve the data involved but in a variety of workloads including netperf, page allocator microbenchmark, pgbench and sqlite, enabling framepointer introduced overhead of around the 5-10% mark. Significance in their results IMO is in the fact that they measured the impact by using PostgreSQL and SQLite. If anything, DBMS are one of the best ways to really stress out the system.
- brendangregg 3y ago
  Those are microbenchmarks.
  - menaerus 3y ago
    pgbench is not a microbenchmark.
    - brendangregg 3y ago
      From the docs: "pgbench is a simple program for running benchmark tests on PostgreSQL. It runs the same sequence of SQL commands over and over" While it might call itself a benchmark, it behaves very microbenchmark-y. The other numbers I and others have shared have been from actual production workloads. Not a simple program that tests same sequence of commands over and over.
      - menaerus 3y ago
        While pgbench might be "simple" program, as in a test-runner, workloads that are run by it are far from it. It runs TPC-B by default but can also run your own arbitrary script that defines whatever the workload is? It also allows to run queries concurrently so I fail to understand the reasoning of it "being simple" or "microbenchmarkey". It's far from the truth I think.
edwintorok 3y ago
You probably already know, but with OCaml 5 the only way to get flamegraphs working is to either: * use framepointers [1] * use LBR (but LBR has a limited depth, and may not work on on all CPUs, I'm assuming due to bugs in perf) * implement some deep changes in how perf works to handle the 2 stacks in OCaml (I don't even know if this would be possible), or write/adapt some eBPF code to do it OCaml 5 has a separate stack for OCaml code and C code, and although GDB can link them based on DWARF info, perf DWARF call-graphs cannot (https://github.com/ocaml/ocaml/issues/12563#issuecomment-1932999879 https://github.com/ocaml/ocaml/issues/12563#issuecomment-193...) If you need more evidence to keep it enabled in future releases, you can use OCaml 5 as an example (unfortunately there aren't many OCaml applications, so that may not carry too much weight on its own). [1]: I haven't actually realised that Fedora39 has already enabled FP by default, nice! (I still do most of my day-to-day profiling on an ~CentOS 7 system with 'perf record --call-graph dwarf -F 47 -a', I was aware that there was a discussion to enable FP by default, but haven't noticed it has actually been done already)
- namibj 3y ago
  No, LBR is an Intel-only feature.
  - edwintorok 3y ago
    https://www.phoronix.com/news/AMD-Zen-4-LbrExtV2 https://www.phoronix.com/news/AMD-Zen-4-LbrExtV2 LBR is supposed to work on AMD too, except it doesn't. I'll have to open a bug report (it records the data, it just can't parse it afterwards)
    - ColonelPhantom 3y ago
      Also, if you want LBR on Zen3 or earlier, you're SOL. I noticed that one the hard way. Stuff like this is making me lean more towards getting an Intel for my next computer, at least on desktop where their worse power efficiency is less of an issue. But then they keep gating AVX-512 due to their E-cores not supporting it... You really can't win these days.
awaythrow999 3y ago
Frame pointers are still a no-go on 32bit so anything that is IoT today. The reason we removed them was not a myth but comes from the pre-64 bit days. Not that long ago actually. Even today if you want to repurpose older 64 bit systems with a new life then this of optimization still makes sense. Ideally it should be the default also for security critical systems because not everything needs to be optimized for "observability"
- Narishma 3y ago
  > Frame pointers are still a no-go on 32bit so anything that is IoT today. Isn't that just 32-bit x86, which isn't used in IoT? The other 32-bit ISAs aren't register-starved like x86.
  - weebull 3y ago
    It would be, yes. x86 had very few registers, so anything you could do to free them up was vital. Arm 32bit has 32 general purpose registers I think, and RISC V certainly does. In fact there's no difference between 32 and 64 bit in that respect. If anything, 64-bit frame pointers make it marginally worse.
    - CountSessine 3y ago
      Sadly, no. 32-bit ARM only has 16 GPR’s (two of which are zero and link), mostly because of the stupid predication bits in the instruction encoding. That said, I don’t know how valuable getting rid of FP on ARM is - I once benchmarked ffmpeg on 32-bit x86 before and after enabling FP and PIC (basically removing 2 GPRs) and the difference was huge (>10%) but that’s an extreme example.
      - fanf2 3y ago
        Arm32 doesn’t have a zero-value register. Its non-general-purpose registers are PC, LR, SP, FP – tho the link register can be used for temporary values.

by rwmj 3y ago

sidewndr46 3y ago

I'm still baffled by this attitude. That "under 1%" overhead is why computers are measurably slower than 30 years ago to use. All those "under 1%" overhead add up

brendangregg 3y ago

Thanks; what was the Python fix?

rwmj 3y ago
This was the investigation: https://discuss.python.org/t/python-3-11-performance-with-frame-pointers/22507 https://discuss.python.org/t/python-3-11-performance-with-fr... Initially we just turned off frame pointers for the Python 3.9 interpreter in Fedora. They are back on in Python 3.12 where it seems the upstream bug has been fixed, although I can't find the actual fix right now. Fedora tracking bug: https://bugzilla.redhat.com/2158729 https://bugzilla.redhat.com/2158729 Fedora change in Python 3.9 to disable frame pointers: https://src.fedoraproject.org/rpms/python3.9/c/9b71f8369141c1840c8d87db1655fcefc576ddff?branch=rawhide https://src.fedoraproject.org/rpms/python3.9/c/9b71f8369141c...
- brendangregg 3y ago
  Ah right, thanks, I remember I saw Andrii's analysis in the other thread. https://pagure.io/fesco/issue/2817#comment-826636 https://pagure.io/fesco/issue/2817#comment-826636

menaerus 3y ago

I believe it's a misrepresentation to say that "actual measured overhead is under 1%". I don't think such a claim can be universally applied because this depends on the very workload you're measuring the overhead with. FWIW your results don't quite match the measurements from Linux kernel folks who claim that the overhead is anywhere between 5-10%. Source: https://lore.kernel.org/lkml/20170602104048.jkkzssljsompjdwy@suse.de/T/#u https://lore.kernel.org/lkml/20170602104048.jkkzssljsompjdwy... I didn't preserve the data involved but in a variety of workloads including netperf, page allocator microbenchmark, pgbench and sqlite, enabling framepointer introduced overhead of around the 5-10% mark. Significance in their results IMO is in the fact that they measured the impact by using PostgreSQL and SQLite. If anything, DBMS are one of the best ways to really stress out the system.

brendangregg 3y ago
Those are microbenchmarks.
- menaerus 3y ago
  pgbench is not a microbenchmark.
  - brendangregg 3y ago
    From the docs: "pgbench is a simple program for running benchmark tests on PostgreSQL. It runs the same sequence of SQL commands over and over" While it might call itself a benchmark, it behaves very microbenchmark-y. The other numbers I and others have shared have been from actual production workloads. Not a simple program that tests same sequence of commands over and over.
    - menaerus 3y ago
      While pgbench might be "simple" program, as in a test-runner, workloads that are run by it are far from it. It runs TPC-B by default but can also run your own arbitrary script that defines whatever the workload is? It also allows to run queries concurrently so I fail to understand the reasoning of it "being simple" or "microbenchmarkey". It's far from the truth I think.

edwintorok 3y ago

You probably already know, but with OCaml 5 the only way to get flamegraphs working is to either: * use framepointers [1] * use LBR (but LBR has a limited depth, and may not work on on all CPUs, I'm assuming due to bugs in perf) * implement some deep changes in how perf works to handle the 2 stacks in OCaml (I don't even know if this would be possible), or write/adapt some eBPF code to do it OCaml 5 has a separate stack for OCaml code and C code, and although GDB can link them based on DWARF info, perf DWARF call-graphs cannot (https://github.com/ocaml/ocaml/issues/12563#issuecomment-1932999879 https://github.com/ocaml/ocaml/issues/12563#issuecomment-193...) If you need more evidence to keep it enabled in future releases, you can use OCaml 5 as an example (unfortunately there aren't many OCaml applications, so that may not carry too much weight on its own). [1]: I haven't actually realised that Fedora39 has already enabled FP by default, nice! (I still do most of my day-to-day profiling on an ~CentOS 7 system with 'perf record --call-graph dwarf -F 47 -a', I was aware that there was a discussion to enable FP by default, but haven't noticed it has actually been done already)

namibj 3y ago
No, LBR is an Intel-only feature.
- edwintorok 3y ago
  https://www.phoronix.com/news/AMD-Zen-4-LbrExtV2 https://www.phoronix.com/news/AMD-Zen-4-LbrExtV2 LBR is supposed to work on AMD too, except it doesn't. I'll have to open a bug report (it records the data, it just can't parse it afterwards)
  - ColonelPhantom 3y ago
    Also, if you want LBR on Zen3 or earlier, you're SOL. I noticed that one the hard way. Stuff like this is making me lean more towards getting an Intel for my next computer, at least on desktop where their worse power efficiency is less of an issue. But then they keep gating AVX-512 due to their E-cores not supporting it... You really can't win these days.

awaythrow999 3y ago

Frame pointers are still a no-go on 32bit so anything that is IoT today. The reason we removed them was not a myth but comes from the pre-64 bit days. Not that long ago actually. Even today if you want to repurpose older 64 bit systems with a new life then this of optimization still makes sense. Ideally it should be the default also for security critical systems because not everything needs to be optimized for "observability"

Narishma 3y ago
> Frame pointers are still a no-go on 32bit so anything that is IoT today. Isn't that just 32-bit x86, which isn't used in IoT? The other 32-bit ISAs aren't register-starved like x86.
- weebull 3y ago
  It would be, yes. x86 had very few registers, so anything you could do to free them up was vital. Arm 32bit has 32 general purpose registers I think, and RISC V certainly does. In fact there's no difference between 32 and 64 bit in that respect. If anything, 64-bit frame pointers make it marginally worse.
  - CountSessine 3y ago
    Sadly, no. 32-bit ARM only has 16 GPR’s (two of which are zero and link), mostly because of the stupid predication bits in the instruction encoding. That said, I don’t know how valuable getting rid of FP on ARM is - I once benchmarked ffmpeg on 32-bit x86 before and after enabling FP and PIC (basically removing 2 GPRs) and the difference was huge (>10%) but that’s an extreme example.
    - fanf2 3y ago
      Arm32 doesn’t have a zero-value register. Its non-general-purpose registers are PC, LR, SP, FP – tho the link register can be used for temporary values.