6 ms·

Author here - any tips on dirty tricks somehow using %fs + offset from a shared library, without calling __tls_get_addr and without recompiling the executable (

by yosefk 2y ago

Author here - any tips on dirty tricks somehow using %fs + offset from a shared library, without calling __tls_get_addr and without recompiling the executable (for instance, from a Python extension module) are most welcome! (I have one such trick - mmaping an address produced by ORing some high bit into %fs, and accessing that - but it's seriously dirty and I doubt it would survive outside a limited set of use cases.) Separately, the post describes one of a few aspects of optimizing funtrace (https://github.com/yosefk/funtrace https://github.com/yosefk/funtrace), which I think is the fastest open-source function tracing profiler for C++ today, and which can be ported relatively easily to Rust/other languages producing native code (the runtime is ~1200 LOC of C++; the trace decoder would need very few changes.) A description of how funtrace works as well as why you'd want a tracing profiler (as opposed to a sampling profiler like perf) and how you'd use it is here: https://yosefk.com/blog/profiling-in-production-with-function-call-traces.html https://yosefk.com/blog/profiling-in-production-with-functio...

atq2119 2y ago
Take a look at https://maskray.me/blog/2021-02-14-all-about-thread-local-storage https://maskray.me/blog/2021-02-14-all-about-thread-local-st... I think what you want is to force the "initial exec" model using an attribute to get the more efficient code. IIRC this (or something equivalent) is what libGL does because basically every OpenGL function call needs to read the thread-local variable holding the current GL context. The downside is that dlopen()ing your library may fail.
- yosefk 2y ago
  Doesn't the "initial exec" model require the user to run the executable with LD_PRELOAD=your.so or linking against your.so (DT_NEEDED)? eg would it work with a Python extension module which is dlopen'd at runtime?
  - fweimer 2y ago
    It depends on the amount of TLS and the number of objects that do this, and the C library. In glibc, we have a small static TLS reservation dedicated for use by dlopen of shared objects with initial-exec.
fweimer 2y ago
With glibc, you can use -ftls-model=initial-exec (or the corresponding variable attribute) to get offset-based TLS. The offset is variable per program (unlike local-exec), but the same for all threads, so it's more efficient. Using too much initial-exec TLS (potential across multiple shared objects) eventually causes dlopen to fail because the TCB cannot be resized. This is not a problem if the shared objects are loaded through dependencies at process start. If initial-exec TLS does not work due to the dlopen issue, on x86-64 and recent-enough distributions, you can use -mtls-dialect=gnu2 to get a faster variant of __tls_get_addr that requires less register spilling. Unfortunately glibc and GCC originally did not agree on the ABI: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113874 https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113874 https://sourceware.org/bugzilla/show_bug.cgi?id=31372 https://sourceware.org/bugzilla/show_bug.cgi?id=31372 This has been fixed for RHEL 10 (which switched to -mtls-dialect=gnu2 for x86-64 for the whole distribution, thereby exposing the ABI bug during development). As the ABI was fixed on the glibc side in dynamically-linked code, the change is backportable, but it's a bit involved because the first XSAVE-using change upstream was buggy, if I recall correctly. But the backport is definitely something you could request from your distribution. Note that there was a previous bug in __tls_get_addr (on all architectures that use it), where the fast path was not always used after dlopen: https://sourceware.org/bugzilla/show_bug.cgi?id=19924 https://sourceware.org/bugzilla/show_bug.cgi?id=19924 This bug introduced way more overhead that just saving registers. I expect that quite a few distributions have backported the fix. This breaks certain interposed mallocs due to a malloc/TLS cyclic dependency, but there is a workaround for that: https://sourceware.org/git/?p=glibc.git;a=commitdiff;h=018f0fc3b818d4d1460a4e2384c24802504b1d20 https://sourceware.org/git/?p=glibc.git;a=commitdiff;h=018f0... The other issue is just that the C++ TLS-with-constructors design isn't that great. You can work around this in the application by using a plain pointer for TLS access, which starts out as NULL and is initialized after a null check. To free the pointer on thread exit, you can use a separate TLS variable or POSIX thread-specific data (pthread_key_create) to register a destructor, and that will only be accessed on initialized and thread exit. This sort of question is probably more suited to libc-help: https://sourceware.org/mailman/listinfo/libc-help/ https://sourceware.org/mailman/listinfo/libc-help/
- dzaima 2y ago
  > Using too much initial-exec TLS (potential across multiple shared objects) eventually causes dlopen to fail because the TCB cannot be resized. This feels like the perfect situation to preallocate a gigabyte or something of virtual memory for extending the TLS, similar to how the stack is. But, testing on my system, looks like the allowed initial-exec TLS size is just ~1700 bytes.
sparkie 2y ago
GCC and Clang both provide `_readfsbase_u64()`, which presumably is just a wrapper around `__asm__("rdfsbase %0" : "=r"(result))`. Also gcc lets you define variables like `int__seg_fs *foo;`if the `__SEG_FS` preprocessor symbol is defined. With Clang, you can use `int __attribute__((address_space(257))) *foo`
- damon_dam 2y ago
  To the author: If you have a specific situation that the codegen is failing to optimize well (e.g. the ctor cases that you ran into) you can store the offset to your variable in a non-TLS global, then manually add the FS/GS to it. Use inline asm if you need to bypass any init checks. An alternative that might be worth looking into is just hashing the FS/GS into a table index. It will be slower than the well-optimized case, but it will let you opt out of the TLS allocation process altogether. This might be a good thing in some cases for a low-level facility like a function tracer.
  - yosefk 2y ago
    I have updated the post with all the suggestions people replied with except this one, because I don't understand it. How do I allocate memory for addressing it with FS/GS? Isn't FS the register pointing to the TLS area - then how is the FS-based access you propose different from how TLS normally works? Isn't GS used "for something" on x86?.. If you could elaborate on this/show some code I would be very grateful! Note that my question is about shared libraries. If the thread_local is linked into an executable, I guess you could indeed save the offset somewhere and then add the value of %fs to it, though if this is a way to work around the constructor issue, I prefer to not have a constructor. The question is if this sort of direction can help for thread-local storage allocated by a shared library.
    - damon_dam 2y ago
      > How do I allocate memory for addressing it with FS/GS? Allocate the variable normally, then compute the offset in one thread (e.g. offset = uintptr_t(&variable) - get_fs()), then access it by adding the offset to FS in any thread (e.g. (vartype *) (offset + get_fs())). The only difference from how it normally works is that you can manually force it to be inlined, sidestepping the codegen problems you described in your post. But if you can avoid those problems by not using constructors instead, that's definitely better. I used "FS/GS" because GS is used instead of FS on some systems for the same purpose. The shared library-specific issues are one of the reasons I was suggesting maybe looking into hashing, e.g. perhaps as a fallback solution when the TLS approach fails.
yxhuvud 2y ago
Could it be productive to report the slowness to the compiler (or ld) makers? Is this inherently bad or could it be fixed by the toolmakers?
rwmj 2y ago
Is using pthread_key_create / pthread_setspecific / pthread_getspecific / pthread_key_delete better or worse or the same? As I understand it, this is the low level API you were looking for.
- nicbn 2y ago
  iirc pthread uses the same ABI under the hood, only its compiled into the libc
- Sesse__ 2y ago
  It's much worse, since you can't inline it; it becomes a shared-library call.
  - yosefk 2y ago
    Ah, but you can "inline" it by hardcoding its implementation details into your code in the most horrible way imaginable, as suggested by a crafty reader and which, scarily enough, seems to be the fastest and most robust TLS access method from a shared library: https://yosefk.com/blog/cxx-thread-local-storage-performance.html#inlining-pthread_getspecific https://yosefk.com/blog/cxx-thread-local-storage-performance...
eqvinox 2y ago
For your specific use case, CPU-local (rather than thread-local) memory might also be an option. Search for rseq on Linux. Relatively fresh feature though, not a lot of experience around it exists yet.
- nwlieb 2y ago
  Known users: - LTTng tracer - tcmalloc Curious if there are other prominent users of rseq.
  - yosefk 2y ago
    For a tracing profiler, you want to know which thread a function call or return was made by. LTTng has kernel modules which it can use to trace context switches, and then a per-CPU trace buffer is fine, provided that you get cheap atomic writes which rseq can be used for. Funtrace on the other hand does support ftrace for tracing context switches (https://yosefk.com/blog/profiling-in-production-with-function-call-traces.html#ftrace-tracing-thread-state-changes https://yosefk.com/blog/profiling-in-production-with-functio...), but it doesn't require ftrace for tracing function calls made by your threads (the problem with ftrace as well LTTng's kernel modules being, of course, permissions; which shouldn't be a problem in any reasonable situation by my standard of "reasonable", but many find themselves in unreasonable situations permissions-wise, sadly.) So I don't think funtrace can use rseq, though I might be missing something.
    - HALtheWise 2y ago
      Presumably you could store the TID in every event, or otherwise check whether the TID has changed since the last time it was logged and push a (timestamp, TID) pair if so. Reading TID should be cheap.
      - yosefk 2y ago
        In what sense should reading the TID be cheap? You would need either a syscall (not cheap) or thread-local storage (the subject of TFA.) Avoiding the use of TLS by reading the TID can't really work
gavinhoward 2y ago
Big fan. I wish you could write more. I am gonna go check the assembly of my own code for this. And I am going to integrate funtrace into my workflow. Question about your build system comment: my build system doesn't stoop to figuring out if `-fPIC` is needed, but it also doesn't add it unless the user asks. Were you talking about that or build systems that add it automatically?
- yosefk 2y ago
  Thanks for you kind words! I don't think build systems add -fPIC automatically, nor remove it automatically. C++ build systems do not stoop to the question of how to best build a C++ program, by and large. They are more task graph executors - either bad ones, like make, or good ones, like Bazel, but mostly task graph executors; the most "C++ support" you will get is native support for scanning #include files (as opposed to doing it yourself like make forces you to.)
  - gavinhoward 2y ago
    Makes sense. I want to build a "standard library" for my build system that would stoop to that. I can do that because my build system is not just a task graph executor; it is backed by a full programming language and can add its own libraries to that language. IOW, I can add a `cpp` package to the build system that implements support for how to best build a C++ program. So if you have a wishlist for that support, I'd love to hear it.
  - smeenai 2y ago
    Idk about Bazel, but Buck2 (and Buck1, but 2 is better in every way) does handle adding -fPIC for shared libraries and leaving it off when statically linking. Each library has a PIC and non-PIC version, and the appropriate one is selected depending on if the library is being linked into an executable or a shared library. (The versions are only built when needed.) I don't remember if it also handles -fPIE similarly smartly though if you do want position-independent executables.
  - 1718627440 2y ago
    > I don't think build systems add -fPIC automatically, nor remove it automatically. GNU Autotools do, if you tell, that you have a shared library. You can also switch between static/dynamic linking at build time and if you also use GNU Libtool, it will figure out the flags of your build platform at build time. > as opposed to doing it yourself like make forces you to You can definitely don't have to do that. GNU Automake does it by default, but if you are using plain Make, you can also use makedepend or the appropriate flags of your compiler. > either bad ones, like make What is wrong/missing with make as a task graph executor? > or good ones, like Bazel What can Bazel do better?
nwlieb 2y ago
Hi, love the article. You mention in the article that a hardware mechanism for tracing should exist -- have you investigated the intel_pt (processor trace) extension? I believe this uses hardware buffers and supports timestamping & cycle counters (at somewhat larger than instruction granularity sadly, although it might issue forced stamps at on branches, not sure). You can also use the PTWRITE instruction to attach metadata to the stream which seems very powerful. Hope we can see such an extension on AMD as well.
- Sesse__ 2y ago
  Intel PT is indeed useful (although very, very slow compared to regular sampling profiling), but there's hardly any CPUs that actually implement PTWRITE. (IIRC there's some obscure Xeon or something?) Typically you get a cycle count every six branches, give or take.
  - yosefk 2y ago
    Sampling profilers are indeed very low-overhead, however they can't help debug tail latency, for which tracing profilers are indispensable: https://yosefk.com/blog/profiling-in-production-with-function-call-traces.html#viztracer-how-to-use-a-great-tracing-profiler https://yosefk.com/blog/profiling-in-production-with-functio... https://danluu.com/perf-tracing/ https://danluu.com/perf-tracing/ Regarding the slowdown - magic-trace reports 2-10% slowdowns which IMO is actually fine even for production (unless this adds up to a huge dollar cost, for most people it won't) since in return for this you are actually capable to debug the rare slowdowns which are the worst part of your user experience. However, the hardware feature that I propose (https://yosefk.com/blog/profiling-in-production-with-function-call-traces.html#hardware-assisted-tracing https://yosefk.com/blog/profiling-in-production-with-functio...) would likely have lower overhead since it relies on software issuing tracing instructions, eg at each function entry & exit (rather than any control flow change), and it could be variously selective (eg exclude short functions without loops; and/or you could configure the hardware to ignore short calls. BTW maybe you can with Intel Performance Trace, too, I'm just not really familiar with it.)
- yosefk 2y ago
mgaunard 2y ago
Calling perf a sampling profiler is somewhat misleading; it can trace and count as well.

by yosefk 2y ago

atq2119 2y ago

Take a look at https://maskray.me/blog/2021-02-14-all-about-thread-local-storage https://maskray.me/blog/2021-02-14-all-about-thread-local-st... I think what you want is to force the "initial exec" model using an attribute to get the more efficient code. IIRC this (or something equivalent) is what libGL does because basically every OpenGL function call needs to read the thread-local variable holding the current GL context. The downside is that dlopen()ing your library may fail.

yosefk 2y ago
Doesn't the "initial exec" model require the user to run the executable with LD_PRELOAD=your.so or linking against your.so (DT_NEEDED)? eg would it work with a Python extension module which is dlopen'd at runtime?
- fweimer 2y ago
  It depends on the amount of TLS and the number of objects that do this, and the C library. In glibc, we have a small static TLS reservation dedicated for use by dlopen of shared objects with initial-exec.

fweimer 2y ago

With glibc, you can use -ftls-model=initial-exec (or the corresponding variable attribute) to get offset-based TLS. The offset is variable per program (unlike local-exec), but the same for all threads, so it's more efficient. Using too much initial-exec TLS (potential across multiple shared objects) eventually causes dlopen to fail because the TCB cannot be resized. This is not a problem if the shared objects are loaded through dependencies at process start. If initial-exec TLS does not work due to the dlopen issue, on x86-64 and recent-enough distributions, you can use -mtls-dialect=gnu2 to get a faster variant of __tls_get_addr that requires less register spilling. Unfortunately glibc and GCC originally did not agree on the ABI: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113874 https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113874 https://sourceware.org/bugzilla/show_bug.cgi?id=31372 https://sourceware.org/bugzilla/show_bug.cgi?id=31372 This has been fixed for RHEL 10 (which switched to -mtls-dialect=gnu2 for x86-64 for the whole distribution, thereby exposing the ABI bug during development). As the ABI was fixed on the glibc side in dynamically-linked code, the change is backportable, but it's a bit involved because the first XSAVE-using change upstream was buggy, if I recall correctly. But the backport is definitely something you could request from your distribution. Note that there was a previous bug in __tls_get_addr (on all architectures that use it), where the fast path was not always used after dlopen: https://sourceware.org/bugzilla/show_bug.cgi?id=19924 https://sourceware.org/bugzilla/show_bug.cgi?id=19924 This bug introduced way more overhead that just saving registers. I expect that quite a few distributions have backported the fix. This breaks certain interposed mallocs due to a malloc/TLS cyclic dependency, but there is a workaround for that: https://sourceware.org/git/?p=glibc.git;a=commitdiff;h=018f0fc3b818d4d1460a4e2384c24802504b1d20 https://sourceware.org/git/?p=glibc.git;a=commitdiff;h=018f0... The other issue is just that the C++ TLS-with-constructors design isn't that great. You can work around this in the application by using a plain pointer for TLS access, which starts out as NULL and is initialized after a null check. To free the pointer on thread exit, you can use a separate TLS variable or POSIX thread-specific data (pthread_key_create) to register a destructor, and that will only be accessed on initialized and thread exit. This sort of question is probably more suited to libc-help: https://sourceware.org/mailman/listinfo/libc-help/ https://sourceware.org/mailman/listinfo/libc-help/

dzaima 2y ago
> Using too much initial-exec TLS (potential across multiple shared objects) eventually causes dlopen to fail because the TCB cannot be resized. This feels like the perfect situation to preallocate a gigabyte or something of virtual memory for extending the TLS, similar to how the stack is. But, testing on my system, looks like the allowed initial-exec TLS size is just ~1700 bytes.

sparkie 2y ago

GCC and Clang both provide `_readfsbase_u64()`, which presumably is just a wrapper around `__asm__("rdfsbase %0" : "=r"(result))`. Also gcc lets you define variables like `int__seg_fs *foo;`if the `__SEG_FS` preprocessor symbol is defined. With Clang, you can use `int __attribute__((address_space(257))) *foo`

damon_dam 2y ago
To the author: If you have a specific situation that the codegen is failing to optimize well (e.g. the ctor cases that you ran into) you can store the offset to your variable in a non-TLS global, then manually add the FS/GS to it. Use inline asm if you need to bypass any init checks. An alternative that might be worth looking into is just hashing the FS/GS into a table index. It will be slower than the well-optimized case, but it will let you opt out of the TLS allocation process altogether. This might be a good thing in some cases for a low-level facility like a function tracer.
- yosefk 2y ago
  I have updated the post with all the suggestions people replied with except this one, because I don't understand it. How do I allocate memory for addressing it with FS/GS? Isn't FS the register pointing to the TLS area - then how is the FS-based access you propose different from how TLS normally works? Isn't GS used "for something" on x86?.. If you could elaborate on this/show some code I would be very grateful! Note that my question is about shared libraries. If the thread_local is linked into an executable, I guess you could indeed save the offset somewhere and then add the value of %fs to it, though if this is a way to work around the constructor issue, I prefer to not have a constructor. The question is if this sort of direction can help for thread-local storage allocated by a shared library.
  - damon_dam 2y ago
    > How do I allocate memory for addressing it with FS/GS? Allocate the variable normally, then compute the offset in one thread (e.g. offset = uintptr_t(&variable) - get_fs()), then access it by adding the offset to FS in any thread (e.g. (vartype *) (offset + get_fs())). The only difference from how it normally works is that you can manually force it to be inlined, sidestepping the codegen problems you described in your post. But if you can avoid those problems by not using constructors instead, that's definitely better. I used "FS/GS" because GS is used instead of FS on some systems for the same purpose. The shared library-specific issues are one of the reasons I was suggesting maybe looking into hashing, e.g. perhaps as a fallback solution when the TLS approach fails.

yxhuvud 2y ago

Could it be productive to report the slowness to the compiler (or ld) makers? Is this inherently bad or could it be fixed by the toolmakers?

rwmj 2y ago

Is using pthread_key_create / pthread_setspecific / pthread_getspecific / pthread_key_delete better or worse or the same? As I understand it, this is the low level API you were looking for.

nicbn 2y ago
iirc pthread uses the same ABI under the hood, only its compiled into the libc
Sesse__ 2y ago
It's much worse, since you can't inline it; it becomes a shared-library call.
- yosefk 2y ago
  Ah, but you can "inline" it by hardcoding its implementation details into your code in the most horrible way imaginable, as suggested by a crafty reader and which, scarily enough, seems to be the fastest and most robust TLS access method from a shared library: https://yosefk.com/blog/cxx-thread-local-storage-performance.html#inlining-pthread_getspecific https://yosefk.com/blog/cxx-thread-local-storage-performance...

eqvinox 2y ago

For your specific use case, CPU-local (rather than thread-local) memory might also be an option. Search for rseq on Linux. Relatively fresh feature though, not a lot of experience around it exists yet.

nwlieb 2y ago
Known users: - LTTng tracer - tcmalloc Curious if there are other prominent users of rseq.
- yosefk 2y ago
  For a tracing profiler, you want to know which thread a function call or return was made by. LTTng has kernel modules which it can use to trace context switches, and then a per-CPU trace buffer is fine, provided that you get cheap atomic writes which rseq can be used for. Funtrace on the other hand does support ftrace for tracing context switches (https://yosefk.com/blog/profiling-in-production-with-function-call-traces.html#ftrace-tracing-thread-state-changes https://yosefk.com/blog/profiling-in-production-with-functio...), but it doesn't require ftrace for tracing function calls made by your threads (the problem with ftrace as well LTTng's kernel modules being, of course, permissions; which shouldn't be a problem in any reasonable situation by my standard of "reasonable", but many find themselves in unreasonable situations permissions-wise, sadly.) So I don't think funtrace can use rseq, though I might be missing something.
  - HALtheWise 2y ago
    Presumably you could store the TID in every event, or otherwise check whether the TID has changed since the last time it was logged and push a (timestamp, TID) pair if so. Reading TID should be cheap.
    - yosefk 2y ago
      In what sense should reading the TID be cheap? You would need either a syscall (not cheap) or thread-local storage (the subject of TFA.) Avoiding the use of TLS by reading the TID can't really work

gavinhoward 2y ago

Big fan. I wish you could write more. I am gonna go check the assembly of my own code for this. And I am going to integrate funtrace into my workflow. Question about your build system comment: my build system doesn't stoop to figuring out if `-fPIC` is needed, but it also doesn't add it unless the user asks. Were you talking about that or build systems that add it automatically?

yosefk 2y ago
Thanks for you kind words! I don't think build systems add -fPIC automatically, nor remove it automatically. C++ build systems do not stoop to the question of how to best build a C++ program, by and large. They are more task graph executors - either bad ones, like make, or good ones, like Bazel, but mostly task graph executors; the most "C++ support" you will get is native support for scanning #include files (as opposed to doing it yourself like make forces you to.)
- gavinhoward 2y ago
  Makes sense. I want to build a "standard library" for my build system that would stoop to that. I can do that because my build system is not just a task graph executor; it is backed by a full programming language and can add its own libraries to that language. IOW, I can add a `cpp` package to the build system that implements support for how to best build a C++ program. So if you have a wishlist for that support, I'd love to hear it.
- smeenai 2y ago
  Idk about Bazel, but Buck2 (and Buck1, but 2 is better in every way) does handle adding -fPIC for shared libraries and leaving it off when statically linking. Each library has a PIC and non-PIC version, and the appropriate one is selected depending on if the library is being linked into an executable or a shared library. (The versions are only built when needed.) I don't remember if it also handles -fPIE similarly smartly though if you do want position-independent executables.
- 1718627440 2y ago
  > I don't think build systems add -fPIC automatically, nor remove it automatically. GNU Autotools do, if you tell, that you have a shared library. You can also switch between static/dynamic linking at build time and if you also use GNU Libtool, it will figure out the flags of your build platform at build time. > as opposed to doing it yourself like make forces you to You can definitely don't have to do that. GNU Automake does it by default, but if you are using plain Make, you can also use makedepend or the appropriate flags of your compiler. > either bad ones, like make What is wrong/missing with make as a task graph executor? > or good ones, like Bazel What can Bazel do better?

nwlieb 2y ago

Hi, love the article. You mention in the article that a hardware mechanism for tracing should exist -- have you investigated the intel_pt (processor trace) extension? I believe this uses hardware buffers and supports timestamping & cycle counters (at somewhat larger than instruction granularity sadly, although it might issue forced stamps at on branches, not sure). You can also use the PTWRITE instruction to attach metadata to the stream which seems very powerful. Hope we can see such an extension on AMD as well.

Sesse__ 2y ago
Intel PT is indeed useful (although very, very slow compared to regular sampling profiling), but there's hardly any CPUs that actually implement PTWRITE. (IIRC there's some obscure Xeon or something?) Typically you get a cycle count every six branches, give or take.
- yosefk 2y ago
  Sampling profilers are indeed very low-overhead, however they can't help debug tail latency, for which tracing profilers are indispensable: https://yosefk.com/blog/profiling-in-production-with-function-call-traces.html#viztracer-how-to-use-a-great-tracing-profiler https://yosefk.com/blog/profiling-in-production-with-functio... https://danluu.com/perf-tracing/ https://danluu.com/perf-tracing/ Regarding the slowdown - magic-trace reports 2-10% slowdowns which IMO is actually fine even for production (unless this adds up to a huge dollar cost, for most people it won't) since in return for this you are actually capable to debug the rare slowdowns which are the worst part of your user experience. However, the hardware feature that I propose (https://yosefk.com/blog/profiling-in-production-with-function-call-traces.html#hardware-assisted-tracing https://yosefk.com/blog/profiling-in-production-with-functio...) would likely have lower overhead since it relies on software issuing tracing instructions, eg at each function entry & exit (rather than any control flow change), and it could be variously selective (eg exclude short functions without loops; and/or you could configure the hardware to ignore short calls. BTW maybe you can with Intel Performance Trace, too, I'm just not really familiar with it.)
yosefk 2y ago

mgaunard 2y ago

Calling perf a sampling profiler is somewhat misleading; it can trace and count as well.