3 ms·

For your specific use case, CPU-local (rather than thread-local) memory might also be an option. Search for rseq on Linux. Relatively fresh feature though, no

For your specific use case, CPU-local (rather than thread-local) memory might also be an option. Search for rseq on Linux. Relatively fresh feature though, not a lot of experience around it exists yet.

nwlieb 2y ago
Known users: - LTTng tracer - tcmalloc Curious if there are other prominent users of rseq.
- yosefk 2y ago
  For a tracing profiler, you want to know which thread a function call or return was made by. LTTng has kernel modules which it can use to trace context switches, and then a per-CPU trace buffer is fine, provided that you get cheap atomic writes which rseq can be used for. Funtrace on the other hand does support ftrace for tracing context switches (https://yosefk.com/blog/profiling-in-production-with-function-call-traces.html#ftrace-tracing-thread-state-changes https://yosefk.com/blog/profiling-in-production-with-functio...), but it doesn't require ftrace for tracing function calls made by your threads (the problem with ftrace as well LTTng's kernel modules being, of course, permissions; which shouldn't be a problem in any reasonable situation by my standard of "reasonable", but many find themselves in unreasonable situations permissions-wise, sadly.) So I don't think funtrace can use rseq, though I might be missing something.
  - HALtheWise 2y ago
    Presumably you could store the TID in every event, or otherwise check whether the TID has changed since the last time it was logged and push a (timestamp, TID) pair if so. Reading TID should be cheap.
    - yosefk 2y ago
      In what sense should reading the TID be cheap? You would need either a syscall (not cheap) or thread-local storage (the subject of TFA.) Avoiding the use of TLS by reading the TID can't really work
      - HALtheWise 2y ago
        It looks like the TID is stored directly in the pthread struct pointed to by %fs itself, at a fixed offset which you can somewhat-hackily compile into your code. [0] In the process of investigating this, I also realized that there's a ton of other unique-per-thread pointers accessible from that structure, most notably including the value of %fs itself (which is unfortunately unobservable afaict), the address of the TCB or TLS structures, the stack guard value, etc. Since the goal is just to have a quickly-readable unique-per-thread value, any of those should work. Windows looks similar, but I haven't investigated as deeply. [0] https://github.com/andikleen/glibc/blob/b0399147730d478ae45160051a8a0f00f91ef965/nptl/descr.h#L169 https://github.com/andikleen/glibc/blob/b0399147730d478ae451... [1] https://github.com/andikleen/glibc/blob/b0399147730d478ae45160051a8a0f00f91ef965/nptl/sysdeps/x86_64/tls.h#L80 https://github.com/andikleen/glibc/blob/b0399147730d478ae451...
loeg 2y ago
This wouldn't have syntax sugar like thread_local, but presumably you're just calling some inlined helper function? How do you avoid the problem of threads migrating between CPUs at arbitrary instruction boundaries?
- eqvinox 2y ago
  > How do you avoid the problem of threads migrating between CPUs at arbitrary instruction boundaries? rseq is short for restartable sequences; you mark beginning and end of a range of instructions, plus an abort path. The kernel checks this during task-switch and if you're within the range the instruction pointer is changed to the abort path on resumption. (It's called restartable because the assumption is that the abort path will try again. Or at least, aborting the block of instructions midway through is recoverable.) The primary limitation is that the "result" of the block in most cases needs to be concentrated into the last instruction of the block (similar to an atomic release write.) Otherwise you'd need to somehow recover from partially executed rseq blocks having changed some visible state but not fully completed.
senderista 2y ago
Be careful, we ran into a segfault with rseq when used in a DSO loaded by another DSO.

nwlieb 2y ago

Known users: - LTTng tracer - tcmalloc Curious if there are other prominent users of rseq.

yosefk 2y ago
For a tracing profiler, you want to know which thread a function call or return was made by. LTTng has kernel modules which it can use to trace context switches, and then a per-CPU trace buffer is fine, provided that you get cheap atomic writes which rseq can be used for. Funtrace on the other hand does support ftrace for tracing context switches (https://yosefk.com/blog/profiling-in-production-with-function-call-traces.html#ftrace-tracing-thread-state-changes https://yosefk.com/blog/profiling-in-production-with-functio...), but it doesn't require ftrace for tracing function calls made by your threads (the problem with ftrace as well LTTng's kernel modules being, of course, permissions; which shouldn't be a problem in any reasonable situation by my standard of "reasonable", but many find themselves in unreasonable situations permissions-wise, sadly.) So I don't think funtrace can use rseq, though I might be missing something.
- HALtheWise 2y ago
  Presumably you could store the TID in every event, or otherwise check whether the TID has changed since the last time it was logged and push a (timestamp, TID) pair if so. Reading TID should be cheap.
  - yosefk 2y ago
    In what sense should reading the TID be cheap? You would need either a syscall (not cheap) or thread-local storage (the subject of TFA.) Avoiding the use of TLS by reading the TID can't really work
    - HALtheWise 2y ago
      It looks like the TID is stored directly in the pthread struct pointed to by %fs itself, at a fixed offset which you can somewhat-hackily compile into your code. [0] In the process of investigating this, I also realized that there's a ton of other unique-per-thread pointers accessible from that structure, most notably including the value of %fs itself (which is unfortunately unobservable afaict), the address of the TCB or TLS structures, the stack guard value, etc. Since the goal is just to have a quickly-readable unique-per-thread value, any of those should work. Windows looks similar, but I haven't investigated as deeply. [0] https://github.com/andikleen/glibc/blob/b0399147730d478ae45160051a8a0f00f91ef965/nptl/descr.h#L169 https://github.com/andikleen/glibc/blob/b0399147730d478ae451... [1] https://github.com/andikleen/glibc/blob/b0399147730d478ae45160051a8a0f00f91ef965/nptl/sysdeps/x86_64/tls.h#L80 https://github.com/andikleen/glibc/blob/b0399147730d478ae451...

loeg 2y ago

This wouldn't have syntax sugar like thread_local, but presumably you're just calling some inlined helper function? How do you avoid the problem of threads migrating between CPUs at arbitrary instruction boundaries?

eqvinox 2y ago
> How do you avoid the problem of threads migrating between CPUs at arbitrary instruction boundaries? rseq is short for restartable sequences; you mark beginning and end of a range of instructions, plus an abort path. The kernel checks this during task-switch and if you're within the range the instruction pointer is changed to the abort path on resumption. (It's called restartable because the assumption is that the abort path will try again. Or at least, aborting the block of instructions midway through is recoverable.) The primary limitation is that the "result" of the block in most cases needs to be concentrated into the last instruction of the block (similar to an atomic release write.) Otherwise you'd need to somehow recover from partially executed rseq blocks having changed some visible state but not fully completed.

senderista 2y ago

Be careful, we ran into a segfault with rseq when used in a DSO loaded by another DSO.