4 ms·

As the article says at the beginning, this is intended for per-CPU data structures, not shared ones. For example, buffer pools where you don't care exactly whic

As the article says at the beginning, this is intended for per-CPU data structures, not shared ones. For example, buffer pools where you don't care exactly which pool a given thread takes its buffers from, but you do care about maximizing performance and minimizing contention.

PaulDavisThe1st 5y ago
That also requires pinning the thread to the cpu, or it can find itself "on the wrong core" when it wants to release the buffer. This narrows the use case even further. To be clear I write software that could nominally benefit from stuff like this, but the tradeoffs (requiring pinning, in particular, which may not be available on all platforms) mean that it seems better to use approaches (e.g. RCU) that will work without restartable sequences.
- uluyol 5y ago
  I think most allocators don't care which core you call free on. It's already the case that you can malloc from any core and free on another. Special purpose allocators maybe. Agree that this mechanism overall is rather niche, but that niche (malloc) is a rather important one.
  - PaulDavisThe1st 5y ago
    In the GP comment from @teraflop: > As the article says at the beginning, this is intended for per-CPU data structures, not shared ones.
    - CaliforniaKarl 5y ago
      Indeed. And even though glibc has only recently introduced support for rseq, the rseq system call has been around for a while. Search around, I found tcmalloc (https://google.github.io/tcmalloc/ https://google.github.io/tcmalloc/) is an example of software that is using rseq today, without glibc's support. And as far as I can tell, tcmalloc is using rseq both with a 'per-CPU' structure (actually portions of a larger block of memory), and without a requirement to pin processes or threads to a particular CPU. Their design document (https://google.github.io/tcmalloc/rseq.html https://google.github.io/tcmalloc/rseq.html) might be of interest to you.
    - uluyol 5y ago
      The original use case (I believe) was tcmalloc. Tcmalloc maintained per-thread caches of free memory, but that's wasteful since it increases fragmentation but has no parallelism advantage compared to per-CPU buffers. OTOH, you can't just use per-CPU data since you can get preempted, hence rseq. malloc from the per-CPU cache without locking, and free to any other cache without locking.
- Veserv 5y ago
  You do not require pinning, rather that is the opposite of the use case. An actual use case where restartable windows are useful is when writing a data entry to a per-core buffer where the entire acquire-write-release sequence can be fit into the restartable window. This guarantees that the entirety of the write will occur on a single core even if it is preempted or moved as the sequence will restart on the new core if you get moved. The advantages of this approach are that you are guaranteed the buffer will be in the cache of the core being executed on guaranteeing excellent cache locality. The disadvantages are that you may have to redo the writes if you get preempted, but that should be very unlikely if your write is not too long. In terms of the general case, restartable windows can be thought of as having a disable_preemption() or disable_core_migration() similar to how you might have a way of disabling interrupts except with some more constraints on what you can do while things are disabled.

PaulDavisThe1st 5y ago

That also requires pinning the thread to the cpu, or it can find itself "on the wrong core" when it wants to release the buffer. This narrows the use case even further. To be clear I write software that could nominally benefit from stuff like this, but the tradeoffs (requiring pinning, in particular, which may not be available on all platforms) mean that it seems better to use approaches (e.g. RCU) that will work without restartable sequences.

uluyol 5y ago
I think most allocators don't care which core you call free on. It's already the case that you can malloc from any core and free on another. Special purpose allocators maybe. Agree that this mechanism overall is rather niche, but that niche (malloc) is a rather important one.
- PaulDavisThe1st 5y ago
  In the GP comment from @teraflop: > As the article says at the beginning, this is intended for per-CPU data structures, not shared ones.
  - CaliforniaKarl 5y ago
    Indeed. And even though glibc has only recently introduced support for rseq, the rseq system call has been around for a while. Search around, I found tcmalloc (https://google.github.io/tcmalloc/ https://google.github.io/tcmalloc/) is an example of software that is using rseq today, without glibc's support. And as far as I can tell, tcmalloc is using rseq both with a 'per-CPU' structure (actually portions of a larger block of memory), and without a requirement to pin processes or threads to a particular CPU. Their design document (https://google.github.io/tcmalloc/rseq.html https://google.github.io/tcmalloc/rseq.html) might be of interest to you.
  - uluyol 5y ago
    The original use case (I believe) was tcmalloc. Tcmalloc maintained per-thread caches of free memory, but that's wasteful since it increases fragmentation but has no parallelism advantage compared to per-CPU buffers. OTOH, you can't just use per-CPU data since you can get preempted, hence rseq. malloc from the per-CPU cache without locking, and free to any other cache without locking.
Veserv 5y ago
You do not require pinning, rather that is the opposite of the use case. An actual use case where restartable windows are useful is when writing a data entry to a per-core buffer where the entire acquire-write-release sequence can be fit into the restartable window. This guarantees that the entirety of the write will occur on a single core even if it is preempted or moved as the sequence will restart on the new core if you get moved. The advantages of this approach are that you are guaranteed the buffer will be in the cache of the core being executed on guaranteeing excellent cache locality. The disadvantages are that you may have to redo the writes if you get preempted, but that should be very unlikely if your write is not too long. In terms of the general case, restartable windows can be thought of as having a disable_preemption() or disable_core_migration() similar to how you might have a way of disabling interrupts except with some more constraints on what you can do while things are disabled.