5 ms·

It's a bit sad that all these low-latency approaches require burning CPU in a loop, which uses tons of power.

fmstephe 12y ago
In this particular case you should be able to use no-op instructions to spin without consuming much power. Since the consumer only spins when the queue is empty we know there will be space on the ring buffer and producers won't be impacted. When the consumer wakes from a no-op loop and finds work on the queue it can switch back to a hot loop for a certain amount of time before returning to a no-op cold loop. My understanding is that even having very small no-op pauses can significantly reduce the amount of energy used, and crucially heat generated, while only having a very modest impact on latency. A good fit for bursty low latency systems. To be clear, I am only relating something that another developer described to me. I've not actually implemented this myself. :)
- mortoray 12y ago
  My loop continually checked the buffers and the locks, it never did a noop. Those fenced instructions are probably not the most power efficient. Power consumption was not a concern, latency was our primary goal.
- a-priori 12y ago
  I think your colleague is talking about the PAUSE instruction (which is also known as REP NOP, since they encode to the same bytes). It's a special instruction that hints to the processor that it's in a spin-loop waiting on a synchronisation variable to change. It's used in tight wait loops like this: wait_loop: pause cmp eax, sync_var jne wait_loop The PAUSE instruction introduces a small delay to synchronise the spin loop to the memory bus frequency. This is a power savings, since the value of sync_var, as seen from the perspective of this code, can't change faster than that. Also, because the CPU can execute much faster than the memory bus, it prevents many memory requests from piling up while in the loop (because of out-of-order execution; those requests will have to be unrolled when the loop exits), making the loop faster to exit. Because of Hyperthreading, the pause also gives another thread a brief opportunity to execute on the same core. https://software.intel.com/sites/default/files/m/d/4/1/d/8/17689_w_spinlock.pdf https://software.intel.com/sites/default/files/m/d/4/1/d/8/1... Intel does, however, recommend other approaches to dealing with tight loops like this. See here: https://software.intel.com/en-us/articles/long-duration-spin-wait-loops-on-hyper-threading-technology-enabled-intel-processors/ https://software.intel.com/en-us/articles/long-duration-spin...
  - fmstephe 12y ago
    I take my hat off to you a-priori
    - a-priori 12y ago
      Glad to help :)
  - arielweisberg 12y ago
    How would there be many requests? Wouldn't it load the cache line once into the shared state and then spin waiting for the line to be invalidated before reloading?
    - a-priori 12y ago
      Honestly, I don't know how this interacts with cache lines. As far as I know, Intel has not released any official details about what the PAUSE instruction does other than that it slows down spin loops to a reasonable rate. The best source I know of for this information is the Intel® 64 and IA-32 Architectures Optimization Reference Manual (http://www.intel.com/content/dam/www/public/us/en/documents/manuals/64-ia-32-architectures-optimization-manual.pdf http://www.intel.com/content/dam/www/public/us/en/documents/...). It is fairly vague about what the instruction does, but gives some useful information in 13.5.3 "Spin-Wait Loops", as well as 8.4 "Thread Synchronization", and in particular 8.4.2 "Synchronization for Short Periods" which says: "On a modern microprocessor with a superscalar speculative execution engine, [a spin loop] results in the issue of multiple simultaneous read requests from the spinning thread. These requests usually execute out-of-order with each read request being allocated a buffer resource. On detection of a write by a worker thread to a load that is in progress, the processor must guarantee no violations of memory order occur. The necessity of maintaining the order of outstanding memory operations inevitably costs the processor a severe penalty that impacts all threads." So it seems that the concern here is that (without a PAUSE) the speculative execution engine effectively unrolls the spin loop and executes a sequence of reads on the sequence variable. This queues up a list of pending memory operations that need to be unrolled in order to ensure that the result is the same as if they were executed sequentially -- even though in this case it wouldn't make any difference.
mortoray 12y ago
It is very unfortunate. On that project we looked at all sorts of ways to avoid burning cores but they all had significant latency issues. If it involved going to the kernel then there was a major context switch overhead. But the big killer is the non-constant time. Each time you go to the kernel there's a danger of loosing your timeslice or being shunted to some other core. This leads to very high variance in performance. (I must always repeat though, this variance is not likely relevant in most applications.) So we spun in loops. And we had the CPUs set to maximum performance (no idle-time scaling). The total real load was probably less than 1%, so yes, it's very unfortunate.
nkurz 12y ago
It would seem like MONITOR/MWAIT (https://blogs.oracle.com/dave/resource/mwait-blog-final.txt https://blogs.oracle.com/dave/resource/mwait-blog-final.txt) would be the best of both worlds: extremely low-power and low-latency. But I see very little about it. Are there reasons that it's not more widely used?

In this particular case you should be able to use no-op instructions to spin without consuming much power. Since the consumer only spins when the queue is empty we know there will be space on the ring buffer and producers won't be impacted. When the consumer wakes from a no-op loop and finds work on the queue it can switch back to a hot loop for a certain amount of time before returning to a no-op cold loop. My understanding is that even having very small no-op pauses can significantly reduce the amount of energy used, and crucially heat generated, while only having a very modest impact on latency. A good fit for bursty low latency systems. To be clear, I am only relating something that another developer described to me. I've not actually implemented this myself. :)

mortoray 12y ago
My loop continually checked the buffers and the locks, it never did a noop. Those fenced instructions are probably not the most power efficient. Power consumption was not a concern, latency was our primary goal.
a-priori 12y ago
I think your colleague is talking about the PAUSE instruction (which is also known as REP NOP, since they encode to the same bytes). It's a special instruction that hints to the processor that it's in a spin-loop waiting on a synchronisation variable to change. It's used in tight wait loops like this: wait_loop: pause cmp eax, sync_var jne wait_loop The PAUSE instruction introduces a small delay to synchronise the spin loop to the memory bus frequency. This is a power savings, since the value of sync_var, as seen from the perspective of this code, can't change faster than that. Also, because the CPU can execute much faster than the memory bus, it prevents many memory requests from piling up while in the loop (because of out-of-order execution; those requests will have to be unrolled when the loop exits), making the loop faster to exit. Because of Hyperthreading, the pause also gives another thread a brief opportunity to execute on the same core. https://software.intel.com/sites/default/files/m/d/4/1/d/8/17689_w_spinlock.pdf https://software.intel.com/sites/default/files/m/d/4/1/d/8/1... Intel does, however, recommend other approaches to dealing with tight loops like this. See here: https://software.intel.com/en-us/articles/long-duration-spin-wait-loops-on-hyper-threading-technology-enabled-intel-processors/ https://software.intel.com/en-us/articles/long-duration-spin...
- fmstephe 12y ago
  I take my hat off to you a-priori
  - a-priori 12y ago
    Glad to help :)
- arielweisberg 12y ago
  How would there be many requests? Wouldn't it load the cache line once into the shared state and then spin waiting for the line to be invalidated before reloading?
  - a-priori 12y ago
    Honestly, I don't know how this interacts with cache lines. As far as I know, Intel has not released any official details about what the PAUSE instruction does other than that it slows down spin loops to a reasonable rate. The best source I know of for this information is the Intel® 64 and IA-32 Architectures Optimization Reference Manual (http://www.intel.com/content/dam/www/public/us/en/documents/manuals/64-ia-32-architectures-optimization-manual.pdf http://www.intel.com/content/dam/www/public/us/en/documents/...). It is fairly vague about what the instruction does, but gives some useful information in 13.5.3 "Spin-Wait Loops", as well as 8.4 "Thread Synchronization", and in particular 8.4.2 "Synchronization for Short Periods" which says: "On a modern microprocessor with a superscalar speculative execution engine, [a spin loop] results in the issue of multiple simultaneous read requests from the spinning thread. These requests usually execute out-of-order with each read request being allocated a buffer resource. On detection of a write by a worker thread to a load that is in progress, the processor must guarantee no violations of memory order occur. The necessity of maintaining the order of outstanding memory operations inevitably costs the processor a severe penalty that impacts all threads." So it seems that the concern here is that (without a PAUSE) the speculative execution engine effectively unrolls the spin loop and executes a sequence of reads on the sequence variable. This queues up a list of pending memory operations that need to be unrolled in order to ensure that the result is the same as if they were executed sequentially -- even though in this case it wouldn't make any difference.

mortoray 12y ago

It is very unfortunate. On that project we looked at all sorts of ways to avoid burning cores but they all had significant latency issues. If it involved going to the kernel then there was a major context switch overhead. But the big killer is the non-constant time. Each time you go to the kernel there's a danger of loosing your timeslice or being shunted to some other core. This leads to very high variance in performance. (I must always repeat though, this variance is not likely relevant in most applications.) So we spun in loops. And we had the CPUs set to maximum performance (no idle-time scaling). The total real load was probably less than 1%, so yes, it's very unfortunate.

nkurz 12y ago

It would seem like MONITOR/MWAIT (https://blogs.oracle.com/dave/resource/mwait-blog-final.txt https://blogs.oracle.com/dave/resource/mwait-blog-final.txt) would be the best of both worlds: extremely low-power and low-latency. But I see very little about it. Are there reasons that it's not more widely used?