6 ms·

Why volatile is hard to specify and implement

38 points by luu 11y ago

mlvljr 11y ago
Smth relevant (note a nice flame in comments :)): https://software.intel.com/en-us/blogs/2007/11/30/volatile-almost-useless-for-multi-threaded-programming https://software.intel.com/en-us/blogs/2007/11/30/volatile-a...
rdtsc 11y ago
Volatile is not just for memory mapped hardware, but is also needed when using shared memory, say between two processes. I have been bit by not having volatile modifiers for data residing in shared memory. Funny thing (and retrospectively obvious) is, it worked in debug mode, but in production data was seen by the other process. It was a fun exercise to find it one.
- Joky 11y ago
  No, volatile is not for multi-threading, this is an old misconception: https://software.intel.com/en-us/blogs/2007/11/30/volatile-almost-useless-for-multi-threaded-programming https://software.intel.com/en-us/blogs/2007/11/30/volatile-a...
  - rdtsc 11y ago
    > No, volatile is not for multi-threading, this is an old misconception: https://software.intel.com/en-us/blogs/2007/11/30/volatile-a.. https://software.intel.com/en-us/blogs/2007/11/30/volatile-a.... No, downvote arrow is only after you read the comment, not before. Can you show where I wrote "multi-threaded" in my comment?
    - Joky 11y ago
      "shared memory" and "other process" quite hints at the frequent use of volatile where fences and other synchronization primitives are needed. (Note: I didn't down-vote anything)
      - rdtsc 11y ago
        Sorry assumed it was you. It was the only comment and a downvote. Shared memory and processes is a different beast than just threads in same process. Memory was mmap-ed between the two OS procs. Didn't use locks in that part of code for a list of reasons..., and memory write order didn't matter as much (so didn't see an immediate need for memory fences, but someone below suggested those might work to fix compiler optimizing out accesses when nor in debug mode). Not sure what hinted at frequent use of volatile, it was the only use in 10 years. That is what made it memorable.
        Avernar 11y ago
        Wasn't me either. Can't downvote yet. :) Did some more research. Compilers have their own intrinsics that only affect the code generated (in addition to the processor ones). VS has _ReadBarrier, _WriteBarrier and _ReadWriteBarrier while GCC has asm volatile("" ::: "memory"); to prevent reordering optimizations. There really isn't any difference between multithread with common processes memory and multiprocess with mmaped shared memory. Both have mutliple threads accessing the same physical memory but in one case the OS made the memory common and the other you made the memory common. BTW, when I say multithreaded I'm talking real threads and not light/green/cooperative/fake/whatever threads.
        rdtsc 11y ago
        Thanks for doing more research. Did you get the idea there are two separate issues at hand -- 1) compiler optimizes memory access away because to its logic it seems the value is not written anywhere in that process. Say it could use a register instead if memory read. This is the effect I was seeing. 2) Compiler inserts the memory read instruction, but depending on the processor, at runtime, some of the writes will not show up at all, or be reorded. In this case running program on different CPU might make it appear to work correctly. I think there is a slight difference between the two from reading about back then. Can the first issue be fixed with a memory barrier?
        Avernar 11y ago
- Avernar 11y ago
  Volatile only affects how the compiler handles memory access. It does not tell the processor anything at all. The processor is still free to reorder reads and writes. You need to use the fence and lock instructions to create memory barriers especially on multi core processors.
  - rdtsc 11y ago
    > Volatile only affects how the compiler handles memory access My comment specifically mentions memory access being optimized away. You did read it, right ;-) ? Even has an example how debug mode made it work. There was nothing there about telling something to the processor. If anything it is wanting to tell something to the compiler... > You need to use the fence and lock instructions to create memory barriers especially on multi core processors. Yap definitely. But will that still prevent the compiler from optimizing away stuff from a shared memory block? Remember looking for memory barriers but didn't know if my ancient system had those exposed in pthreads library.
    - loeg 11y ago
      Yep, any useful compiler support for fence/lock intrinsics will prevent optimizing out those intrinsics.
      - rdtsc 11y ago
        Are memory barriers transactional i.e. something like begin barrier(), do some shm writes, end_barrier? I was declaring data structures in shared mmap-ed memory as volatile so compiler does not optimise it away. How would the compiler know not to optimise away operation between memory fences? A fence to me say something about the guarantee of the order in which operations are seen. The order wasn't the problem, write never making it to the other side was the issue...
        Avernar 11y ago
        There are many types of barriers depending on the processor architecture. They're usually used in pairs so for example a read barrier would be used when mutex is acquired and the a write barrier when it's released. If the compiler is smart about its memory barrier intrinsics it will make sure to reread anything after the read barrier and write everything before the write barrier. This will speed things up by not having to use volatile. Between the barriers it can cache reads and inside registers and optimize out multiple writes like normal. If the compiler can't do that then you need to use volatile. Volatile was meant for I/O where every read has to hit the bus. So if you're forced to use it then you just copy the volatile variable into a non volatile one for use inside your mutex to get some speed back.
- burfog 11y ago
  It's not for memory-mapped hardware. This is unsafe. You need something to flush the pipeline, prevent store merging, prevent caching, and more. It's nearly banned from the Linux kernel. It gets used in one place, for the clock tick counter. There is one other good use: suppression of compiler bugs.
  - gizmo686 11y ago
    Linux's reasoning is documented here [0]. The justification for not using it for memory-mapped IO is that you are supposed to use accessor functions. Volotile still has uses for implementing those accesor functions (with care taken for the other concerns; and, by there nature, these functions are architecture dependent anyway), or when you are not using a kernel. [0] https://www.kernel.org/doc/Documentation/volatile-considered-harmful.txt https://www.kernel.org/doc/Documentation/volatile-considered...
  - rdtsc 11y ago
    It was memory used for a low latency system, didn't have locks even spinlocks. In that case it was manipulating data directly in shm and compiler was optimizating away access to data. What is the current/modern mechanism to ensure a stucture or flag won't be optimized away from shared memory mapped between two processes? Are access functions available in userspace libraries?
  - Avernar 11y ago
    Volatile is for memory mapped IO. Some processors may need extra instuctions or configuration but that doesn't replace the need for volatile with MMIO. On x86 the MTTR/PAT mechanisms is what disables caching, write combining and enables write through to MMIO address ranges.
    - burfog 11y ago
      Yes it does replace the need for volatile. The actual access is done via assembly language, either inline (better) or not. You could make volatile work, probably, by putting inline assembly both before and after it. In that case you could probably even skip marking things valatile, because the inline assembly will make the compiler behave. Why bother? This is more complicated than just doing the access from assembly. MTRR/PAT is often not enough, or it would only be enough if you killed performance elsewhere. (different types of MMIO in the same page) It's also not something you can rely on if you want to write portable code.
deleted 11y ago
[deleted]
caf 11y ago
The example saying: This means that the compiler is allowed to optimize volatile int *p = somevalue; int s = *p + *p; as if it said volatile int *p = somevalue; int tmp = *p; int s = tmp + tmp; is incorrect. The clause about sequence points starts with "Futhermore, " so it is an additional restriction, and to boot it only talks about stores to the volatile-qualified object, not loads, so it is entirely inapplicable to this example. The prior clause saying "Therefore any expression referring to such an object shall be evaluated strictly according to the rules of the abstract machine, as described in 5.1.2.3." applies, and the rules of the abstract machine for the binary + operator are that each operand is evaluated (in an unspecified order). So both dereferences must be evaluated.
- vrfcodf 11y ago
  I agree, I think this part makes it clear: except as modified by the unknown factors mentioned previously. Although this is quite contradicting: 5.1.2.3. p4: In the abstract machine, all expressions are evaluated as specified by the semantics. An actual implementation need not evaluate part of an expression if it can deduce that its value is not used and that no needed side effects are produced (including any caused by calling a function or accessing a volatile object)

38 points by luu 11y ago

Smth relevant (note a nice flame in comments :)): https://software.intel.com/en-us/blogs/2007/11/30/volatile-almost-useless-for-multi-threaded-programming https://software.intel.com/en-us/blogs/2007/11/30/volatile-a...

rdtsc 11y ago

Volatile is not just for memory mapped hardware, but is also needed when using shared memory, say between two processes. I have been bit by not having volatile modifiers for data residing in shared memory. Funny thing (and retrospectively obvious) is, it worked in debug mode, but in production data was seen by the other process. It was a fun exercise to find it one.

Joky 11y ago
No, volatile is not for multi-threading, this is an old misconception: https://software.intel.com/en-us/blogs/2007/11/30/volatile-almost-useless-for-multi-threaded-programming https://software.intel.com/en-us/blogs/2007/11/30/volatile-a...
- rdtsc 11y ago
  > No, volatile is not for multi-threading, this is an old misconception: https://software.intel.com/en-us/blogs/2007/11/30/volatile-a.. https://software.intel.com/en-us/blogs/2007/11/30/volatile-a.... No, downvote arrow is only after you read the comment, not before. Can you show where I wrote "multi-threaded" in my comment?
  - Joky 11y ago
    "shared memory" and "other process" quite hints at the frequent use of volatile where fences and other synchronization primitives are needed. (Note: I didn't down-vote anything)
    - rdtsc 11y ago
      Sorry assumed it was you. It was the only comment and a downvote. Shared memory and processes is a different beast than just threads in same process. Memory was mmap-ed between the two OS procs. Didn't use locks in that part of code for a list of reasons..., and memory write order didn't matter as much (so didn't see an immediate need for memory fences, but someone below suggested those might work to fix compiler optimizing out accesses when nor in debug mode). Not sure what hinted at frequent use of volatile, it was the only use in 10 years. That is what made it memorable.
      - Avernar 11y ago
        Wasn't me either. Can't downvote yet. :) Did some more research. Compilers have their own intrinsics that only affect the code generated (in addition to the processor ones). VS has _ReadBarrier, _WriteBarrier and _ReadWriteBarrier while GCC has asm volatile("" ::: "memory"); to prevent reordering optimizations. There really isn't any difference between multithread with common processes memory and multiprocess with mmaped shared memory. Both have mutliple threads accessing the same physical memory but in one case the OS made the memory common and the other you made the memory common. BTW, when I say multithreaded I'm talking real threads and not light/green/cooperative/fake/whatever threads.
        rdtsc 11y ago
        Thanks for doing more research. Did you get the idea there are two separate issues at hand -- 1) compiler optimizes memory access away because to its logic it seems the value is not written anywhere in that process. Say it could use a register instead if memory read. This is the effect I was seeing. 2) Compiler inserts the memory read instruction, but depending on the processor, at runtime, some of the writes will not show up at all, or be reorded. In this case running program on different CPU might make it appear to work correctly. I think there is a slight difference between the two from reading about back then. Can the first issue be fixed with a memory barrier?
        Avernar 11y ago
Avernar 11y ago
Volatile only affects how the compiler handles memory access. It does not tell the processor anything at all. The processor is still free to reorder reads and writes. You need to use the fence and lock instructions to create memory barriers especially on multi core processors.
- rdtsc 11y ago
  > Volatile only affects how the compiler handles memory access My comment specifically mentions memory access being optimized away. You did read it, right ;-) ? Even has an example how debug mode made it work. There was nothing there about telling something to the processor. If anything it is wanting to tell something to the compiler... > You need to use the fence and lock instructions to create memory barriers especially on multi core processors. Yap definitely. But will that still prevent the compiler from optimizing away stuff from a shared memory block? Remember looking for memory barriers but didn't know if my ancient system had those exposed in pthreads library.
  - loeg 11y ago
    Yep, any useful compiler support for fence/lock intrinsics will prevent optimizing out those intrinsics.
    - rdtsc 11y ago
      Are memory barriers transactional i.e. something like begin barrier(), do some shm writes, end_barrier? I was declaring data structures in shared mmap-ed memory as volatile so compiler does not optimise it away. How would the compiler know not to optimise away operation between memory fences? A fence to me say something about the guarantee of the order in which operations are seen. The order wasn't the problem, write never making it to the other side was the issue...
      - Avernar 11y ago
        There are many types of barriers depending on the processor architecture. They're usually used in pairs so for example a read barrier would be used when mutex is acquired and the a write barrier when it's released. If the compiler is smart about its memory barrier intrinsics it will make sure to reread anything after the read barrier and write everything before the write barrier. This will speed things up by not having to use volatile. Between the barriers it can cache reads and inside registers and optimize out multiple writes like normal. If the compiler can't do that then you need to use volatile. Volatile was meant for I/O where every read has to hit the bus. So if you're forced to use it then you just copy the volatile variable into a non volatile one for use inside your mutex to get some speed back.
burfog 11y ago
It's not for memory-mapped hardware. This is unsafe. You need something to flush the pipeline, prevent store merging, prevent caching, and more. It's nearly banned from the Linux kernel. It gets used in one place, for the clock tick counter. There is one other good use: suppression of compiler bugs.
- gizmo686 11y ago
  Linux's reasoning is documented here [0]. The justification for not using it for memory-mapped IO is that you are supposed to use accessor functions. Volotile still has uses for implementing those accesor functions (with care taken for the other concerns; and, by there nature, these functions are architecture dependent anyway), or when you are not using a kernel. [0] https://www.kernel.org/doc/Documentation/volatile-considered-harmful.txt https://www.kernel.org/doc/Documentation/volatile-considered...
- rdtsc 11y ago
  It was memory used for a low latency system, didn't have locks even spinlocks. In that case it was manipulating data directly in shm and compiler was optimizating away access to data. What is the current/modern mechanism to ensure a stucture or flag won't be optimized away from shared memory mapped between two processes? Are access functions available in userspace libraries?
- Avernar 11y ago
  Volatile is for memory mapped IO. Some processors may need extra instuctions or configuration but that doesn't replace the need for volatile with MMIO. On x86 the MTTR/PAT mechanisms is what disables caching, write combining and enables write through to MMIO address ranges.
  - burfog 11y ago
    Yes it does replace the need for volatile. The actual access is done via assembly language, either inline (better) or not. You could make volatile work, probably, by putting inline assembly both before and after it. In that case you could probably even skip marking things valatile, because the inline assembly will make the compiler behave. Why bother? This is more complicated than just doing the access from assembly. MTRR/PAT is often not enough, or it would only be enough if you killed performance elsewhere. (different types of MMIO in the same page) It's also not something you can rely on if you want to write portable code.

deleted 11y ago

[deleted]

caf 11y ago

The example saying: This means that the compiler is allowed to optimize volatile int *p = somevalue; int s = *p + *p; as if it said volatile int *p = somevalue; int tmp = *p; int s = tmp + tmp; is incorrect. The clause about sequence points starts with "Futhermore, " so it is an additional restriction, and to boot it only talks about stores to the volatile-qualified object, not loads, so it is entirely inapplicable to this example. The prior clause saying "Therefore any expression referring to such an object shall be evaluated strictly according to the rules of the abstract machine, as described in 5.1.2.3." applies, and the rules of the abstract machine for the binary + operator are that each operand is evaluated (in an unspecified order). So both dereferences must be evaluated.

vrfcodf 11y ago
I agree, I think this part makes it clear: except as modified by the unknown factors mentioned previously. Although this is quite contradicting: 5.1.2.3. p4: In the abstract machine, all expressions are evaluated as specified by the semantics. An actual implementation need not evaluate part of an expression if it can deduce that its value is not used and that no needed side effects are produced (including any caused by calling a function or accessing a volatile object)