6 ms·

> It can be helpful to know that, on all modern processors, you can safety assign a value to a int, double, float, bool, BOOL or pointer variable on one thread,

by exDM69 10y ago

> It can be helpful to know that, on all modern processors, you can safety assign a value to a int, double, float, bool, BOOL or pointer variable on one thread, and read it on a different thread without worrying about tearing This is true on the CPU side (for some CPUs) but what about compiler optimizations? Using "volatile" should make stores and loads happen where they are written in the code, but can it be relied on in multi threaded use cases? It's generally frowned upon for a good reason, but perhaps it's acceptable if you're targetting a limited range of CPUs (the article seems to be focused on iOS and ARM only). A safer bet would be to use __atomic_load and __atomic_store (from [0] or [1]) or C11 atomics if you have the appropriate headers for your platform. They provide safe loads and stores for all CPU architectures, and provide the appropriate memory barriers for cache coherency (for architectures that do care). [0] https://gcc.gnu.org/onlinedocs/gcc/_005f_005fatomic-Builtins.html#g_t_005f_005fatomic-Builtins https://gcc.gnu.org/onlinedocs/gcc/_005f_005fatomic-Builtins... [1] http://clang.llvm.org/docs/LanguageExtensions.html#langext-c11-atomic http://clang.llvm.org/docs/LanguageExtensions.html#langext-c...

Bromskloss 10y ago
Do you mean that a compiler optimisation could cause tearing of the mentioned data types? That's surprising to me and I'd like to know more about it.
- _pmf_ 10y ago
  I think the extreme case would be a packed struct where a multibyte member happens to span cache lines. This would not be causedby compiler optimization, but by manual deoptimization.
- biokoda 10y ago
  Not tearing, but it could reorder the assignment so that it happens someplace else than you think it will. For this kind of thread synchronization memory barriers should be used at least.
  - exDM69 10y ago
    Exactly. To give a concrete example (but artificial) what might happen, consider this case: global_flag = 0; for(this loop takes a long time) { ... } global_flag = 1; for(this loop takes a long time) { ... } global_flag = 0; If it is trying to communicate something to another thread (or audio callback) with global_flag, it might fail miserably because the compiler is free to drop two of the assignments and place `global_flag = 0` wherever in the code. Using `volatile` should prevent this compiler optimization but it doesn't give any guarantees about cache coherency, etc. As far as I know, using atomic load and store is the only way (apart from locks) to guarantee correctness and portability.
    - Bromskloss 10y ago
      I thought _tearing_ referred to a value being read while in the midst of being updated, so that, for example, some of the read bits comes from the old value and some from the new value.
      - barrkel 10y ago
        It does. It's a distinct issue and still very relevant if you're using shared memory without locks to communicate between threads.
    - cesarb 10y ago
      Elaborating on your example. my_buffer->something = 42; global_pointer = my_buffer; global_flag = 1; Even if everything is declared `volatile`, in C or C++, the compiler or the processor can reorder the stores and reads, so that the reading thread can see the "global_pointer = my_buffer" before the "->something = 42". The only safe way to do it is to add the appropriate memory barriers on both the writing and reading sides, which force the compiler and the processor to not reorder the writes/reads. my_buffer->something = 42; write_barrier(); // Not the actual function; will vary depending on your environment. global_pointer = my_buffer; write_barrier(); // Not the actual function; will vary depending on your environment. global_flag = 1; And on the reading side, the corresponding read barriers. It's simpler to use atomic loads and stores or locks, since their implementation already has the required barriers with all the details (quick: what's the difference between an acquire and a release atomic access?). (Note that this is different in Java, where `volatile` always implies a memory barrier.)
dmm 10y ago
If you're doing this in assembly, no problem so long as you understand the memory model of the cpu architecture you're targeting. The memory model implemented by c compilers absolutely do not allow this. The problem is not cache coherency but optimization. The compiler assumes it understands the visibility of the variables and reorders, combines, and eliminates reads and writes. C is not a high-level assembler. > A safer bet would be to use __atomic_load ... C11 atomics These should work but be careful. Use the semantics defined by these constructs not the semantics of any assembly you might imagine the compiler generating.
- planteen 10y ago
  Sure, but isn't your assembley code going to look a lot like C code with atomics anyway as you use barrier or in order execution instructions (e.g., dmb on ARM or eieio on PPC)?
  - deleted 10y ago
    [deleted]
biokoda 10y ago
> A safer bet would be to use __atomic_load and __atomic_store Safer and an order of magnitude slower.
- exDM69 10y ago
  Sure, but the only way to be sure that a non-atomic store works as intended is to read the assembly output from the compiler. It's also not very easy to predict the relative difference because it depends on which core(s) the threads get scheduled on. If both threads happen to get on the same core and the data stays in caches, we're talking about 1ns (L1) to 5ns (L2). An atomic load/store would be about 20ns (4..20x slower). If they happen to get scheduled on different cores, the CPU interconnect will have to deal with its cache coherency protocol (similar to atomic load/store) and the result will be around the same. We're talking about pretty miniscule amount of time here. It's a choice between fast if you're lucky but possibly incorrect anyway or marginally slower but guaranteed correctness.
  - Unklejoe 10y ago
    Wouldn't this be the case anyway (cache coherency overhead) even if we used types that just-so-happened to be atomic on the platform, such as "int"? In other words, isn't this a non issue when comparing the performance of atomic types and "regular" types, as they both would require this? Also, do the atomic types handle ordering? I could be going crazy, but I remember seeing some of those atomic types simply being typedef'd to "int" in some OS code before. EDIT: I was thinking of the "atomic_t, atomic_inc, atomic_set, etc..." in the Linux Kernel. It guarantees atomic behavior, but not ordering. Memory barriers are still required when using it between different threads. The C11 types appear to be different, as they accept an ordering argument in their load/store functions.
    - anarazel 10y ago
      Many of them actually do provide guarantees (as e.g. evidenced that a typical implementation of an x86 memory barrier is an "lock; addl $0,0(%%rsp)"). That's why there's now stuff like __smp_mb__before_atomic()/__smp_mb__after_atomic(), to avoid unnecessary atomic ops.
Unklejoe 10y ago
> [Using "volatile" should make stores and loads happen where they are written in the code...] Really? I was under the impression that "volatile" basically did nothing other than prevent the compiler from optimizing out a variable that appears to not ever be written to, but is actually written to via an ISR or something. Also, how does the compiler usually handle global variables? Since they can be modified by some code that was linked in, does it assume them to be volatile? Of course, this issue is slightly different than the atomic-ness of certain data types. We're talking more about situations where you have a series of atomic operations which you expect to happen in the order of which they're written.
- exDM69 10y ago
  > Really? I was under the impression that "volatile" basically did nothing other than prevent the compiler from optimizing out a variable that appears to not ever be written to, but is actually written to via an ISR or something. I think it also stops loads and stores to be reordered with other volatile loads and stores, and yes, it indeed is typically useful for memory mapped i/o, interrupt handlers and such. But most of the time it's not the right thing to do. > Also, how does the compiler usually handle global variables? Since they can be modified by some code that was linked in, does it assume them to be volatile? No, they're not considered volatile but may have stricter guarantees with reordering than local variables (esp. when combined with calls to foreign functions). If multithreaded code needs to be correct and portable, using locks or atomics is the way to go.
  - vardump 10y ago
    > I think it also stops loads and stores to be reordered with other volatile loads and stores, and yes, it indeed is typically useful for memory mapped i/o, interrupt handlers and such. But most of the time it's not the right thing to do. It only prevents reordering by the compiler. CPU can still reorder them as much as it pleases, volatile doesn't affect it at all. Of course you're pretty safe on x86. Not so on other platforms. Typically you find this kind of issues when porting x86 code to some other platform. X86 writes are always in order with other writes. Loads are always in order to other loads. Just need to remember loads are sometimes ordered before stores. Again, volatile does not prevent CPU from reordering stores and loads.

by exDM69 10y ago

Bromskloss 10y ago

Do you mean that a compiler optimisation could cause tearing of the mentioned data types? That's surprising to me and I'd like to know more about it.

_pmf_ 10y ago
I think the extreme case would be a packed struct where a multibyte member happens to span cache lines. This would not be causedby compiler optimization, but by manual deoptimization.
biokoda 10y ago
Not tearing, but it could reorder the assignment so that it happens someplace else than you think it will. For this kind of thread synchronization memory barriers should be used at least.
- exDM69 10y ago
  Exactly. To give a concrete example (but artificial) what might happen, consider this case: global_flag = 0; for(this loop takes a long time) { ... } global_flag = 1; for(this loop takes a long time) { ... } global_flag = 0; If it is trying to communicate something to another thread (or audio callback) with global_flag, it might fail miserably because the compiler is free to drop two of the assignments and place `global_flag = 0` wherever in the code. Using `volatile` should prevent this compiler optimization but it doesn't give any guarantees about cache coherency, etc. As far as I know, using atomic load and store is the only way (apart from locks) to guarantee correctness and portability.
  - Bromskloss 10y ago
    I thought _tearing_ referred to a value being read while in the midst of being updated, so that, for example, some of the read bits comes from the old value and some from the new value.
    - barrkel 10y ago
      It does. It's a distinct issue and still very relevant if you're using shared memory without locks to communicate between threads.
  - cesarb 10y ago
    Elaborating on your example. my_buffer->something = 42; global_pointer = my_buffer; global_flag = 1; Even if everything is declared `volatile`, in C or C++, the compiler or the processor can reorder the stores and reads, so that the reading thread can see the "global_pointer = my_buffer" before the "->something = 42". The only safe way to do it is to add the appropriate memory barriers on both the writing and reading sides, which force the compiler and the processor to not reorder the writes/reads. my_buffer->something = 42; write_barrier(); // Not the actual function; will vary depending on your environment. global_pointer = my_buffer; write_barrier(); // Not the actual function; will vary depending on your environment. global_flag = 1; And on the reading side, the corresponding read barriers. It's simpler to use atomic loads and stores or locks, since their implementation already has the required barriers with all the details (quick: what's the difference between an acquire and a release atomic access?). (Note that this is different in Java, where `volatile` always implies a memory barrier.)

dmm 10y ago

If you're doing this in assembly, no problem so long as you understand the memory model of the cpu architecture you're targeting. The memory model implemented by c compilers absolutely do not allow this. The problem is not cache coherency but optimization. The compiler assumes it understands the visibility of the variables and reorders, combines, and eliminates reads and writes. C is not a high-level assembler. > A safer bet would be to use __atomic_load ... C11 atomics These should work but be careful. Use the semantics defined by these constructs not the semantics of any assembly you might imagine the compiler generating.

planteen 10y ago
Sure, but isn't your assembley code going to look a lot like C code with atomics anyway as you use barrier or in order execution instructions (e.g., dmb on ARM or eieio on PPC)?
- deleted 10y ago
  [deleted]

biokoda 10y ago

> A safer bet would be to use __atomic_load and __atomic_store Safer and an order of magnitude slower.

exDM69 10y ago
Sure, but the only way to be sure that a non-atomic store works as intended is to read the assembly output from the compiler. It's also not very easy to predict the relative difference because it depends on which core(s) the threads get scheduled on. If both threads happen to get on the same core and the data stays in caches, we're talking about 1ns (L1) to 5ns (L2). An atomic load/store would be about 20ns (4..20x slower). If they happen to get scheduled on different cores, the CPU interconnect will have to deal with its cache coherency protocol (similar to atomic load/store) and the result will be around the same. We're talking about pretty miniscule amount of time here. It's a choice between fast if you're lucky but possibly incorrect anyway or marginally slower but guaranteed correctness.
- Unklejoe 10y ago
  Wouldn't this be the case anyway (cache coherency overhead) even if we used types that just-so-happened to be atomic on the platform, such as "int"? In other words, isn't this a non issue when comparing the performance of atomic types and "regular" types, as they both would require this? Also, do the atomic types handle ordering? I could be going crazy, but I remember seeing some of those atomic types simply being typedef'd to "int" in some OS code before. EDIT: I was thinking of the "atomic_t, atomic_inc, atomic_set, etc..." in the Linux Kernel. It guarantees atomic behavior, but not ordering. Memory barriers are still required when using it between different threads. The C11 types appear to be different, as they accept an ordering argument in their load/store functions.
  - anarazel 10y ago
    Many of them actually do provide guarantees (as e.g. evidenced that a typical implementation of an x86 memory barrier is an "lock; addl $0,0(%%rsp)"). That's why there's now stuff like __smp_mb__before_atomic()/__smp_mb__after_atomic(), to avoid unnecessary atomic ops.

Unklejoe 10y ago

> [Using "volatile" should make stores and loads happen where they are written in the code...] Really? I was under the impression that "volatile" basically did nothing other than prevent the compiler from optimizing out a variable that appears to not ever be written to, but is actually written to via an ISR or something. Also, how does the compiler usually handle global variables? Since they can be modified by some code that was linked in, does it assume them to be volatile? Of course, this issue is slightly different than the atomic-ness of certain data types. We're talking more about situations where you have a series of atomic operations which you expect to happen in the order of which they're written.

exDM69 10y ago
> Really? I was under the impression that "volatile" basically did nothing other than prevent the compiler from optimizing out a variable that appears to not ever be written to, but is actually written to via an ISR or something. I think it also stops loads and stores to be reordered with other volatile loads and stores, and yes, it indeed is typically useful for memory mapped i/o, interrupt handlers and such. But most of the time it's not the right thing to do. > Also, how does the compiler usually handle global variables? Since they can be modified by some code that was linked in, does it assume them to be volatile? No, they're not considered volatile but may have stricter guarantees with reordering than local variables (esp. when combined with calls to foreign functions). If multithreaded code needs to be correct and portable, using locks or atomics is the way to go.
- vardump 10y ago
  > I think it also stops loads and stores to be reordered with other volatile loads and stores, and yes, it indeed is typically useful for memory mapped i/o, interrupt handlers and such. But most of the time it's not the right thing to do. It only prevents reordering by the compiler. CPU can still reorder them as much as it pleases, volatile doesn't affect it at all. Of course you're pretty safe on x86. Not so on other platforms. Typically you find this kind of issues when porting x86 code to some other platform. X86 writes are always in order with other writes. Loads are always in order to other loads. Just need to remember loads are sometimes ordered before stores. Again, volatile does not prevent CPU from reordering stores and loads.