12 ms·

Myths Programmers Believe about CPU Caches (2018)

176 points by whack 3y ago

deleted 3y ago
[deleted]
winstonprivacy 3y ago
Ok, interesting and all but just tell me how I can get 1 ns cross thread access to variables in C++
- gpderetta 3y ago
  I don't know the answer, but I'm sure it involves the use of some FTL solution.
- CyberDildonics 3y ago
  Ok, that's the latency of the L1 cache and that isn't shared between threads, so you might not want to count on being able to do this.
  - winstonprivacy 3y ago
    Exactly. I was referring to that bit at the end of the article: "In the case of Java volatiles, part of the solution is to force all reads/writes to bypass the local registers, and immediately trigger cache reads/writes instead. As soon as the data is read/written to the L1 cache, the hardware-coherency protocol takes over and provides guaranteed coherency across all global threads. Thus ensuring that if multiple threads are reading/writing to the same variable, they are all kept in sync with one another. And this is how you can achieve inter-thread coordination in as little as 1ns."
    - CyberDildonics 3y ago
      Ok, but why are you asking how to do it? They are saying how to do it, even though it's nonsense.
    - ComputerGuru 3y ago
      That last sentence can be pedantically true, if you look at in a very strict context. It doesn't say the variable can be returned in 1ns - but even if it did, it says the inter-thread coordination can take place in "as little as" 1ns. If a L1 cache immediately returning it's cached value for a memory address because it has the data in E or M state, that's technically a form of inter-thread coordination, just using already-coordinated state.
mannyv 3y ago
In Java, threads that are accessing non synchronized variables can read different information. It's in the jvm spec, and was a surprise back then. In C you could assume (at least in some runtimes) that you only need to sync on write.
celdon25 3y ago
2018
ninepoints 3y ago
Hmm, I think the discussion is missing a few things needed for a complete picture of the situation. First, ARM and x86 coherency models differ, so a big disclaimer is needed regarding the protocol. Most ARM processors use the MOESI protocol instead of the MESI protocol. Second, synchronization isn't just because of register volatility and such. Synchronization is needed in general because without the appropriate lock/barrier instructions, compilers make assumptions about how loads and stores may be reordered with respect to one another.
- chrismarlow9 3y ago
  Any recommendations for something more thorough to read or watch?
  - ninepoints 3y ago
    Honestly, the best resources I know are the ISA/architecture manuals directly from each vendor (Intel, AMD, Arm) for the various CPUs and/or GPUs of interest. Prior to that, my experience largely comes from doing (working with lower level code, benchmarks, looking at assembly, profiling hardware counters, etc.).
- monocasa 3y ago
  > First, ARM and x86 coherency models differ, so a big disclaimer is needed regarding the protocol. Most ARM processors use the MOESI protocol instead of the MESI protocol. MOESI is called out. > The above are just some of the possible scenarios that can occur. In reality, there are numerous variations of the above design, and no 2 implementations are the same. For example, some designs have an O/F state. However that's not an ARM v x86 thing; AMD has at least previously used MOESI. > Second, synchronization isn't just because of register volatility and such. Synchronization is needed in general because without the appropriate lock/barrier instructions, compilers make assumptions about how loads and stores may be reordered with respect to one another. Compiler barriers are different than hardware memory barriers. There are reasons to have one, but not the other.
  - 3y ago
chasil 3y ago
SQLite database locks can be more quickly obtained and released if CPU affinity is set for the database processes, allowing all I/O activity to share the same cache(es). I have read (but cannot remember where) that this can increase performance by thousands of DML operations per second.
- ComputerGuru 3y ago
  This is really going to be hardware- and OS-dependent. Most schedulers try to avoid changing CPU affinity for active processes, so you're likely looking at the delta for the worst-case scenario rather than the average case. But yes, affinity definitely helps!
ggm 3y ago
My favourite myth is that I still believe it's possible to write code which operates totally out of L1 and L2 cache. Not just tight ASM on bare metal: C code or similar, compiled down, to run under a modern UNIX/POSIX os on a multi-core host. I have never explored HOW this would work, or WHAT I would do to achieve it, but I believe it, implicitly. For it to be true I would have to understand the implications of every function and procedure call, stack operations, actual size of objects/structs under alignment, optimisations in the assembler to align or select code which runs faster in the given ALU, none of which I know (if I ever did, certainly not any more) But still: I believe this myth. I believe I know somebody who has achieved it, to test some ideas about CPU's ability to saturate a NIC at wire-rate, with pre-formed packets. He was able to show by binding affinity to specific cores and running this code, he could basically flood any link his CPU was capable of being exposed to, for a given PCI generation of NIC speeds available to him. (as I understand it) -But that assumes that walking off L2 cache would somehow make it run SLOW enough, to not be able to do this. So I think it remains a myth, to me.
- harry8 3y ago
  I guess that means my favourite myth that I've done it. Get the os out of the way, cpu pinning, user level network stack, no system calls after setup, lay out all your memory carefully in a cache line aware fashion noting the virtual addresses & bits 6-10. io via shared memory to another core on the same package. I convinced myself looking at latency benchmarks as I went about optimising it that it was all in L1.
- SomeHacker44 3y ago
  ISTR that is how a modern CPU will boot, before the DRAM interfaces have been calibrated and brought up. The initial startup code uses cache as RAM.
- ComputerGuru 3y ago
  Some SoCs will let you directly/explicitly use the cache as (ostensibly secondary) RAM, remapping your configuration from [cpu] -- [cache] -- ram to [ cpu ] | | [cache] [ram] Here's TI's documentation on doing just that CC13x2/CC26x2 family of MCUs: [0]. In this case, it's a seemingly paltry 8K of "RAM" but that's a lot for microcontroller developers! In this configuration and for this particular MCU, the CPU will actually run at a reduced speed (60% of its norm), probably because it needs to synchronize directly against the RAM. If you were to use it in CaR-only mode (which I don't think TI exposes), i.e. with the cache being used instead of rather than in addition to any RAM, you probably wouldn't need to run at those reduced speeds (and might even be able to run at higher clock speeds!). [0]: https://software-dl.ti.com/simplelink/esd/simplelink_cc13x2_26x2_sdk/3.30.00.03/exports/docs/proprietary-rf/proprietary-rf-users-guide/proprietary-rf-guide/cache-as-ram.html https://software-dl.ti.com/simplelink/esd/simplelink_cc13x2_...
Dylan16807 3y ago
This article gives the impression that everything is the compiler's fault when you end up with conflicting reads in different cores, but that's not right. From the point of view of someone outside the CPU, yes you can say that simultaneous reads might never give different answers. But everything happening at that level barely resembles the original software. Dozens of instructions are happening at any moment, overlapping each other, starting and finishing in very different orders from how they're stored in memory. This happens even if you directly write machine code byte by byte. From the point of view of the software, running a bunch of instructions in sequence, you do get stale values. You can have two threads wait for a signal, then both read a value, and both get different results. You can have a thread set a flag after it's done editing some values, have another thread wait for the flag, and then after waiting it sees the edits as still incomplete. The exact types of nonsense depend on the memory model, but things as simple as two MOV instructions in a row can violate strict ordering. It's not just that the compiler might stash something in a register, but that the CPU itself will make swaths of very observable changes within the rules set by the memory model. You can't trust the hardware coherency protocol to do much for you until you follow the platform-specific rules to tell the CPU to make something act coherent.
- zerohp 3y ago
  The author has a good grasp of coherence but either ignores or fails to understand memory ordering.
  - tylerhou 3y ago
    Where did you get this impression? That’s not how I understood the article; I thought it was good.
  - rossjudson 3y ago
    Uh, that's uncalled for. I think your hint is right there in the title: Myths Programmers Believe about CPU Caches. Was there an inaccuracy in the article with respect to CPU Caches?
- hinkley 3y ago
ChrisMarshallNY 3y ago
We worked with the Intel guys, in my last gig. They were incredibly helpful. They were an impressive lot, and they helped us out, quite a bit. They often sent engineers over, for weeks at a time, to help us optimize. The cache thing was a 100X improvement thing, and it came from the oddest places. There's a lot of "that doesn't make sense!" stuff, with preserving caches. I don't remember all the tricks, but we were constantly surprised. One thing that saved us, was instrumentation. Intel had a bunch of utilities that they wrote, and that kept showing us that the clever thing we did, was not so clever.
- tayo42 3y ago
  What kind of work were you doing that's required that? Like domain of programing? Sounds interesting >that kept showing us that the clever thing we did, was not so clever. I had the same experience talking with some people network people from Intel. Was pretty funny. To any Intel lurkers here, nice job! Lol
  - ChrisMarshallNY 3y ago
    We wrote image processing pipeline code. Very complex algorithms, on lots of data, that needed to be done quickly.
- automatic6131 3y ago
  It seems Intel is full of great engineers. I wonder why their latest products have been behind the competition for the last 5-6 years?
  - layer8 3y ago
    They are still the king of single-thread performance.
notacoward 3y ago
Dealing with caches, memory ordering, and memory barriers can be truly mind-warping stuff, even for those who have spent years dealing with basic cache coherency before. If you want a challenge, try to absorb all this in one sitting. https://www.kernel.org/doc/Documentation/memory-barriers.txt https://www.kernel.org/doc/Documentation/memory-barriers.txt I kept an earlier version of this close to hand at all times a couple of jobs ago where we were using our own chips with a very weak memory ordering. The implementation team had mostly come from Alpha, which had the weakest memory ordering ever, and in the intervening years a lot of bugs related to missing memory barriers had crept into the kernel because nobody was using anything nearly as weak. I specifically remember at least one in NBD, at least one in NFS, and many in Lustre. Pain in the ass to debug, because by the time you can look at anything the values have "settled" and seem correct. For extra fun, as weak as the memory ordering was, the first run of chips didn't even get that right. LL/SC wouldn't work reliably unless the LL was issued twice, so we actually modified compilers to do that. Ew.
- jcranmer 3y ago
  You don't need that full documentation. Really, I could simplify what most programmers would need to understand about memory ordering down to this text: There are a few models of cross-thread memory ordering that you can choose between. If you have never been exposed to this field before, the naïve model of memory ordering you probably think is going on is sequential consistency. This is not implemented in hardware because oh-gosh-it's-expensive, and if there's no indication of what memory ordering model your library or language is using, it's likely defaulting to sequential consistency. But don't worry, you don't have to worry about the more complex memory orderings, because you get to pretend everything is sequentially consistent if you write proper synchronization. The easiest way to satisfy proper synchronization is an acquire-release model. Before reading any data that may have been written by a different thread, you need to do an acquire load. After writing any data that may be read by a different thread, you need to do a release store. The basic flow is write then release store, then change thread, then acquire load, then read. Follow this rules, and things stay simple. There is a theoretical slight relaxation of the above model that works on most hardware called release-consume in the C/C++ memory model. It isn't implemented by any compiler for arcane compiler reasons I won't get into, but the Linux kernel (which implements the memory model itself for $REASONS) does rely heavily on it. The final major memory ordering is relaxed atomics. Their semantics are... weird. Essentially, you get what the hardware gives you, plus whatever fun the compiler can toss in, and the guarantees are minimal. I can't recommend many cases where you can safely use it, and if you have to ask if you should use it, the answer is no.
dmagda 3y ago
Language-specific synchronization primitives such as mutaxes or volatile variable use memory barries to ensure the caches are flushed and/or a CPU core reads from the main memory directly: https://en.wikipedia.org/wiki/Memory_barrier https://en.wikipedia.org/wiki/Memory_barrier
101011 3y ago
Loved the article, appreciate the share. A meta topic but I wish people would spend more time discussing what they like about something than the opposite. This author took great pains to give many caveats around their writing and distilled down a very complex topic into something that could be consumed in less than 10 minutes. I'm smarter for having read this. Best of luck to any poor soul who attempts to summarize the work of many architects and then share it with this group of people.
garfieldandthe 3y ago
The central myth is that the average programmer should care. The typical programmer should treat CPU caches as what they are designed to be: mostly transparent. You work in a high level language and leave the tricky details to a library and your compiler. It's only a small minority that should really worry about these things. In my daily work, I see more often premature microoptimizations (in part using the myths from the article) which are entirely unnecessary rather than code that needs to optimize for those things.
- pajko 3y ago
  This is partly wrong. For example, it's important how the fields are organized in a structure, and only the developer knows what belongs together and only they know the access patterns across threads (or at least should know). This is far from being a micro-optimization.
  - garfieldandthe 3y ago
    How many of your structs does your code access million times a second? For most programs, that number is firmly 0. While you are optimizing your struct layout and decrease you app start time from 1.122765 seconds to 1.122764 seconds, I ship production code at 2x the rate because I only optimize for performance where it matters while otherwise optimizing for maintainability, testability, extensibility, dev fun and actually shipping.
  - brabel 3y ago
    If I do it wrong, are we talking about my application (say, some web app) becoming noticeably slower? Or we would need to be running several million operations per second before anything a user could notice the difference? I suspect only very high end games, image processing apps and that kind of thing would ever need to care about the order of fields in a struct... perhaps the author of my web server as well, but even that only if I have a ridiculously high load on my server (which I don't, almost no one does). So, no I don't think I should know that until I develop one of those highly niche applications myself.
anonymousDan 3y ago
On the related topic of store buffers and write coalescing, can anyone point me to some good articles discussing how this works in practice for merging small writes to the same cache line? I have a workload where we are appending different size payloads (of sizes between 1 and 64 bytes) to an in memory log. One option is to just store each entry in its own cache line. A more space efficient option is to pack each write tightly one after the other. My hypothesis is that due to write coalescing in store buffers this will also be more memory bandwidth efficient since writes to the same cache line will be merged. Is this correct? Note that metadata for each entry (e.g. the size) will be stored separately.
- gpderetta 3y ago
  On a typical write back cache, the cache line is written back to memory (or to a lower hierarchy) only when needed: either some other core wants it or it needs to be evicted to make space for other data. For a write mostly task, if you write to the same line again, it is likely to be still in the M state and data can be written to it immediately. Later, when the line is written back, the separate writes are effectively coalesced. But if some other core is concurrently reading from the head of the log, it can be suboptimal as the cacheline will keep bouncing between M and S state generating coherence traffic. This is still more scalable than two threads writing to the same cacheline of course.

176 points by whack 3y ago

deleted 3y ago

[deleted]

winstonprivacy 3y ago

Ok, interesting and all but just tell me how I can get 1 ns cross thread access to variables in C++

gpderetta 3y ago
I don't know the answer, but I'm sure it involves the use of some FTL solution.
CyberDildonics 3y ago
Ok, that's the latency of the L1 cache and that isn't shared between threads, so you might not want to count on being able to do this.
- winstonprivacy 3y ago
  Exactly. I was referring to that bit at the end of the article: "In the case of Java volatiles, part of the solution is to force all reads/writes to bypass the local registers, and immediately trigger cache reads/writes instead. As soon as the data is read/written to the L1 cache, the hardware-coherency protocol takes over and provides guaranteed coherency across all global threads. Thus ensuring that if multiple threads are reading/writing to the same variable, they are all kept in sync with one another. And this is how you can achieve inter-thread coordination in as little as 1ns."
  - CyberDildonics 3y ago
    Ok, but why are you asking how to do it? They are saying how to do it, even though it's nonsense.
  - ComputerGuru 3y ago
    That last sentence can be pedantically true, if you look at in a very strict context. It doesn't say the variable can be returned in 1ns - but even if it did, it says the inter-thread coordination can take place in "as little as" 1ns. If a L1 cache immediately returning it's cached value for a memory address because it has the data in E or M state, that's technically a form of inter-thread coordination, just using already-coordinated state.

mannyv 3y ago

In Java, threads that are accessing non synchronized variables can read different information. It's in the jvm spec, and was a surprise back then. In C you could assume (at least in some runtimes) that you only need to sync on write.

celdon25 3y ago

2018

ninepoints 3y ago

Hmm, I think the discussion is missing a few things needed for a complete picture of the situation. First, ARM and x86 coherency models differ, so a big disclaimer is needed regarding the protocol. Most ARM processors use the MOESI protocol instead of the MESI protocol. Second, synchronization isn't just because of register volatility and such. Synchronization is needed in general because without the appropriate lock/barrier instructions, compilers make assumptions about how loads and stores may be reordered with respect to one another.

chrismarlow9 3y ago
Any recommendations for something more thorough to read or watch?
- ninepoints 3y ago
  Honestly, the best resources I know are the ISA/architecture manuals directly from each vendor (Intel, AMD, Arm) for the various CPUs and/or GPUs of interest. Prior to that, my experience largely comes from doing (working with lower level code, benchmarks, looking at assembly, profiling hardware counters, etc.).
monocasa 3y ago
> First, ARM and x86 coherency models differ, so a big disclaimer is needed regarding the protocol. Most ARM processors use the MOESI protocol instead of the MESI protocol. MOESI is called out. > The above are just some of the possible scenarios that can occur. In reality, there are numerous variations of the above design, and no 2 implementations are the same. For example, some designs have an O/F state. However that's not an ARM v x86 thing; AMD has at least previously used MOESI. > Second, synchronization isn't just because of register volatility and such. Synchronization is needed in general because without the appropriate lock/barrier instructions, compilers make assumptions about how loads and stores may be reordered with respect to one another. Compiler barriers are different than hardware memory barriers. There are reasons to have one, but not the other.
- 3y ago

chasil 3y ago

SQLite database locks can be more quickly obtained and released if CPU affinity is set for the database processes, allowing all I/O activity to share the same cache(es). I have read (but cannot remember where) that this can increase performance by thousands of DML operations per second.

ComputerGuru 3y ago
This is really going to be hardware- and OS-dependent. Most schedulers try to avoid changing CPU affinity for active processes, so you're likely looking at the delta for the worst-case scenario rather than the average case. But yes, affinity definitely helps!

ggm 3y ago

My favourite myth is that I still believe it's possible to write code which operates totally out of L1 and L2 cache. Not just tight ASM on bare metal: C code or similar, compiled down, to run under a modern UNIX/POSIX os on a multi-core host. I have never explored HOW this would work, or WHAT I would do to achieve it, but I believe it, implicitly. For it to be true I would have to understand the implications of every function and procedure call, stack operations, actual size of objects/structs under alignment, optimisations in the assembler to align or select code which runs faster in the given ALU, none of which I know (if I ever did, certainly not any more) But still: I believe this myth. I believe I know somebody who has achieved it, to test some ideas about CPU's ability to saturate a NIC at wire-rate, with pre-formed packets. He was able to show by binding affinity to specific cores and running this code, he could basically flood any link his CPU was capable of being exposed to, for a given PCI generation of NIC speeds available to him. (as I understand it) -But that assumes that walking off L2 cache would somehow make it run SLOW enough, to not be able to do this. So I think it remains a myth, to me.

harry8 3y ago
I guess that means my favourite myth that I've done it. Get the os out of the way, cpu pinning, user level network stack, no system calls after setup, lay out all your memory carefully in a cache line aware fashion noting the virtual addresses & bits 6-10. io via shared memory to another core on the same package. I convinced myself looking at latency benchmarks as I went about optimising it that it was all in L1.
SomeHacker44 3y ago
ISTR that is how a modern CPU will boot, before the DRAM interfaces have been calibrated and brought up. The initial startup code uses cache as RAM.
ComputerGuru 3y ago
Some SoCs will let you directly/explicitly use the cache as (ostensibly secondary) RAM, remapping your configuration from [cpu] -- [cache] -- ram to [ cpu ] | | [cache] [ram] Here's TI's documentation on doing just that CC13x2/CC26x2 family of MCUs: [0]. In this case, it's a seemingly paltry 8K of "RAM" but that's a lot for microcontroller developers! In this configuration and for this particular MCU, the CPU will actually run at a reduced speed (60% of its norm), probably because it needs to synchronize directly against the RAM. If you were to use it in CaR-only mode (which I don't think TI exposes), i.e. with the cache being used instead of rather than in addition to any RAM, you probably wouldn't need to run at those reduced speeds (and might even be able to run at higher clock speeds!). [0]: https://software-dl.ti.com/simplelink/esd/simplelink_cc13x2_26x2_sdk/3.30.00.03/exports/docs/proprietary-rf/proprietary-rf-users-guide/proprietary-rf-guide/cache-as-ram.html https://software-dl.ti.com/simplelink/esd/simplelink_cc13x2_...

Dylan16807 3y ago

This article gives the impression that everything is the compiler's fault when you end up with conflicting reads in different cores, but that's not right. From the point of view of someone outside the CPU, yes you can say that simultaneous reads might never give different answers. But everything happening at that level barely resembles the original software. Dozens of instructions are happening at any moment, overlapping each other, starting and finishing in very different orders from how they're stored in memory. This happens even if you directly write machine code byte by byte. From the point of view of the software, running a bunch of instructions in sequence, you do get stale values. You can have two threads wait for a signal, then both read a value, and both get different results. You can have a thread set a flag after it's done editing some values, have another thread wait for the flag, and then after waiting it sees the edits as still incomplete. The exact types of nonsense depend on the memory model, but things as simple as two MOV instructions in a row can violate strict ordering. It's not just that the compiler might stash something in a register, but that the CPU itself will make swaths of very observable changes within the rules set by the memory model. You can't trust the hardware coherency protocol to do much for you until you follow the platform-specific rules to tell the CPU to make something act coherent.

zerohp 3y ago
The author has a good grasp of coherence but either ignores or fails to understand memory ordering.
- tylerhou 3y ago
  Where did you get this impression? That’s not how I understood the article; I thought it was good.
- rossjudson 3y ago
  Uh, that's uncalled for. I think your hint is right there in the title: Myths Programmers Believe about CPU Caches. Was there an inaccuracy in the article with respect to CPU Caches?
hinkley 3y ago

ChrisMarshallNY 3y ago

We worked with the Intel guys, in my last gig. They were incredibly helpful. They were an impressive lot, and they helped us out, quite a bit. They often sent engineers over, for weeks at a time, to help us optimize. The cache thing was a 100X improvement thing, and it came from the oddest places. There's a lot of "that doesn't make sense!" stuff, with preserving caches. I don't remember all the tricks, but we were constantly surprised. One thing that saved us, was instrumentation. Intel had a bunch of utilities that they wrote, and that kept showing us that the clever thing we did, was not so clever.

tayo42 3y ago
What kind of work were you doing that's required that? Like domain of programing? Sounds interesting >that kept showing us that the clever thing we did, was not so clever. I had the same experience talking with some people network people from Intel. Was pretty funny. To any Intel lurkers here, nice job! Lol
- ChrisMarshallNY 3y ago
  We wrote image processing pipeline code. Very complex algorithms, on lots of data, that needed to be done quickly.
automatic6131 3y ago
It seems Intel is full of great engineers. I wonder why their latest products have been behind the competition for the last 5-6 years?
- layer8 3y ago
  They are still the king of single-thread performance.

notacoward 3y ago

Dealing with caches, memory ordering, and memory barriers can be truly mind-warping stuff, even for those who have spent years dealing with basic cache coherency before. If you want a challenge, try to absorb all this in one sitting. https://www.kernel.org/doc/Documentation/memory-barriers.txt https://www.kernel.org/doc/Documentation/memory-barriers.txt I kept an earlier version of this close to hand at all times a couple of jobs ago where we were using our own chips with a very weak memory ordering. The implementation team had mostly come from Alpha, which had the weakest memory ordering ever, and in the intervening years a lot of bugs related to missing memory barriers had crept into the kernel because nobody was using anything nearly as weak. I specifically remember at least one in NBD, at least one in NFS, and many in Lustre. Pain in the ass to debug, because by the time you can look at anything the values have "settled" and seem correct. For extra fun, as weak as the memory ordering was, the first run of chips didn't even get that right. LL/SC wouldn't work reliably unless the LL was issued twice, so we actually modified compilers to do that. Ew.

jcranmer 3y ago
You don't need that full documentation. Really, I could simplify what most programmers would need to understand about memory ordering down to this text: There are a few models of cross-thread memory ordering that you can choose between. If you have never been exposed to this field before, the naïve model of memory ordering you probably think is going on is sequential consistency. This is not implemented in hardware because oh-gosh-it's-expensive, and if there's no indication of what memory ordering model your library or language is using, it's likely defaulting to sequential consistency. But don't worry, you don't have to worry about the more complex memory orderings, because you get to pretend everything is sequentially consistent if you write proper synchronization. The easiest way to satisfy proper synchronization is an acquire-release model. Before reading any data that may have been written by a different thread, you need to do an acquire load. After writing any data that may be read by a different thread, you need to do a release store. The basic flow is write then release store, then change thread, then acquire load, then read. Follow this rules, and things stay simple. There is a theoretical slight relaxation of the above model that works on most hardware called release-consume in the C/C++ memory model. It isn't implemented by any compiler for arcane compiler reasons I won't get into, but the Linux kernel (which implements the memory model itself for $REASONS) does rely heavily on it. The final major memory ordering is relaxed atomics. Their semantics are... weird. Essentially, you get what the hardware gives you, plus whatever fun the compiler can toss in, and the guarantees are minimal. I can't recommend many cases where you can safely use it, and if you have to ask if you should use it, the answer is no.

dmagda 3y ago

Language-specific synchronization primitives such as mutaxes or volatile variable use memory barries to ensure the caches are flushed and/or a CPU core reads from the main memory directly: https://en.wikipedia.org/wiki/Memory_barrier https://en.wikipedia.org/wiki/Memory_barrier

101011 3y ago

Loved the article, appreciate the share. A meta topic but I wish people would spend more time discussing what they like about something than the opposite. This author took great pains to give many caveats around their writing and distilled down a very complex topic into something that could be consumed in less than 10 minutes. I'm smarter for having read this. Best of luck to any poor soul who attempts to summarize the work of many architects and then share it with this group of people.

garfieldandthe 3y ago

The central myth is that the average programmer should care. The typical programmer should treat CPU caches as what they are designed to be: mostly transparent. You work in a high level language and leave the tricky details to a library and your compiler. It's only a small minority that should really worry about these things. In my daily work, I see more often premature microoptimizations (in part using the myths from the article) which are entirely unnecessary rather than code that needs to optimize for those things.

pajko 3y ago
This is partly wrong. For example, it's important how the fields are organized in a structure, and only the developer knows what belongs together and only they know the access patterns across threads (or at least should know). This is far from being a micro-optimization.
- garfieldandthe 3y ago
  How many of your structs does your code access million times a second? For most programs, that number is firmly 0. While you are optimizing your struct layout and decrease you app start time from 1.122765 seconds to 1.122764 seconds, I ship production code at 2x the rate because I only optimize for performance where it matters while otherwise optimizing for maintainability, testability, extensibility, dev fun and actually shipping.
- brabel 3y ago
  If I do it wrong, are we talking about my application (say, some web app) becoming noticeably slower? Or we would need to be running several million operations per second before anything a user could notice the difference? I suspect only very high end games, image processing apps and that kind of thing would ever need to care about the order of fields in a struct... perhaps the author of my web server as well, but even that only if I have a ridiculously high load on my server (which I don't, almost no one does). So, no I don't think I should know that until I develop one of those highly niche applications myself.

anonymousDan 3y ago

On the related topic of store buffers and write coalescing, can anyone point me to some good articles discussing how this works in practice for merging small writes to the same cache line? I have a workload where we are appending different size payloads (of sizes between 1 and 64 bytes) to an in memory log. One option is to just store each entry in its own cache line. A more space efficient option is to pack each write tightly one after the other. My hypothesis is that due to write coalescing in store buffers this will also be more memory bandwidth efficient since writes to the same cache line will be merged. Is this correct? Note that metadata for each entry (e.g. the size) will be stored separately.

gpderetta 3y ago
On a typical write back cache, the cache line is written back to memory (or to a lower hierarchy) only when needed: either some other core wants it or it needs to be evicted to make space for other data. For a write mostly task, if you write to the same line again, it is likely to be still in the M state and data can be written to it immediately. Later, when the line is written back, the separate writes are effectively coalesced. But if some other core is concurrently reading from the head of the log, it can be suboptimal as the cacheline will keep bouncing between M and S state generating coherence traffic. This is still more scalable than two threads writing to the same cacheline of course.