2 ms·

About 30 years ago, I was porting an Ethernet driver to the MC88K, which also didn't allow misaligned accesses. Guess what happens when you fill a page-aligned

About 30 years ago, I was porting an Ethernet driver to the MC88K, which also didn't allow misaligned accesses. Guess what happens when you fill a page-aligned buffer with an Ethernet packet, go past the 14-byte Ethernet header, and try to read the first 4-byte value in the IP header as a single word? You guessed it: SIGBUS. Every time. The same problem existed many other places, so I had to go find all of them. Eventually I implemented an exception handler that would emulate the access, and also log an error message. For months afterward, this turned up ever more obscure cases until there seemed to be none left. Coming next: how modern programmers can't imagine a different word size or byte order. After that: different memory-ordering models. That one's really fun.

anticensor 7y ago
Mixed memory order as in, a memory write stores the data in an order like 5, 7, 6, 4 instead of 4, 5, 6, 7?
- notacoward 7y ago
  Basically yeah, but it gets much more complicated. For example, when a write hits memory is actually not the same thing as when it's observable by some particular other processor. There might be particular rules for writes that have dependencies too. Here's some good reading material if you have a spare afternoon. https://www.kernel.org/doc/Documentation/memory-barriers.txt https://www.kernel.org/doc/Documentation/memory-barriers.txt Amusingly, it treats dependency issues as obsolete, but I worked on a processor as late as 2009 that had all of the weak-memory-ordering issues of the Alpha (because it was the same design team). We uncovered lots of missing memory barriers in the Linux kernel. Good times.
  - loeg 7y ago
    The modern thing is using acquire/release semantics on the variables you actually care about intra-thread ordering on, rather than full barriers, which impose a higher cost.
    - notacoward 7y ago
      I think we're talking about different kinds of barriers. Acquire/release at the C++ level translates roughly into an rmb/wmb pair at the processor level, unless it's implemented using an actual interlocked instruction which is worse than barriers.
      - loeg 7y ago
        Nope. Acquire/release at C level translates into a compiler barrier and the exact machine implementation will depend on hardware. However, on x86, for example, with its strong memory semantics, ordinary stores already have release semantics and do not require an architectural barrier (wmb/sfence). No sfence, no lock prefix.
        notacoward 7y ago
        First you said that the modern way is to use acquire/release instead of a barrier, then admitted that it is a barrier at the compiler level. The fact that it doesn't require a barrier instruction on the x86 is pretty much irrelevant. Not all the world's an x86, and even the x86 hasn't always been the same in this regard. So your rude "nope" is unwarranted. This attitude of "as it is (in my world) now, so it has ever been and ever will be (everywhere)" is exactly why these bugs occur.
        loeg 7y ago
        Compiler barriers and machine barriers are completely different things, despite having a similar name. In about the era you described, the Linux kernel community did go around throwing rmb()s and wmb()s everywhere, and those did translate to hardware fences. (And in general, it is not safe to elide a hardware fence on x86 despite relatively strong memory ordering.) This is what I believed you were talking about; maybe I misunderstood: apologies. Putting explicit compiler barriers in code is still not quite the modern model for relaxed atomic consumers. You use the abstract acquire/release load/store pseudo-functions, and they do the right thing depending on implementation. The compiler barriers in the relaxed atomics model are an internal implementation detail, not the API. x86 is just an example where the cheapest way to do a release-semantics store is a plain store. I never said all the world was x86 — if it were, abstract acquire/release relaxed atomics would be kind of pointless.

anticensor 7y ago

Mixed memory order as in, a memory write stores the data in an order like 5, 7, 6, 4 instead of 4, 5, 6, 7?

notacoward 7y ago
Basically yeah, but it gets much more complicated. For example, when a write hits memory is actually not the same thing as when it's observable by some particular other processor. There might be particular rules for writes that have dependencies too. Here's some good reading material if you have a spare afternoon. https://www.kernel.org/doc/Documentation/memory-barriers.txt https://www.kernel.org/doc/Documentation/memory-barriers.txt Amusingly, it treats dependency issues as obsolete, but I worked on a processor as late as 2009 that had all of the weak-memory-ordering issues of the Alpha (because it was the same design team). We uncovered lots of missing memory barriers in the Linux kernel. Good times.
- loeg 7y ago
  The modern thing is using acquire/release semantics on the variables you actually care about intra-thread ordering on, rather than full barriers, which impose a higher cost.
  - notacoward 7y ago
    I think we're talking about different kinds of barriers. Acquire/release at the C++ level translates roughly into an rmb/wmb pair at the processor level, unless it's implemented using an actual interlocked instruction which is worse than barriers.
    - loeg 7y ago
      Nope. Acquire/release at C level translates into a compiler barrier and the exact machine implementation will depend on hardware. However, on x86, for example, with its strong memory semantics, ordinary stores already have release semantics and do not require an architectural barrier (wmb/sfence). No sfence, no lock prefix.
      - notacoward 7y ago
        First you said that the modern way is to use acquire/release instead of a barrier, then admitted that it is a barrier at the compiler level. The fact that it doesn't require a barrier instruction on the x86 is pretty much irrelevant. Not all the world's an x86, and even the x86 hasn't always been the same in this regard. So your rude "nope" is unwarranted. This attitude of "as it is (in my world) now, so it has ever been and ever will be (everywhere)" is exactly why these bugs occur.
        loeg 7y ago
        Compiler barriers and machine barriers are completely different things, despite having a similar name. In about the era you described, the Linux kernel community did go around throwing rmb()s and wmb()s everywhere, and those did translate to hardware fences. (And in general, it is not safe to elide a hardware fence on x86 despite relatively strong memory ordering.) This is what I believed you were talking about; maybe I misunderstood: apologies. Putting explicit compiler barriers in code is still not quite the modern model for relaxed atomic consumers. You use the abstract acquire/release load/store pseudo-functions, and they do the right thing depending on implementation. The compiler barriers in the relaxed atomics model are an internal implementation detail, not the API. x86 is just an example where the cheapest way to do a release-semantics store is a plain store. I never said all the world was x86 — if it were, abstract acquire/release relaxed atomics would be kind of pointless.