14 ms·
Examining ARM vs. x86 Memory Models with Rust
- RcouF1uZ4gsC 6y ago> The x86 processor was able to run the test successfully all 10,000 times, but the ARM processor failed on the 35th attempt. I think this issue might prove a problem in the long tail of desktop and server software running on ARM. A lot of desktop and server applications try to take advantage of all the cores. Many times, they are using libraries that were either implemented prior to C and C++ having defined memory models or else without that much care for memory model as long as it ran without issues on the developer computer (x86) and server (x86). Going to ARM is going to expose a lot of these bugs as developers recompile their code for ARM without making sure that their code actually adheres to the C/C++ memory models.
- reitzensteinm 6y agoIt's possible to provide stronger guarantees than the spec requires. It wouldn't surprise me if server focused ARM chips ended up providing x86 style ordering to ensure compatibility with ported software.
- Jweb_Guru 6y agoIt would very much surprise me, since the weaker ordering requirements are a significant part of how ARM achieves lower power consumption.
- gpderetta 6y agoI would be surprised if that's the case. Why would memory ordering have any effect on power consumption?
- Jweb_Guru 6y agoBecause ensuring release/acquire for every operation (at least in the x86 way) requires supporting total store order, which requires much stronger cache coherence in the normal (no barrier) case and hence results in a lot more bus traffic. ARM didn't just make the memory model weaker for no reason.
- bestboy 6y agoSeems plausible to me. I guess it would be pretty hard to maintain a TPD of about 200W for 80 3GHz ARM cores [1]. The bus traffic to synchronize 80 CPU caches can probably be significant. [1] https://www.anandtech.com/show/15871/amperes-product-list-80-cores-up-to-33-ghz-at-250-w-128-core-in-q4 https://www.anandtech.com/show/15871/amperes-product-list-80...
- gpderetta 6y agoDo you have a concrete example were a weaker memory model would allow you to avoid memory coherence traffic?
- BeeOnRope 6y agoI am pretty sure this is not correct. Every reordering effect I'm aware of is a core-local effect. That is, it happens in the core before (or at the moment) the data hits the L1D. It does not occur due to a weaker cache coherence system. Having a cache coherence system which was itself weaker, allowing reorderings consistent with the memory model, makes barriers and "implicit barriers" like address dependencies very expensive, and there is little evidence this is the case. Even in a hypothetical core which had a cache coherency system coupled to the memeory model, you aren't really avoiding any coherence traffic, just allowing certain reorderings such as satisfying requests out of order.
- reitzensteinm 6y agoI agree that it will be a large power penalty, but the alternative may end up being the long tail of software suffering subtle concurrency correctness issues that just aren't there on x86. It may sacrifice some of their advantage but avoid being known as the CPU equivalent of a Ford Pinto. There's still plenty of room to undercut Intel's historical 60% gross margins even if you're shipping largely interchangeable products. The other way it might play out is a race to very high core counts as the main differentiator, providing single socket performance worth the hassle of not being able to run everything on it in a rock solid way. Postgres will work great. That redis fork that adds threads maybe not. Or Intel may start to allow developers to relax memory correctness guarantees on a process by process granularity in their own progress to high core counts. It's hard to imagine their current methods scaling to 1k cores. But if you asked me ten years ago I'd have said they wouldn't have made it to 56 cores, either.
- zozbot234 6y ago> Or Intel may start to allow developers to relax memory correctness guarantees on a process by process granularity If Intel can allow developers/OS's to relax guarantees, why can't ARM allow OS's to strengthen them as necessary, while still keeping a relaxed memory model most of the time?
- reitzensteinm 6y agoSure, although you'd probably be better off with strong by default with opt in weakness. The amount of testing and verification in making sure something like Postgres runs well on a different memory order more is substantial. Adding the flag at the end is trivial. This way around, software is correct by default unless it explicitly asks to be unsafe.
- chrismorgan 6y agoYeah, feels similar to Make concurrency bugs, where the makefile was developed with --jobs=1, and then years down the track someone says “let’s make it faster!” and tries --jobs=8 or similar, but that leads to compilation failures (or worse, succeeding in compiling the wrong thing), and because all the makefile was made years ago, it’s hard to track down exactly where the prerequisites are lacking; whereas if --jobs=8 had been used from the start, it would have been caught early.
- xenadu02 6y agoSome of them are latent bugs on x86 too, just rarer due to the stronger guarantees x86 provides.
- vbezhenar 6y agoSmartphones used ARM for a while. I think that many of the major libraries were used in one or another project. So I think that this problem won't be as severe, because those bugs hopefully were fixed. Also Raspberry Pi was a popular choice for many tinkerers for years which also helps with ARM penetration.
- nindalf 6y agoThere’s now 2 incentive to support ARM better - Apple’s move to ARM on the desktop and cheaper cloud bills if you’re willing to use ARM instances. Either one wouldn’t be enough of an incentive, but together it will cause a shift in the next 3-5 years. Developers will become more aware of the differences between the architectures, tool chains will accommodate both better, people and software will stop assuming they are running on x86 as default. ARM won’t “win” the desktop or the server market, but it will become a viable alternative, squeezing the profits of companies who depend on x86.
- jfkebwjsbx 6y ago> cheaper cloud bills That remains to be seen.
- nindalf 6y agoGraviton2 instances are available on AWS. They claim 40% better performance than their x86 peers - https://aws.amazon.com/ec2/instance-types/m6/ https://aws.amazon.com/ec2/instance-types/m6/
- Someone 6y agoOld software will have been tested on CPUs with weaker memory models. Also, Multi-threaded programming is hard, so the really long tail probably already is buggy on x86. Surfacing their bugs more often may be a blessing in disguise.
- zozbot234 6y agoIn practice, CPU's with weaker memory models tend to be newer - a single-threaded CPU being the strongest model of them all. There are some interesting exceptions, such as e.g. the Alpha architecture - which is so 'weak' it needs a memory barrier even on pointer/array indirections, if multiple threads might be involved! (I actually did some digging and explained the issue in https://news.ycombinator.com/item?id=22282049 https://news.ycombinator.com/item?id=22282049 this old thread.)
- gpderetta 6y agoARM, POWER, early SPARCs all have a weaker memory model than intel. It is true though that with near total x86 domination i the server market for the las 20 years, newer server software might have a lot of x86-isms.
- floatboth 6y agoSo far I have not encountered any problems with e.g. desktop Firefox (no surprise, gecko has been running on Android for ages) and various server things (mostly the BEAM.. and even some Haskell, even though before 8.8 GHC did not put proper memory barriers everywhere, and I used 8.6 back when I compiled the app). Most libraries and applications use stuff like mutexes, btw :) it's not like people who don't care about memory models try to make lockfree things often.
- ramshanker 6y agoAnd the good news here is that many libraries have accelerated their ARM port / investigations.
- userbinator 6y agoIf we initialize the contents of both pointers to 0 Is this a "Rust-ism"? I had a double-take while reading that, because in C that would mean a null pointer, and in the terminology I'm used to, the intent is to set the pointee to 0. Note that x86 does allow some memory reordering: https://preshing.com/20120515/memory-reordering-caught-in-the-act/ https://preshing.com/20120515/memory-reordering-caught-in-th... (I have experience debugging and fixing an extremely rare bug caused by the above subtle reordering, which occurred approximately once every 3-4 months.)
- kibwen 6y agoC programmers use "contents" to refer to the address of the pointer? As a non-C programmer, asking what is "contained" in the pointer would certainly refer to the pointee.
- jfkebwjsbx 6y agoA pointer does not contain what it points to, it just points to it. The content (the value) of a pointer is an address.
- wahern 6y agoIn C a pointer is an object in its own right, so referring to the contents of a pointer, p, means the same thing as referring to the contents of an integer, i; namely, the value held by the object. A pointer value has semantic meaning and usefulness the same as an integer value, completely divorced from the machine representation, which in a pure C program should always be irrelevant for pointer and arithmetic types alike. I don't think I can articulate well why it's important to keep in mind and appreciate value semantics. Certainly many C programmers, especially the newer ones, are far too concerned with value representation rather than the abstract value itself, often conflating the two. I guess one good example of why it's important to understand pointers as proper objects with abstract values is when working with pointers-to-pointers, pointers-to-pointers-to-pointers, etc. Pointer arithmetic is another case, which doesn't overlap with the former as much as you'd think. In these cases understanding pointers as values is important to understanding the semantics of a program, and more generally how to leverage the language efficiently and safely. Note that these semantics have no analog in references, the construct in, e.g., C++. It doesn't make sense to conceptualize references as independent objects; rather, a reference is a syntactic construct, no more an object than a list initializer. Yeah, as a C programmer I would find it kind of odd if someone seemed to conceptually elide the character of pointers as proper first-class objects.
- barskern 6y agoIsn't the following statement always true, as casting using `as` will silently ~~overflow~~ truncate the `u32` if `usize` is 64-bits? assert!((samples as u32) <= u32::MAX); EDIT: I know it's a contrived example, but I was just curious if my understanding is correct. I also found this page in the nomicon about casting: https://doc.rust-lang.org/nomicon/casts.html https://doc.rust-lang.org/nomicon/casts.html EDIT2: As I thought casting a `usize` which is 64-bits to a `u32` causes it to be truncated and hence the assertion is always true. Further by using a number that's bigger than a `u32`, this example contains undefined behavior. This is due to the use of `slice::from_raw_parts` where `self.samples` is left as a `usize` and hence takes a much bigger slice than what was allocated (the leftover of the truncate operation). I made a small playground which demonstrates the segfault. https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=2d6a189182daa3884eebc7524d1b8223 https://play.rust-lang.org/?version=stable&mode=debug&editio.... The assertion should rather be: assert!(samples <= u32::MAX as usize); Don't get me wrong, I think the blogpost is a great explanatory article about memory ordering and the example is rather contrived. I just wanted to reassure myself that my understanding was correct and further perhaps help someone not seeing this issue (as this is a very easy trap to fall into).
- redbluemonkey 6y agoYou are correct. Thanks for the pickup. Fixed the post and repo.
- estebank 6y agoTo be honest, this check deserves to be in clippy.
- ajross 6y ago> Where the memory model of ARM differs from X86 is that ARM CPU’s will re-order writes relative to other writes, whereas X86 will not. May. Not will. The difference here is important, because the actual memory ordering presented is an issue of hardware implementation choice (and of course the local vagaries like cache line alignment, interrupt order and the behavior of other CPUs on the bus). You can't just write some sample code to demonstrate it and expect it's going to work the same on "ARM". In fact I'd be really curious how Apple handles this during the Mac transition. I wouldn't be at all surprised if, purely for the sake of compatibility, they implement an strongly ordered x86-style cache hierarchy. Bugs in this world can be extremely difficult to diagnose, and honestly cache coherence transistors aren't that expensive relative to the rest of the system.
- Retr0spectrum 6y agoThe cost in terms of transistors may be small, but what about performance? (This is not a rhetorical question, I'm curious to know)
- ajross 6y agoThe fastest general purpose CPUs in the world implement a strongly ordered architecture, so clearly it can be done.
- throwaway_pdp09 6y agoMy question was exactly @Retr0spectrum's, the cost in performance, but I don't accept your answer (yet). I presume you mean intel and intel has a large transistor budget to burn, and quite like did to get that fastest perf. If you don't have to enforce strong ordering, how much extra a) actual speed b) transistor budget could you free up? (not a hardware guy
- BeeOnRope 6y agoReordering rarely or never comes from "cache coherence" on CPUs, but rather core local effects like store buffering, out of order execution, coalescing and out of order commit in the store buffer, etc.
- truth_seeker 6y agoIsn't this something Rust std lib api or perhaps LLVM backend should take care of ?
- secondcoming 6y agoThe problem is that there's likely to be a fair amount of logically broken 3rd party code out there because x86's memory model semantics doesn't reveal the issues. The compiler can't help here, it'll just do what the developer told it to do. It's a whole new 'it works on my machine' issue (for some people).
- devit 6y agoIt does as long as the programmer uses it properly (i.e. you don't use atomics weaker than SeqCst or unsafe code with shared + mutable accesses unless you have proven that it's correct).
- monocasa 6y agoSee how the functions are all unsafe? The primitives given by the Rust standard library actually do handle this; the author is simply going off the beaten path to illustrate one of the aspects those primitives need to paper over for you.
- Diggsey 6y agoOne thing omitted from this article is that it's not only the physical hardware that may (effectively) re-order operations. The compiler may also perform these re-orderings. The compiler's re-orderings will always be valid according to the abstract memory model rather than the hardware's, so even on x86 you must use the correct memory orderings, or risk subtle bugs due to compiler optimisations.
- comex 6y agoIn particular, the “multi-threaded using volatile” example is technically incorrect because the compiler is allowed to move non-volatile accesses across volatile ones. [1] To make it correct (but still only on x86), you’d have to use std::sync::atomic::compiler_barrier. [1] https://gcc.gnu.org/onlinedocs/gcc/Volatiles.html https://gcc.gnu.org/onlinedocs/gcc/Volatiles.html
- jhoechtl 6y agoDid this page just loaded for me with the glimpse of an eye? Way to do it!