6 ms·

> The x86 processor was able to run the test successfully all 10,000 times, but the ARM processor failed on the 35th attempt. I think this issue might prove a

> The x86 processor was able to run the test successfully all 10,000 times, but the ARM processor failed on the 35th attempt. I think this issue might prove a problem in the long tail of desktop and server software running on ARM. A lot of desktop and server applications try to take advantage of all the cores. Many times, they are using libraries that were either implemented prior to C and C++ having defined memory models or else without that much care for memory model as long as it ran without issues on the developer computer (x86) and server (x86). Going to ARM is going to expose a lot of these bugs as developers recompile their code for ARM without making sure that their code actually adheres to the C/C++ memory models.

reitzensteinm 6y ago
It's possible to provide stronger guarantees than the spec requires. It wouldn't surprise me if server focused ARM chips ended up providing x86 style ordering to ensure compatibility with ported software.
- Jweb_Guru 6y ago
  It would very much surprise me, since the weaker ordering requirements are a significant part of how ARM achieves lower power consumption.
  - gpderetta 6y ago
    I would be surprised if that's the case. Why would memory ordering have any effect on power consumption?
    - Jweb_Guru 6y ago
      Because ensuring release/acquire for every operation (at least in the x86 way) requires supporting total store order, which requires much stronger cache coherence in the normal (no barrier) case and hence results in a lot more bus traffic. ARM didn't just make the memory model weaker for no reason.
      - bestboy 6y ago
        Seems plausible to me. I guess it would be pretty hard to maintain a TPD of about 200W for 80 3GHz ARM cores [1]. The bus traffic to synchronize 80 CPU caches can probably be significant. [1] https://www.anandtech.com/show/15871/amperes-product-list-80-cores-up-to-33-ghz-at-250-w-128-core-in-q4 https://www.anandtech.com/show/15871/amperes-product-list-80...
      - gpderetta 6y ago
        Do you have a concrete example were a weaker memory model would allow you to avoid memory coherence traffic?
      - BeeOnRope 6y ago
        I am pretty sure this is not correct. Every reordering effect I'm aware of is a core-local effect. That is, it happens in the core before (or at the moment) the data hits the L1D. It does not occur due to a weaker cache coherence system. Having a cache coherence system which was itself weaker, allowing reorderings consistent with the memory model, makes barriers and "implicit barriers" like address dependencies very expensive, and there is little evidence this is the case. Even in a hypothetical core which had a cache coherency system coupled to the memeory model, you aren't really avoiding any coherence traffic, just allowing certain reorderings such as satisfying requests out of order.
  - reitzensteinm 6y ago
    I agree that it will be a large power penalty, but the alternative may end up being the long tail of software suffering subtle concurrency correctness issues that just aren't there on x86. It may sacrifice some of their advantage but avoid being known as the CPU equivalent of a Ford Pinto. There's still plenty of room to undercut Intel's historical 60% gross margins even if you're shipping largely interchangeable products. The other way it might play out is a race to very high core counts as the main differentiator, providing single socket performance worth the hassle of not being able to run everything on it in a rock solid way. Postgres will work great. That redis fork that adds threads maybe not. Or Intel may start to allow developers to relax memory correctness guarantees on a process by process granularity in their own progress to high core counts. It's hard to imagine their current methods scaling to 1k cores. But if you asked me ten years ago I'd have said they wouldn't have made it to 56 cores, either.
    - zozbot234 6y ago
      > Or Intel may start to allow developers to relax memory correctness guarantees on a process by process granularity If Intel can allow developers/OS's to relax guarantees, why can't ARM allow OS's to strengthen them as necessary, while still keeping a relaxed memory model most of the time?
      - reitzensteinm 6y ago
        Sure, although you'd probably be better off with strong by default with opt in weakness. The amount of testing and verification in making sure something like Postgres runs well on a different memory order more is substantial. Adding the flag at the end is trivial. This way around, software is correct by default unless it explicitly asks to be unsafe.
chrismorgan 6y ago
Yeah, feels similar to Make concurrency bugs, where the makefile was developed with --jobs=1, and then years down the track someone says “let’s make it faster!” and tries --jobs=8 or similar, but that leads to compilation failures (or worse, succeeding in compiling the wrong thing), and because all the makefile was made years ago, it’s hard to track down exactly where the prerequisites are lacking; whereas if --jobs=8 had been used from the start, it would have been caught early.
xenadu02 6y ago
Some of them are latent bugs on x86 too, just rarer due to the stronger guarantees x86 provides.
vbezhenar 6y ago
Smartphones used ARM for a while. I think that many of the major libraries were used in one or another project. So I think that this problem won't be as severe, because those bugs hopefully were fixed. Also Raspberry Pi was a popular choice for many tinkerers for years which also helps with ARM penetration.
- naavis 6y ago
  What do you mean "for a while"? AFAIK most smartphones are still based on ARM SoCs.
  - nicoburns 6y ago
    I would read "for a while" as "for a number of years, since the iPhone".
    - deleted 6y ago
      [deleted]
  - bluehex 6y ago
    Maybe they meant to say “have been using” instead of “used”.
nindalf 6y ago
There’s now 2 incentive to support ARM better - Apple’s move to ARM on the desktop and cheaper cloud bills if you’re willing to use ARM instances. Either one wouldn’t be enough of an incentive, but together it will cause a shift in the next 3-5 years. Developers will become more aware of the differences between the architectures, tool chains will accommodate both better, people and software will stop assuming they are running on x86 as default. ARM won’t “win” the desktop or the server market, but it will become a viable alternative, squeezing the profits of companies who depend on x86.
- jfkebwjsbx 6y ago
  > cheaper cloud bills That remains to be seen.
  - nindalf 6y ago
    Graviton2 instances are available on AWS. They claim 40% better performance than their x86 peers - https://aws.amazon.com/ec2/instance-types/m6/ https://aws.amazon.com/ec2/instance-types/m6/
Someone 6y ago
Old software will have been tested on CPUs with weaker memory models. Also, Multi-threaded programming is hard, so the really long tail probably already is buggy on x86. Surfacing their bugs more often may be a blessing in disguise.
- zozbot234 6y ago
  In practice, CPU's with weaker memory models tend to be newer - a single-threaded CPU being the strongest model of them all. There are some interesting exceptions, such as e.g. the Alpha architecture - which is so 'weak' it needs a memory barrier even on pointer/array indirections, if multiple threads might be involved! (I actually did some digging and explained the issue in https://news.ycombinator.com/item?id=22282049 https://news.ycombinator.com/item?id=22282049 this old thread.)
  - gpderetta 6y ago
    ARM, POWER, early SPARCs all have a weaker memory model than intel. It is true though that with near total x86 domination i the server market for the las 20 years, newer server software might have a lot of x86-isms.
floatboth 6y ago
So far I have not encountered any problems with e.g. desktop Firefox (no surprise, gecko has been running on Android for ages) and various server things (mostly the BEAM.. and even some Haskell, even though before 8.8 GHC did not put proper memory barriers everywhere, and I used 8.6 back when I compiled the app). Most libraries and applications use stuff like mutexes, btw :) it's not like people who don't care about memory models try to make lockfree things often.

reitzensteinm 6y ago

It's possible to provide stronger guarantees than the spec requires. It wouldn't surprise me if server focused ARM chips ended up providing x86 style ordering to ensure compatibility with ported software.

Jweb_Guru 6y ago
It would very much surprise me, since the weaker ordering requirements are a significant part of how ARM achieves lower power consumption.
- gpderetta 6y ago
  I would be surprised if that's the case. Why would memory ordering have any effect on power consumption?
  - Jweb_Guru 6y ago
    Because ensuring release/acquire for every operation (at least in the x86 way) requires supporting total store order, which requires much stronger cache coherence in the normal (no barrier) case and hence results in a lot more bus traffic. ARM didn't just make the memory model weaker for no reason.
    - bestboy 6y ago
      Seems plausible to me. I guess it would be pretty hard to maintain a TPD of about 200W for 80 3GHz ARM cores [1]. The bus traffic to synchronize 80 CPU caches can probably be significant. [1] https://www.anandtech.com/show/15871/amperes-product-list-80-cores-up-to-33-ghz-at-250-w-128-core-in-q4 https://www.anandtech.com/show/15871/amperes-product-list-80...
    - gpderetta 6y ago
      Do you have a concrete example were a weaker memory model would allow you to avoid memory coherence traffic?
    - BeeOnRope 6y ago
      I am pretty sure this is not correct. Every reordering effect I'm aware of is a core-local effect. That is, it happens in the core before (or at the moment) the data hits the L1D. It does not occur due to a weaker cache coherence system. Having a cache coherence system which was itself weaker, allowing reorderings consistent with the memory model, makes barriers and "implicit barriers" like address dependencies very expensive, and there is little evidence this is the case. Even in a hypothetical core which had a cache coherency system coupled to the memeory model, you aren't really avoiding any coherence traffic, just allowing certain reorderings such as satisfying requests out of order.
- reitzensteinm 6y ago
  I agree that it will be a large power penalty, but the alternative may end up being the long tail of software suffering subtle concurrency correctness issues that just aren't there on x86. It may sacrifice some of their advantage but avoid being known as the CPU equivalent of a Ford Pinto. There's still plenty of room to undercut Intel's historical 60% gross margins even if you're shipping largely interchangeable products. The other way it might play out is a race to very high core counts as the main differentiator, providing single socket performance worth the hassle of not being able to run everything on it in a rock solid way. Postgres will work great. That redis fork that adds threads maybe not. Or Intel may start to allow developers to relax memory correctness guarantees on a process by process granularity in their own progress to high core counts. It's hard to imagine their current methods scaling to 1k cores. But if you asked me ten years ago I'd have said they wouldn't have made it to 56 cores, either.
  - zozbot234 6y ago
    > Or Intel may start to allow developers to relax memory correctness guarantees on a process by process granularity If Intel can allow developers/OS's to relax guarantees, why can't ARM allow OS's to strengthen them as necessary, while still keeping a relaxed memory model most of the time?
    - reitzensteinm 6y ago
      Sure, although you'd probably be better off with strong by default with opt in weakness. The amount of testing and verification in making sure something like Postgres runs well on a different memory order more is substantial. Adding the flag at the end is trivial. This way around, software is correct by default unless it explicitly asks to be unsafe.

chrismorgan 6y ago

Yeah, feels similar to Make concurrency bugs, where the makefile was developed with --jobs=1, and then years down the track someone says “let’s make it faster!” and tries --jobs=8 or similar, but that leads to compilation failures (or worse, succeeding in compiling the wrong thing), and because all the makefile was made years ago, it’s hard to track down exactly where the prerequisites are lacking; whereas if --jobs=8 had been used from the start, it would have been caught early.

xenadu02 6y ago

Some of them are latent bugs on x86 too, just rarer due to the stronger guarantees x86 provides.

vbezhenar 6y ago

Smartphones used ARM for a while. I think that many of the major libraries were used in one or another project. So I think that this problem won't be as severe, because those bugs hopefully were fixed. Also Raspberry Pi was a popular choice for many tinkerers for years which also helps with ARM penetration.

naavis 6y ago
What do you mean "for a while"? AFAIK most smartphones are still based on ARM SoCs.
- nicoburns 6y ago
  I would read "for a while" as "for a number of years, since the iPhone".
  - deleted 6y ago
    [deleted]
- bluehex 6y ago
  Maybe they meant to say “have been using” instead of “used”.

nindalf 6y ago

There’s now 2 incentive to support ARM better - Apple’s move to ARM on the desktop and cheaper cloud bills if you’re willing to use ARM instances. Either one wouldn’t be enough of an incentive, but together it will cause a shift in the next 3-5 years. Developers will become more aware of the differences between the architectures, tool chains will accommodate both better, people and software will stop assuming they are running on x86 as default. ARM won’t “win” the desktop or the server market, but it will become a viable alternative, squeezing the profits of companies who depend on x86.

jfkebwjsbx 6y ago
> cheaper cloud bills That remains to be seen.
- nindalf 6y ago
  Graviton2 instances are available on AWS. They claim 40% better performance than their x86 peers - https://aws.amazon.com/ec2/instance-types/m6/ https://aws.amazon.com/ec2/instance-types/m6/

Someone 6y ago

Old software will have been tested on CPUs with weaker memory models. Also, Multi-threaded programming is hard, so the really long tail probably already is buggy on x86. Surfacing their bugs more often may be a blessing in disguise.

zozbot234 6y ago
In practice, CPU's with weaker memory models tend to be newer - a single-threaded CPU being the strongest model of them all. There are some interesting exceptions, such as e.g. the Alpha architecture - which is so 'weak' it needs a memory barrier even on pointer/array indirections, if multiple threads might be involved! (I actually did some digging and explained the issue in https://news.ycombinator.com/item?id=22282049 https://news.ycombinator.com/item?id=22282049 this old thread.)
- gpderetta 6y ago
  ARM, POWER, early SPARCs all have a weaker memory model than intel. It is true though that with near total x86 domination i the server market for the las 20 years, newer server software might have a lot of x86-isms.

floatboth 6y ago

So far I have not encountered any problems with e.g. desktop Firefox (no surprise, gecko has been running on Android for ages) and various server things (mostly the BEAM.. and even some Haskell, even though before 8.8 GHC did not put proper memory barriers everywhere, and I used 8.6 back when I compiled the app). Most libraries and applications use stuff like mutexes, btw :) it's not like people who don't care about memory models try to make lockfree things often.