9 ms·

Post got the big one: Total Store Ordering (TSO). The rest are all techniques in reasonably common use, but unless you have hardware support for x86's strong m

• jsheard 2y ago
  It's surprising that (AFAIK) Qualcomm didn't implement TSO in the chips they made for the recent-ish Windows ARM machines. If anything they need fast x86 emulation even more than Apple does since Windows has a much longer tail of software support than macOS, there's going to be important Windows apps that stubbornly refuse to support native ARM basically forever.
  • scottlamb 2y ago
    Does Windows's translation take advantage of those where they exist? E.g. if I launch an aarch64 Windows VM on my M2, does it use the M2's support for TSO when running x86_64 .exes or does it insert these memory barriers? If not, it makes sense that Qualcomm didn't bother adding them.
    • Syonyk 2y ago
      I would expect it to not use TSO, because the toggle for it isn't, to the best of my knowledge, a general userspace toggle. It's something the kernel has to toggle, and so a VM may or may not (probably does not) even have access to the SCRs (system control registers) to change it.
      • zeusk 2y ago
        TSO toggle on Apple Silicon is a user-space accessible/writable register. It is used when you install rosetta2 for Linux VMs https://developer.apple.com/documentation/virtualization/running_intel_binaries_in_linux_vms_with_rosetta https://developer.apple.com/documentation/virtualization/run...
        • Syonyk 2y ago
          Are you sure it's userspace accessible? Based on https://github.com/saagarjha/TSOEnabler/blob/master/TSOEnabler/TSOEnabler.c https://github.com/saagarjha/TSOEnabler/blob/master/TSOEnabl..., it's a field in ACTLR_EL1, which is explicitly (per the ARMv8 spec, at least...) not accessible to userspace (EL0) execution. There may be some kernel interface to allow userspace to toggle that, but that's not the same as being a userspace-accessible SCR (and I also wouldn't expect it to be passed through to a VM - you'd likely need a hypercall to toggle it, unless the hypervisor emulated that, though admittedly I'm not quite as deep weeds on ARMv8 virtualization as I would prefer at the moment.
          • zeusk 2y ago
            Hmm, you’re right - maybe my memory serves incorrectly but yeah it seems it is privileged access but the interface is open to all processes to toggle the bit.
        • shadowfacts 2y ago
          It is not directly accessible from user-space. Making it so requires kernel support. Apple published a set of patches for doing this on Linux: https://developer.apple.com/documentation/virtualization/accelerating_the_performance_of_rosetta https://developer.apple.com/documentation/virtualization/acc... Without that kernel support, all processes in the VM (not just Rosetta-translated ones) are opted-in to TSO: > Without selective enablement, the system opts all processes into this memory mode [TSO], which degrades performance for native ARM processes that don’t need it.
          • mrpippy 2y ago
            Before Sequoia, a Linux VM using Rosetta would have TSO enabled all the time. With Sequoia, TSO is not enabled for Linux VMs, and that kernel patch (posted in the last few weeks) is required for Rosetta to be able to enable TSO for itself. If the kernel patch isn't present, Rosetta has a non-TSO fallback mode.
      • saagarjha 2y ago
        This is exposed to guest kernels of Sequoia (and maybe earlier?).
    • zeusk 2y ago
      The OS can use what hardware supports, Mac OS does because SEG is a tightly integrated group at Apple whereas Microsoft treats hardware vendors at arm's length (pun unintended). There are roadmap sharing, planning events through leadership but it is not as cohesive as it is at Apple.
    • saagarjha 2y ago
      No because Windows is not aware of how Apple does it. There exist Linux patches documenting how to do so, though.
      • scottlamb 2y ago
        The article says the following: > As far as I know this is not part of the ARM standard, but it also isn’t Apple specific: Nvidia Denver/Carmel and Fujitsu A64fx are other 64-bit ARM processors that also implement TSO (thanks to marcan for these details). I'm not sure how to interpret that—do these other parameters have distinct/proprietary TSO extensions? Are they referring to a single published (optional) extension that all three implement? The linked tweet has been deleted so no clues there, and I stopped digging.
        • saagarjha 2y ago
          Those are just TSO all the time I think. So they are stronger than the ARM requirement
  • deaddodo 2y ago
    Microsoft's AoT+JiT techniques still pull off impressive performance (90+% in almost every case, 96-99% in the majority). But yes, if they were actually serious about Windows on ARM, they would have implemented TSO in their "custom" Qualcomm SQ1/SQ2 chips.
    • wtallis 2y ago
      Last time I checked, the default behavior for Microsoft's translation was to pretend that the hardware is doing TSO, and hope it works out. So that should obviously be fast, but occasionally wrong.
      • saagarjha 2y ago
        They're a decent bit smarter than that but yes their emulation is not quite correct.
    • 486sx33 2y ago
      Its funny, Microsoft and ARM emulation is good because of Qualcomm, or rather in spite of Qualcomm's limitations. If Qualcomm had done better, then the software wouldn't have to be so good, and they'd likely have maintained more market share. Instead, Microsoft had to make their x86 on arm emu good enough to work on Qualcomm's crap, which now works really nicely on apple arm.
  • Syonyk 2y ago
    My guess is that the sort of "legacy x86-forever" apps for Windows don't really need much in the way of performance. Think your classic Visual Basic 6 sort of thing that a business relies on for decades. I'm also fairly certain that the TSO changes to the memory system are non-trivial, and it's possible that Qualcomm doesn't see it as a value-add in their chips - and they're probably right. Windows machines are such a hot mess that outside a relatively small group of users (who probably run Linux anyway, so aren't anyone's target market), nobody would know or care what TSO is. If it add costs and power and doesn't matter, why bother?
    • jsheard 2y ago
      > My guess is that the sort of "legacy x86-forever" apps for Windows don't really need much in the way of performance. Games are a pretty notable exception that demand high performance and for the most part will be stuck on x86 forever. Brand new games might start shipping native ARM Windows binaries if the platform gets enough momentum, but games have very limited support lifecycles so it's unlikely that many released before that point will ever be updated to ARM native.
      • doctorpangloss 2y ago
        > Brand new games might start shipping native ARM Windows binaries if the platform gets enough momentum, but games have very limited support lifecycles so it's unlikely that many released before that point will ever be updated to ARM native. Unity supports Windows ARM. Unreal: probably never. IMO, the PC gaming market is so fragmented, short of Microsoft developing games for the platform, like pre-sales scale multi-millions that EGS did, games on ARM will only happen by complete accident, not because it makes sense.
    • tiagod 2y ago
      > My guess is that the sort of "legacy x86-forever" apps for Windows don't really need much in the way of performance. Think your classic Visual Basic 6 sort of thing that a business relies on for decades. In my experience, there's a lot of that kind of software around that was initially designed for a much simpler use-case, and has decades of badly coded features bolted in, with questionable algorithmic choices. It can be unreasonably slow in modern hardware. Old government database sites are the worst examples in my experience. Clearly tested with a few hundred records, but 15 years later there's a few million and nobody bothered to create a bunch of indexes so searches take a couple minutes. I guess this way they can just charge to upgrade the hardware once in a while instead.
  • p_l 2y ago
    Qualcomm has been phoning it in in various forms for over a decade, including forcing MS to ship machines that do not really pass windows requirements (like broken firmware support). Maybe it got fixed with recent Snapdragon X, but I won't hold my breath. We're talking about a company that, if certain personal sources are to be believed, started the Snapdragon brand by deciding to cheapen out on memory bandwidth despite feedback that increasing it was critical and leaving the client to find out too late in the integration stage. Deciding that they make better money by not spending on implementing TSO, or not spending transistors on bigger caches, and getting more volume at lower cost, is perfectly normal.
  • mdasen 2y ago
    It's definitely surprising that Qualcomm didn't. Not only does Windows have a longer tail of software to support, but given that the vast majority of Windows machines will continue to be x86-64, there's little incentive to do work to support ARM. With the Mac, Apple told everyone "we're moving to ARM and that's final." With Windows, Microsoft is saying, "these ARM chips could be cool, what do you think?" On the Mac, you either got on board or were left behind. Users knew that the future was ARM and bought machines even if there might be some short-term growing pains. Developers knew that the future was ARM and worked hard to support it. But with Windows, there isn't a huge incentive for users to switch to ARM and there isn't an incentive for developers to encourage it. You can say there's some incentive if the ARM chips are better. While Qualcomm's chips are good, the benchmarks aren't really ahead of Intel/AMD and they aren't the power-sipping processors that Apple is putting out. If Apple hadn't implemented TSO, Mac users/developers would still switch because Apple told them to. Qualcomm has to convince users that their chips are worth the short-term pain - and that users shouldn't wait a few years to make the switch when the ecosystem is more mature. That's a much bigger hill to climb. Still, for Qualcomm, they might not even care about losing a little money for 5-10 years if it means they become one of the largest desktop processor vendors for the following 20+ years. As long as they can keep Microsoft's interest in ARM as a platform, they can bide their time.
    • bee_rider 2y ago
      I wonder if possible Qualcomm doesn’t super care about the long tail of software? Like maybe MS has some stats indicating that a very large percentage of software that they think will be used on these devices is first party, or stuff that reasonably should be expected to be compiled for ARM. How does the windows App Store work anyway, can they guarantee that all the stuff there gets compiled for ARM? Anyway, it is Windows not MacOS. The users expect some rough edges and poor craftsmanship, right?
      • RockRobotRock 2y ago
        I switched completely to macOS and trust me there are plenty of rough edges too. They're just in different places.
  • saagarjha 2y ago
    You can use RCpc atomics which are part of the standard architecture
  • dundarious 2y ago
    On a first order analysis, Qualcomm doesn't want good x64 support, because good x64 support furthers the lifetime of x64, and delays the "transition" to ARM. In the final analysis, I doubt that is an economically rational strategy, because even if there is to be a transition away from x64, you need a good legacy and migration story. And I doubt such a transition will happen in the next 10 years, and certainly not spurred by anything in Microsoft land. So maybe it's rational after all, because they know these Windows ARM products will never succeed, so they're just saving themselves the cost/effort of good support.
    • wolpoli 2y ago
      > On a first order analysis, Qualcomm doesn't want good x64 support, because good x64 support furthers the lifetime of x64, and delays the "transition" to ARM. The logical thing for Qualcomm in their current market share to do is to implement TSO now, then after they get momentum, create a high-end/low-end tier, and disable TSO for the low-end tier to force vendors to target both ARM/x68. What Qualcomm is doing now makes them look like they just don't care.
      • Someone 2y ago
        > create a high-end/low-end tier, and disable TSO for the low-end tier Wouldn’t that make the low-end tier run faster than the high-end tier, or force them to leave some performance on the table there? Also, would a per-process flag that controls TSO be possible? Ignoring whether it’s easy to do in the hardware, the only problem I can think of with that is that the OS would have to set that on processes when they start using shared memory, or forbid using shared memory by processes that do not have it set.
    • rowanG077 2y ago
      I would disagree that this is first order. First order is making the transition as smooth as possible, which obviously means having a very good translation layer. Only then you should even think about competing on comparability.
      • 2y ago
  • Const-me 2y ago
    > Qualcomm didn't implement TSO in the chips they made I’m not sure they can do that. Under Technology License Agreement, Qualcomm can build chips using ARM-designed CPU cores. Specifically, Qualcomm SQ1 uses ARM Cortex A76/A55 for the fast/slow CPU cores. I don’t think using ARM designed cores is enough to implement TSO, need custom ARM cores instead of the stock ones. To design custom ARM cores, Qualcomm needs architecture license from ARM which was recently been cancelled.
    • jsheard 2y ago
      SQ1/SQ2 was their previous attempt, the recently released Snapdragon Elite chips use the fully custom Oryon ARM cores they got from the Nuvia acquisition. That acquisition is what caused the current licensing drama with ARM, but for the time being they do have custom cores.
      • Const-me 2y ago
        > custom Oryon ARM cores they got from the Nuvia acquisition Nuvia was developing server CPUs. By now, I believe backward compatibility with x86 and AMD64 is rather unimportant for servers. Hosting providers and public clouds have been offering ARM64 Linux servers for quite a few years now, all important server-running software already has native ARM64 builds.
• ant6n 2y ago
  perhaps you could keep each process on one core. But that would kill multi-threaded performance.
• saagarjha 2y ago
  TSO is nice to have but it's definitely not necessary. Rosetta doesn't even require TSO on Linux anymore by default. It performs fine for typical applications.
• vlovich123 2y ago
  Is TSO something other than doing atomics with seq_cst?
  • j16sdiz 2y ago
    TSO is what x86 do when you are _not_ using atomics.
    • adrian_b 2y ago
      True, and it is a little more relaxed than sequential consistency. For simple loads and stores, the x86 CPUs do not reorder the loads between themselves or the stores between themselves. Also the stores are not done before previous loads. Only some special kinds of stores can be reordered, i.e. those caused by string instructions or the stores of vector registers that are marked as NT (non-temporal). So x86 does not need release stores, any simple store is suitable for this. Also store barriers are not normally needed. Acquire fences a.k.a. acquire barriers are sometimes needed, but much less often than on CPUs with weaker ordering for the memory accesses (for acquire fences both x86 and Arm Aarch64 have confusing mnemonics, i.e. LFENCE on x86 and DMB/DSB of the LD kind on Aarch64; in both cases these instructions are not load fences as suggested by the mnemonics, but acquire fences). When converting x86 code to Aarch64 code, there are many cases when simple stores must be replaced with release stores (a.k.a. Store-Release instructions in the Arm documentation) and there are many places where acquire barriers must be inserted, or, less frequently, store barriers must be inserted (for non-optimally written concurrent code it may also be necessary to replace some simple loads with Load-Acquire instructions of Aaarch64).
  • deleted 2y ago
    [deleted]
• wbl 2y ago
  Barrier injection isn't the issue as much as the barriers becoming expensive. There's no reason a CPU can't use the techniques of TSO support to support lesser barriers just as cheaply.
• commandersaki 2y ago
  I get pretty close to native performance with Rosetta 2 for Linux and I don't believe TSO is being used or taken advantage of. I wonder how important it really is.