4 ms·

Support for the TSO memory model on Arm CPUs (2024)

34 points by weinzierl 10mo ago

brcmthrowaway 10mo ago
It's a pity that Hector Martin stepped away from all this great work (under that name, anyway)
rbanffy 10mo ago
We need a TLA authority to help prevent collisions in the acronym space. It’s enough that MCP is also the Burroughs/Unisys mainframe operating system, now TSO is also the time-sharing option on IBM mainframes.
- yvdriess 10mo ago
  TSO? Did you mean tso.architecture.cpu?
  - rbanffy 10mo ago
    Either that or https://www.ibm.com/docs/en/zos-basic-skills?topic=interfaces-what-is-tso https://www.ibm.com/docs/en/zos-basic-skills?topic=interface... I would prefer we didn’t have so many collisions in the TLA space.
    - tyrellj 10mo ago
      The last company I worked at created TLAs for everything, or at least engineering always did. New company doesn't seem to have caught that bug yet though, thankfully.
dmitrygr 10mo ago
The focus on user space fragmentation is wrong, IMHO. One of the maintainers (Catalin Marinas) made [0] a much more important point: Apple makes no promises about how their "TSO" bits work now or will work in the future. This mode was designed for Rosetta2, not the general public. It is not documented formally. Someone saying "it is TSO" is not documentation. A formal definition of a memory model is usually a very long document describing a lot of corner cases, for example [1] is a SUMMARY of the ARMv8 memory model, it is 31 pages long. It is a summary! The full spec makes up chapters D7 and D8 in [2], totaling 243 pages. Even there, there are corners that it does not touch on and people get wrong. Without such a spec for Apple's TSO mode, how can anyone rely on how it might or might not work? Additionally, you might find silicon bugs if you do something in this mode that Rosetta2 doesn't or didn't. Consider that the only first-party user of this mode was Rosetta2. Anything it does not do that you do might find a bug. The stated linux kernel policy of "do not break user space" is impossible to deliver on, if built on an undocumented hardware feature that might change at any time and was never fully publicly specified. The maintainers are right to reject this. [0] https://lwn.net/ml/linux-kernel/ZiKyWGKTw6Aqntod@arm.com/ https://lwn.net/ml/linux-kernel/ZiKyWGKTw6Aqntod@arm.com/ [1] https://developer.arm.com/-/media/Arm%20Developer%20Community/PDF/Learn%20the%20Architecture/Armv8-A%20memory%20model%20guide.pdf?revision=58b1dd0a-3800-4218-b21a-f95a0332034c https://developer.arm.com/-/media/Arm%20Developer%20Communit... [2] https://documentation-service.arm.com/static/6943ef0c79820936800f7f48?token= https://documentation-service.arm.com/static/6943ef0c7982093...
- GeekyBear 10mo ago
  As mentioned in the article, TSO is not exclusive to Apple's ARM implementation. > Some NVIDIA and Fujitsu CPUs run with TSO at all times; Apple's CPUs provide it as an optional feature that can be enabled at run time.
  - dmitrygr 10mo ago
    > As mentioned in the article, TSO is not exclusive to Apple's ARM implementation. I thought I had been quite clear. I guess I'll try again even more clearly. "TSO" is three letters. It is not a spec. "We all do TSO" is as meaningful as "we all want world peace". Everyone has their own meaning for those words, and the meanings may differ significantly. Each is a memory model, and each can be called "TSO". But just like not every "John Smith" is the same person, nor is everything called "TSO" the same. Does NVIDIA's TSO order ALL reads with respect to ALL writes? Does Apple's? What does x86 do in that case? What does a Fujitsu CPU do? "TSO" does not mean the same thing to everyone just like "world peace" does not. If, for example, NVIDIA came out and said "our TSO mode complies 100% with x86 memory model and will always continue to", then Fujitsu did the same, and then (LOL) Apple also publicly promised that, then and only then would your comment make sense. As it stands, four entities use the same acronym to each mean their own thing, and you are assuming absolute equality because the three letters match. Fun story: I know FOR A FACT the answer to my above question about ordering of all reads vs all writes is not the same for x86, Apple's TSO, NVIDIA's TSO, and Fujitsu's TSO. Do you? Do you know how? Do you know how the answers might change with time and hardware revisions, given that at least Apple made no promises as to how their undocumented TSO mode works today or will work tomorrow? Exactly... One cannot build a stable f{ea,u}ture on undocumented un[der]specified hardware features.
    - GeekyBear 10mo ago
      Were you aware that all the BIOS implementations used in PC compatible computers (Compaq, AMI, Phoenix, etc.) were not identical and were compatible to a greater or lesser extent with the original IBM BIOS, yet Linux somehow supported PC compatible computers? > Someone saying "it is TSO" is not documentation. Trying to re-implement what IBM's BIOS did was not documentation either. The original sets the standard, whether a given implementation is perfectly equivalent or not.
slabtickler 10mo ago
imo the fragmentation argument is very flaky. this is not like, e.g. the old ARM big endian mode which accidentally created a separate ISA for really dubious reasons. the only thing that really have a need for alternate store orders on ARM at this point are just (x86) emulators. This argument may have made more sense 10, 15 years ago but ARM has become so ubiquitous that the idea of “oh my port of this originally x86 userland software crashes on ARM due to a synchronization bug, better use this TSO thing as a get-out-of-jail-free card “ really doesn’t have much water to it now
- gpderetta 10mo ago
  There is still interest in emulating x86 binaries on ARM. See the efforts from valve for example.

34 points by weinzierl 10mo ago

It's a pity that Hector Martin stepped away from all this great work (under that name, anyway)

We need a TLA authority to help prevent collisions in the acronym space. It’s enough that MCP is also the Burroughs/Unisys mainframe operating system, now TSO is also the time-sharing option on IBM mainframes.

yvdriess 10mo ago
TSO? Did you mean tso.architecture.cpu?
- rbanffy 10mo ago
  Either that or https://www.ibm.com/docs/en/zos-basic-skills?topic=interfaces-what-is-tso https://www.ibm.com/docs/en/zos-basic-skills?topic=interface... I would prefer we didn’t have so many collisions in the TLA space.
  - tyrellj 10mo ago
    The last company I worked at created TLAs for everything, or at least engineering always did. New company doesn't seem to have caught that bug yet though, thankfully.

dmitrygr 10mo ago

The focus on user space fragmentation is wrong, IMHO. One of the maintainers (Catalin Marinas) made [0] a much more important point: Apple makes no promises about how their "TSO" bits work now or will work in the future. This mode was designed for Rosetta2, not the general public. It is not documented formally. Someone saying "it is TSO" is not documentation. A formal definition of a memory model is usually a very long document describing a lot of corner cases, for example [1] is a SUMMARY of the ARMv8 memory model, it is 31 pages long. It is a summary! The full spec makes up chapters D7 and D8 in [2], totaling 243 pages. Even there, there are corners that it does not touch on and people get wrong. Without such a spec for Apple's TSO mode, how can anyone rely on how it might or might not work? Additionally, you might find silicon bugs if you do something in this mode that Rosetta2 doesn't or didn't. Consider that the only first-party user of this mode was Rosetta2. Anything it does not do that you do might find a bug. The stated linux kernel policy of "do not break user space" is impossible to deliver on, if built on an undocumented hardware feature that might change at any time and was never fully publicly specified. The maintainers are right to reject this. [0] https://lwn.net/ml/linux-kernel/ZiKyWGKTw6Aqntod@arm.com/ https://lwn.net/ml/linux-kernel/ZiKyWGKTw6Aqntod@arm.com/ [1] https://developer.arm.com/-/media/Arm%20Developer%20Community/PDF/Learn%20the%20Architecture/Armv8-A%20memory%20model%20guide.pdf?revision=58b1dd0a-3800-4218-b21a-f95a0332034c https://developer.arm.com/-/media/Arm%20Developer%20Communit... [2] https://documentation-service.arm.com/static/6943ef0c79820936800f7f48?token= https://documentation-service.arm.com/static/6943ef0c7982093...

GeekyBear 10mo ago
As mentioned in the article, TSO is not exclusive to Apple's ARM implementation. > Some NVIDIA and Fujitsu CPUs run with TSO at all times; Apple's CPUs provide it as an optional feature that can be enabled at run time.
- dmitrygr 10mo ago
  > As mentioned in the article, TSO is not exclusive to Apple's ARM implementation. I thought I had been quite clear. I guess I'll try again even more clearly. "TSO" is three letters. It is not a spec. "We all do TSO" is as meaningful as "we all want world peace". Everyone has their own meaning for those words, and the meanings may differ significantly. Each is a memory model, and each can be called "TSO". But just like not every "John Smith" is the same person, nor is everything called "TSO" the same. Does NVIDIA's TSO order ALL reads with respect to ALL writes? Does Apple's? What does x86 do in that case? What does a Fujitsu CPU do? "TSO" does not mean the same thing to everyone just like "world peace" does not. If, for example, NVIDIA came out and said "our TSO mode complies 100% with x86 memory model and will always continue to", then Fujitsu did the same, and then (LOL) Apple also publicly promised that, then and only then would your comment make sense. As it stands, four entities use the same acronym to each mean their own thing, and you are assuming absolute equality because the three letters match. Fun story: I know FOR A FACT the answer to my above question about ordering of all reads vs all writes is not the same for x86, Apple's TSO, NVIDIA's TSO, and Fujitsu's TSO. Do you? Do you know how? Do you know how the answers might change with time and hardware revisions, given that at least Apple made no promises as to how their undocumented TSO mode works today or will work tomorrow? Exactly... One cannot build a stable f{ea,u}ture on undocumented un[der]specified hardware features.
  - GeekyBear 10mo ago
    Were you aware that all the BIOS implementations used in PC compatible computers (Compaq, AMI, Phoenix, etc.) were not identical and were compatible to a greater or lesser extent with the original IBM BIOS, yet Linux somehow supported PC compatible computers? > Someone saying "it is TSO" is not documentation. Trying to re-implement what IBM's BIOS did was not documentation either. The original sets the standard, whether a given implementation is perfectly equivalent or not.

slabtickler 10mo ago

imo the fragmentation argument is very flaky. this is not like, e.g. the old ARM big endian mode which accidentally created a separate ISA for really dubious reasons. the only thing that really have a need for alternate store orders on ARM at this point are just (x86) emulators. This argument may have made more sense 10, 15 years ago but ARM has become so ubiquitous that the idea of “oh my port of this originally x86 userland software crashes on ARM due to a synchronization bug, better use this TSO thing as a get-out-of-jail-free card “ really doesn’t have much water to it now

gpderetta 10mo ago
There is still interest in emulating x86 binaries on ARM. See the efforts from valve for example.