5 ms·

The focus on user space fragmentation is wrong, IMHO. One of the maintainers (Catalin Marinas) made [0] a much more important point: Apple makes no promises ab

by dmitrygr 10mo ago

The focus on user space fragmentation is wrong, IMHO. One of the maintainers (Catalin Marinas) made [0] a much more important point: Apple makes no promises about how their "TSO" bits work now or will work in the future. This mode was designed for Rosetta2, not the general public. It is not documented formally. Someone saying "it is TSO" is not documentation. A formal definition of a memory model is usually a very long document describing a lot of corner cases, for example [1] is a SUMMARY of the ARMv8 memory model, it is 31 pages long. It is a summary! The full spec makes up chapters D7 and D8 in [2], totaling 243 pages. Even there, there are corners that it does not touch on and people get wrong. Without such a spec for Apple's TSO mode, how can anyone rely on how it might or might not work? Additionally, you might find silicon bugs if you do something in this mode that Rosetta2 doesn't or didn't. Consider that the only first-party user of this mode was Rosetta2. Anything it does not do that you do might find a bug. The stated linux kernel policy of "do not break user space" is impossible to deliver on, if built on an undocumented hardware feature that might change at any time and was never fully publicly specified. The maintainers are right to reject this. [0] https://lwn.net/ml/linux-kernel/ZiKyWGKTw6Aqntod@arm.com/ https://lwn.net/ml/linux-kernel/ZiKyWGKTw6Aqntod@arm.com/ [1] https://developer.arm.com/-/media/Arm%20Developer%20Community/PDF/Learn%20the%20Architecture/Armv8-A%20memory%20model%20guide.pdf?revision=58b1dd0a-3800-4218-b21a-f95a0332034c https://developer.arm.com/-/media/Arm%20Developer%20Communit... [2] https://documentation-service.arm.com/static/6943ef0c79820936800f7f48?token= https://documentation-service.arm.com/static/6943ef0c7982093...

GeekyBear 10mo ago
As mentioned in the article, TSO is not exclusive to Apple's ARM implementation. > Some NVIDIA and Fujitsu CPUs run with TSO at all times; Apple's CPUs provide it as an optional feature that can be enabled at run time.
- dmitrygr 10mo ago
  > As mentioned in the article, TSO is not exclusive to Apple's ARM implementation. I thought I had been quite clear. I guess I'll try again even more clearly. "TSO" is three letters. It is not a spec. "We all do TSO" is as meaningful as "we all want world peace". Everyone has their own meaning for those words, and the meanings may differ significantly. Each is a memory model, and each can be called "TSO". But just like not every "John Smith" is the same person, nor is everything called "TSO" the same. Does NVIDIA's TSO order ALL reads with respect to ALL writes? Does Apple's? What does x86 do in that case? What does a Fujitsu CPU do? "TSO" does not mean the same thing to everyone just like "world peace" does not. If, for example, NVIDIA came out and said "our TSO mode complies 100% with x86 memory model and will always continue to", then Fujitsu did the same, and then (LOL) Apple also publicly promised that, then and only then would your comment make sense. As it stands, four entities use the same acronym to each mean their own thing, and you are assuming absolute equality because the three letters match. Fun story: I know FOR A FACT the answer to my above question about ordering of all reads vs all writes is not the same for x86, Apple's TSO, NVIDIA's TSO, and Fujitsu's TSO. Do you? Do you know how? Do you know how the answers might change with time and hardware revisions, given that at least Apple made no promises as to how their undocumented TSO mode works today or will work tomorrow? Exactly... One cannot build a stable f{ea,u}ture on undocumented un[der]specified hardware features.
  - GeekyBear 10mo ago
    Were you aware that all the BIOS implementations used in PC compatible computers (Compaq, AMI, Phoenix, etc.) were not identical and were compatible to a greater or lesser extent with the original IBM BIOS, yet Linux somehow supported PC compatible computers? > Someone saying "it is TSO" is not documentation. Trying to re-implement what IBM's BIOS did was not documentation either. The original sets the standard, whether a given implementation is perfectly equivalent or not.
    - dmitrygr 10mo ago
      I see no further point for this discussion. Either you truly do not understand or are pretending to not understand the difference between memory models (affect literally every memory access as long as the system is powered up) and BIOS (not used once the OS is up, and thus one-time at-boot quirks handling code can work around most issues). Either way, g'day. Oh, and to answer your question, yes, quite aware, actually. I've done quite a bit of low level work over the decades, including, curiously, working in the Apple platform kernel team at the time when this TSO bit appeared.
  - Dylan16807 10mo ago
    > I know FOR A FACT the answer to my above question about ordering of all reads vs all writes is not the same for x86, Apple's TSO, NVIDIA's TSO, and Fujitsu's TSO. Well of course they differ. TSO says that some reorderings are banned and some are optional, and there's a million factors that go into deciding when those options are taken. > "TSO" is three letters. It is not a spec. It's a few rules that you can depend on. Are those rules not enough to build a program on top of? The simpler you make your rules, the less spec you need. On the other end of the spectrum, a dozen specialized memory barriers need a ton of explanation.
    - dmitrygr 10mo ago
      >> "TSO" is three letters. It is not a spec. >It's a few rules that you can depend on. Until properly specified they are not "rules" but "hopes". Apple made no promises and provided no specs for their TSO mode. What makes you sure that that TSO bit on AppleM4pro acts the same as on AppleM1? That same "TSO" bit might mean yet a third thing on AppleM7megaMaxProEliteG2 in 2031. How do you know that an OS update that also updated iBoot on your Mac did not change some internal chip config MSR and now even on your AppleM4pro CPU whose TSO you understood, it acts differently due to this config bit change?
garaetjjte 10mo ago
Linux never had "must be publicly documented" as a general rule.
- dmitrygr 10mo ago
  That is fair. Reverse-engineered drivers are common. But depending on undocumented CPU core features seems a bit insane (as indeed the LKML post mentions)
spijdar 10mo ago
I'd posit that Linux having a hard policy of "do not break user space" has always just been "mostly true", especially around architecture specific stuff, and stuff tied to corporations. I bring this up only because recently, I made it my mission to get IBM's "PowerVM Lx86" dynamic translation system running on a POWER9 system running modern Debian Linux. This required a lot of hackery I won't go into here, but it revealed two things to me: 1. The lx86 system depended on an undocumented (??) pseudo-syscall to switch between big and little endian modes. This was "syscall 0x1EBE", which was implemented in the exception handler for PowerPC. In other words, it wasn't a real syscall, and tools like strace do NOT capture/log it. It was a conditional branch in the assembly entry point for the process exception handler which switches endianness and then returns. Quicker than a "real" syscall. Also, long gone in the Linux kernel, replaced with syscall switch_endian, hex 0x16B. Adding this in wasn't too hard, but it'd sure as heck never make it upstream again ;) 2. A lot of other Linux calls have had bits added which break old applications that are "rigidly coded". For example, netlink has had extra data stuffed into various structures, which code like lx86's translator panics on. To get network stuff running, even with an antique x86 userland, required removing a bunch of stuff from the kernel. Not disabling it via configuration, but deleting code. All this to say, there is a precedent for breaking the Linux user facing ABI for hardware features like this. I'm not saying that's a good thing, but it is a thing.
- xxpor 10mo ago
  >For example, netlink has had extra data stuffed into various structures If the ABI has a documented extension mechanism like netlink, this doesn't count as breaking user space.
saagarjha 10mo ago
It is actually documented now: https://developer.apple.com/documentation/virtualization/accelerating-the-performance-of-rosetta https://developer.apple.com/documentation/virtualization/acc...

GeekyBear 10mo ago

As mentioned in the article, TSO is not exclusive to Apple's ARM implementation. > Some NVIDIA and Fujitsu CPUs run with TSO at all times; Apple's CPUs provide it as an optional feature that can be enabled at run time.

dmitrygr 10mo ago
> As mentioned in the article, TSO is not exclusive to Apple's ARM implementation. I thought I had been quite clear. I guess I'll try again even more clearly. "TSO" is three letters. It is not a spec. "We all do TSO" is as meaningful as "we all want world peace". Everyone has their own meaning for those words, and the meanings may differ significantly. Each is a memory model, and each can be called "TSO". But just like not every "John Smith" is the same person, nor is everything called "TSO" the same. Does NVIDIA's TSO order ALL reads with respect to ALL writes? Does Apple's? What does x86 do in that case? What does a Fujitsu CPU do? "TSO" does not mean the same thing to everyone just like "world peace" does not. If, for example, NVIDIA came out and said "our TSO mode complies 100% with x86 memory model and will always continue to", then Fujitsu did the same, and then (LOL) Apple also publicly promised that, then and only then would your comment make sense. As it stands, four entities use the same acronym to each mean their own thing, and you are assuming absolute equality because the three letters match. Fun story: I know FOR A FACT the answer to my above question about ordering of all reads vs all writes is not the same for x86, Apple's TSO, NVIDIA's TSO, and Fujitsu's TSO. Do you? Do you know how? Do you know how the answers might change with time and hardware revisions, given that at least Apple made no promises as to how their undocumented TSO mode works today or will work tomorrow? Exactly... One cannot build a stable f{ea,u}ture on undocumented un[der]specified hardware features.
- GeekyBear 10mo ago
  Were you aware that all the BIOS implementations used in PC compatible computers (Compaq, AMI, Phoenix, etc.) were not identical and were compatible to a greater or lesser extent with the original IBM BIOS, yet Linux somehow supported PC compatible computers? > Someone saying "it is TSO" is not documentation. Trying to re-implement what IBM's BIOS did was not documentation either. The original sets the standard, whether a given implementation is perfectly equivalent or not.
  - dmitrygr 10mo ago
    I see no further point for this discussion. Either you truly do not understand or are pretending to not understand the difference between memory models (affect literally every memory access as long as the system is powered up) and BIOS (not used once the OS is up, and thus one-time at-boot quirks handling code can work around most issues). Either way, g'day. Oh, and to answer your question, yes, quite aware, actually. I've done quite a bit of low level work over the decades, including, curiously, working in the Apple platform kernel team at the time when this TSO bit appeared.
- Dylan16807 10mo ago
  > I know FOR A FACT the answer to my above question about ordering of all reads vs all writes is not the same for x86, Apple's TSO, NVIDIA's TSO, and Fujitsu's TSO. Well of course they differ. TSO says that some reorderings are banned and some are optional, and there's a million factors that go into deciding when those options are taken. > "TSO" is three letters. It is not a spec. It's a few rules that you can depend on. Are those rules not enough to build a program on top of? The simpler you make your rules, the less spec you need. On the other end of the spectrum, a dozen specialized memory barriers need a ton of explanation.
  - dmitrygr 10mo ago
    >> "TSO" is three letters. It is not a spec. >It's a few rules that you can depend on. Until properly specified they are not "rules" but "hopes". Apple made no promises and provided no specs for their TSO mode. What makes you sure that that TSO bit on AppleM4pro acts the same as on AppleM1? That same "TSO" bit might mean yet a third thing on AppleM7megaMaxProEliteG2 in 2031. How do you know that an OS update that also updated iBoot on your Mac did not change some internal chip config MSR and now even on your AppleM4pro CPU whose TSO you understood, it acts differently due to this config bit change?

garaetjjte 10mo ago

Linux never had "must be publicly documented" as a general rule.

dmitrygr 10mo ago
That is fair. Reverse-engineered drivers are common. But depending on undocumented CPU core features seems a bit insane (as indeed the LKML post mentions)

spijdar 10mo ago

I'd posit that Linux having a hard policy of "do not break user space" has always just been "mostly true", especially around architecture specific stuff, and stuff tied to corporations. I bring this up only because recently, I made it my mission to get IBM's "PowerVM Lx86" dynamic translation system running on a POWER9 system running modern Debian Linux. This required a lot of hackery I won't go into here, but it revealed two things to me: 1. The lx86 system depended on an undocumented (??) pseudo-syscall to switch between big and little endian modes. This was "syscall 0x1EBE", which was implemented in the exception handler for PowerPC. In other words, it wasn't a real syscall, and tools like strace do NOT capture/log it. It was a conditional branch in the assembly entry point for the process exception handler which switches endianness and then returns. Quicker than a "real" syscall. Also, long gone in the Linux kernel, replaced with syscall switch_endian, hex 0x16B. Adding this in wasn't too hard, but it'd sure as heck never make it upstream again ;) 2. A lot of other Linux calls have had bits added which break old applications that are "rigidly coded". For example, netlink has had extra data stuffed into various structures, which code like lx86's translator panics on. To get network stuff running, even with an antique x86 userland, required removing a bunch of stuff from the kernel. Not disabling it via configuration, but deleting code. All this to say, there is a precedent for breaking the Linux user facing ABI for hardware features like this. I'm not saying that's a good thing, but it is a thing.

xxpor 10mo ago
>For example, netlink has had extra data stuffed into various structures If the ABI has a documented extension mechanism like netlink, this doesn't count as breaking user space.

saagarjha 10mo ago

It is actually documented now: https://developer.apple.com/documentation/virtualization/accelerating-the-performance-of-rosetta https://developer.apple.com/documentation/virtualization/acc...