3 ms·

Very old Macs used this trick to squeeze their ROM routines down a bit, operating with 24 bit addressing and using the top bits for flags and whatnot. Of cours

Very old Macs used this trick to squeeze their ROM routines down a bit, operating with 24 bit addressing and using the top bits for flags and whatnot. Of course they ran into trouble when machines with 16MB of memory started appearing. If you do this you might be making more work for yourself in the future when you buy a new machine with 256EB of main memory.

eschneider 3y ago
Yeah, this was definitely a thing on old macs. Us olds also remember the pain and agony of cleaning this up for 32 bit addressing. Cool as these hacks are, do yourself a favor and avoid the (eventual) agony.
Dylan16807 3y ago
x86_64 (among others) specifically avoided that incompatibility, by the CPU forcing programs to mask those bits out instead of ignoring them. So programs are very compatible into the future, they just need to limit the range their memory allocator uses. Then it later added explicit automatic masking, which also avoids the problem. As long as your program can make do with smaller amounts of memory, there are no downsides.
- masklinn 3y ago
  It does not actually make the software future proof: if the kernel returns a large address the software will either fault if it checks, or stomp over then mask out the wrong thing. Virtual memory is what mitigates the issue, as it lets the OS provide large virtual addresses on a per-process basis, and independent of physical addressing. This was already used on 32b x86: PAE decorrelated the physical address space from the virtual (giving the OS 36 bits physical address — with a theoretical upper bound of 64 — even as individual processes were still restricted to 32); and on windows the /LARGEADDRESSAWARE link switch allowed manipulating the virtual address space on a per-process granularity.
  - Dylan16807 3y ago
    > if the kernel returns a large address That's why I said you need to control your allocator. It's so much easier than rewriting everything that deals with pointers. And of course everything has virtual memory. But even without it, you're not really worse off when you upgrade the CPU if masking was already enforced. > This was already used on 32b x86: PAE decorrelated the physical address space from the virtual (giving the OS 36 bits physical address — with a theoretical upper bound of 64 — even as individual processes were still restricted to 32); and on windows the /LARGEADDRESSAWARE link switch allowed manipulating the virtual address space on a per-process granularity. I wouldn't characterize this as very similar. Physical and virtual address space already had to be decorrelated to go above about 1GB, and PAE doesn't meaningfully extend how much memory a 32 bit processes can have. And LAA is based on software assumptions that were never particularly tied to hardware, and it can be relevant with or without PAE.
    - masklinn 3y ago
      > That's why I said you need to control your allocator. The allocator does not matter unless by "control your allocator" you mean "the OS provides an API which can restrict the virtual memory system", which has nothign to do with the allocator. If you're using 19 bits for tagging and the OS returns a 52 bit pointer, you don't have the space for your tagging scheme, no matter what your allocator is. > PAE doesn't meaningfully extend how much memory a 32 bit processes can have. Of course not. What it does is show that there is no correlation between the physical and virtual address spaces, so you can have a 64 bits physical address space and only provide a 48 bits virtual address space to applications. > And LAA is based on software assumptions that were never particularly tied to hardware, and it can be relevant with or without PAE. Missing the point, the example of LAA shows that the OS can provide different virtual address spaces to different processes. So the OS can provide 48 bit virtual address spaces by default, and a full 64 bits virtual address space to applications which request it. The former can then keep using tagged pointers just fine.
      - Dylan16807 3y ago
        > The allocator does not matter The allocator gets address space. The OS doesn't just shove things into your process in general. It can't return a 52 bit pointer out of nowhere. My first draft mentioned OS explicitly but it's not really the OS that's making the decisions. > What it does is show that there is no correlation between the physical and virtual address spaces It does show that, but that decorrelation is neither necessary nor sufficient to solve this problem. > Missing the point, the example of LAA shows that the OS can provide different virtual address spaces to different processes. I agree with that. In the context of your overall post it wasn't clear to me that that was why you mentioned it.
utopcell 3y ago
Slight correction: 256 TiB, not EiB.
- cozzyd 3y ago
  2^20 is perhaps more than slight
drfuchs 3y ago
What’s old is new again: IBM 360 architecture also only used 24 bits for addresses back in the 1960s, so you could stuff data in the upper byte of pointers. Eventually, programs had to declare themselves to be in “Extended Mode” if they wanted to get access to 31-bit addressing instead.
vidarh 3y ago
I'm guessing the problem occurred specifically with CPUs newer than the 68000 irrespective of amount of RAM? (Microsoft's) Amiga Basic also did this and stopped working on newer CPUs, as the 68000 only has 24 address lines and ignores the top 8 bits of a 32 bit address, but 68020 and up uses all 32 (I don't remember about the 68010, but that was pin compatible with the 68000 so I'm guessing not)
- ithkuil 3y ago
  The 68010 was indeed pin compatible with the 6800 and had only 24 address bits. But the 68012 (which was otherwise software compatible with the 68010) extended the address bus (and had more pins) and was the first m68k CPU to exhibit this compat issue. The 68020 was more popular but came a couple of years later
  - vidarh 3y ago
    That's interesting. I'm not sure I was ever aware of the 68012.
    - ithkuil 3y ago
      Possibly only used by Alliant
- KMag 3y ago
  There was hardware (not sure if it was CPU state or state in the memory controller) to zero-out all of the address bits above the 24th, allowing backward compatibility. I forget the shortcut (option-I? option-M?), but if you selected an executable in Finder, you could go in and change the hardware compatibility mode used to run it.
  - vidarh 3y ago
    That's interesting. I don't think the address hack was widely used on the Amiga, so we didn't get anything like that. Amiga Basic was the main one and that was ditched for Arexx for the newer OS versions (a shame, because as much as I liked to mock Microsoft at the time, Amiga Basic was quite decent). I don't think the incompatibility had anything to do with ditching it - people have patched it and it wasn't a big deal; I'm assuming it didn't get an official patch because Commodore had already decided to replace it.
KMag 3y ago
And in later versions of Classic Mac OS, for each executable, metadata was stored on whether to set the hardware for 32-bit or 24-bit addressing. As a kid, I definitely monkeyed with the checkbox there not understanding what it did. Having seen no change in behavior, I guess all the applications I used were 32-bit clean.

eschneider 3y ago

Yeah, this was definitely a thing on old macs. Us olds also remember the pain and agony of cleaning this up for 32 bit addressing. Cool as these hacks are, do yourself a favor and avoid the (eventual) agony.

Dylan16807 3y ago

x86_64 (among others) specifically avoided that incompatibility, by the CPU forcing programs to mask those bits out instead of ignoring them. So programs are very compatible into the future, they just need to limit the range their memory allocator uses. Then it later added explicit automatic masking, which also avoids the problem. As long as your program can make do with smaller amounts of memory, there are no downsides.

masklinn 3y ago
It does not actually make the software future proof: if the kernel returns a large address the software will either fault if it checks, or stomp over then mask out the wrong thing. Virtual memory is what mitigates the issue, as it lets the OS provide large virtual addresses on a per-process basis, and independent of physical addressing. This was already used on 32b x86: PAE decorrelated the physical address space from the virtual (giving the OS 36 bits physical address — with a theoretical upper bound of 64 — even as individual processes were still restricted to 32); and on windows the /LARGEADDRESSAWARE link switch allowed manipulating the virtual address space on a per-process granularity.
- Dylan16807 3y ago
  > if the kernel returns a large address That's why I said you need to control your allocator. It's so much easier than rewriting everything that deals with pointers. And of course everything has virtual memory. But even without it, you're not really worse off when you upgrade the CPU if masking was already enforced. > This was already used on 32b x86: PAE decorrelated the physical address space from the virtual (giving the OS 36 bits physical address — with a theoretical upper bound of 64 — even as individual processes were still restricted to 32); and on windows the /LARGEADDRESSAWARE link switch allowed manipulating the virtual address space on a per-process granularity. I wouldn't characterize this as very similar. Physical and virtual address space already had to be decorrelated to go above about 1GB, and PAE doesn't meaningfully extend how much memory a 32 bit processes can have. And LAA is based on software assumptions that were never particularly tied to hardware, and it can be relevant with or without PAE.
  - masklinn 3y ago
    > That's why I said you need to control your allocator. The allocator does not matter unless by "control your allocator" you mean "the OS provides an API which can restrict the virtual memory system", which has nothign to do with the allocator. If you're using 19 bits for tagging and the OS returns a 52 bit pointer, you don't have the space for your tagging scheme, no matter what your allocator is. > PAE doesn't meaningfully extend how much memory a 32 bit processes can have. Of course not. What it does is show that there is no correlation between the physical and virtual address spaces, so you can have a 64 bits physical address space and only provide a 48 bits virtual address space to applications. > And LAA is based on software assumptions that were never particularly tied to hardware, and it can be relevant with or without PAE. Missing the point, the example of LAA shows that the OS can provide different virtual address spaces to different processes. So the OS can provide 48 bit virtual address spaces by default, and a full 64 bits virtual address space to applications which request it. The former can then keep using tagged pointers just fine.
    - Dylan16807 3y ago
      > The allocator does not matter The allocator gets address space. The OS doesn't just shove things into your process in general. It can't return a 52 bit pointer out of nowhere. My first draft mentioned OS explicitly but it's not really the OS that's making the decisions. > What it does is show that there is no correlation between the physical and virtual address spaces It does show that, but that decorrelation is neither necessary nor sufficient to solve this problem. > Missing the point, the example of LAA shows that the OS can provide different virtual address spaces to different processes. I agree with that. In the context of your overall post it wasn't clear to me that that was why you mentioned it.

utopcell 3y ago

Slight correction: 256 TiB, not EiB.

cozzyd 3y ago
2^20 is perhaps more than slight

drfuchs 3y ago

What’s old is new again: IBM 360 architecture also only used 24 bits for addresses back in the 1960s, so you could stuff data in the upper byte of pointers. Eventually, programs had to declare themselves to be in “Extended Mode” if they wanted to get access to 31-bit addressing instead.

vidarh 3y ago

I'm guessing the problem occurred specifically with CPUs newer than the 68000 irrespective of amount of RAM? (Microsoft's) Amiga Basic also did this and stopped working on newer CPUs, as the 68000 only has 24 address lines and ignores the top 8 bits of a 32 bit address, but 68020 and up uses all 32 (I don't remember about the 68010, but that was pin compatible with the 68000 so I'm guessing not)

ithkuil 3y ago
The 68010 was indeed pin compatible with the 6800 and had only 24 address bits. But the 68012 (which was otherwise software compatible with the 68010) extended the address bus (and had more pins) and was the first m68k CPU to exhibit this compat issue. The 68020 was more popular but came a couple of years later
- vidarh 3y ago
  That's interesting. I'm not sure I was ever aware of the 68012.
  - ithkuil 3y ago
    Possibly only used by Alliant
KMag 3y ago
There was hardware (not sure if it was CPU state or state in the memory controller) to zero-out all of the address bits above the 24th, allowing backward compatibility. I forget the shortcut (option-I? option-M?), but if you selected an executable in Finder, you could go in and change the hardware compatibility mode used to run it.
- vidarh 3y ago
  That's interesting. I don't think the address hack was widely used on the Amiga, so we didn't get anything like that. Amiga Basic was the main one and that was ditched for Arexx for the newer OS versions (a shame, because as much as I liked to mock Microsoft at the time, Amiga Basic was quite decent). I don't think the incompatibility had anything to do with ditching it - people have patched it and it wasn't a big deal; I'm assuming it didn't get an official patch because Commodore had already decided to replace it.

KMag 3y ago

And in later versions of Classic Mac OS, for each executable, metadata was stored on whether to set the hardware for 32-bit or 24-bit addressing. As a kid, I definitely monkeyed with the checkbox there not understanding what it did. Having seen no change in behavior, I guess all the applications I used were 32-bit clean.