This is what, for example, Android's ART does. On 64-bit platforms all the JVM references remained 32-bit. They know the heap base address so they can trivially

This is what, for example, Android's ART does. On 64-bit platforms all the JVM references remained 32-bit. They know the heap base address so they can trivially expand the references out on deref, saving the memory overhead. But in this case you can't do it entirely in userspace because things like mmap exist. You get pointers from syscalls all the time, and those can't really be compressed down since there's not a single fixed base address to add onto them. You'd have to use a different type to represent malloc'd memory that is conforming to 32-bit compressed references, but that gets real messy real fast.