Only as long as the program in question doesn't touch the pointer values in weird ways, like storing metadata in otherwise unused bits or storing the pointers e

Only as long as the program in question doesn't touch the pointer values in weird ways, like storing metadata in otherwise unused bits or storing the pointers encoded as json values in some kind of global registry structure.