I use Unison every day to synchronize my work computer and my home computer, using an USB thumb drive as the intermediary. The thing I like the most about this

I use Unison every day to synchronize my work computer and my home computer, using an USB thumb drive as the intermediary. The thing I like the most about this arrangement is that I don't need a direct SSH between home and work, that there is no cloud service involved, and that the GUI lets me double check what files are going to be written to/from the usb drive before it synchronizes. There is one issue that I have run into though. If I format the USB as fat32 it can't properly store permissions. And if I format it as ext4 then I need to make sure that my numeric user ID is exactly the same in both my home and work computers (otherwise the permissions get messed up). Is there a way to make my arrangement more robust so that I can sync files through and USB drive, without needing to use the same numeric user id in all my computers?