My setup is quite unusual in that I VNC into Linux over Samsung DeX. After working around its annoying unchangeable keybindings, I remap the keys back as they s

My setup is quite unusual in that I VNC into Linux over Samsung DeX. After working around its annoying unchangeable keybindings, I remap the keys back as they should be on the Linux side with xmodmap. But then it seems to only change keycode and not keysym or something, so many programs don't respect that (vim included). I can't get rid of the muscle memory for Esc, no matter how much I tried to use Ctrl+[ or jj or similar, so I use xte to simulate a key press which is slooow when you're really going fast. If anyone has any ideas on how to get around this please lmk!