I do something similar, so I know it's feasible. I whitelist the dotfiles that I want to keep hold of (.vim, .bashrc, &c.) in my homefolder and whenever I updat

I do something similar, so I know it's feasible. I whitelist the dotfiles that I want to keep hold of (.vim, .bashrc, &c.) in my homefolder and whenever I update something, I just push to the remote. It's really enlightening to see just how quickly a perfectly clean home folder can get filled with hidden directories.