Theoretically you can mount rclone to a local directory, but realistically it doesn't work because in the absence of serverside borg it is very heavy on network

Theoretically you can mount rclone to a local directory, but realistically it doesn't work because in the absence of serverside borg it is very heavy on network IO and slows to a crawl, even for sshfs over a short physical distance. To use it remotely, you really need to have borg installed on a server. I currently do as you describe.