4 ms·
Which lib/approach would you recommend if i need run a remote batch process using local files? For example, on the local system: remote-execute gcc localfile.
by so_tired 6y ago
Which lib/approach would you recommend if i need run a remote batch process using local files?
For example, on the local system:
remote-execute gcc localfile.c -o localfile.o
And on the remote system, there is a gcc and a remote agent which receives and retransmits the files
- toomuchtodo 6y agoSSH for job running and management, NFS for filesystem access. Have remote systems mount the local file system. If NFS isn’t available, have the remote job pull files over rsync from your local system, then rsync back the resulting output.
- so_tired 6y agoUgh! Thanks but no thanks I dont want to mount and expose nfs, or allow a full ssh login. This will be a custom agent/daemon on both ends
- deleted 6y ago[deleted]
- hnlmorg 6y agoI wouldn't recommend NFS over the open internet but there's a few options based around SFTP (rsync, scp, sshfs) and being based on SFTP means they can run without granting that user full SSH login access while still taking advantage of the security benefits that SSH brings. For job execution you could write your own agent but doing likely wouldn't be any more secure than SSH. Just make sure you have disabled password logins (use keys instead) and fail2ban or equivalent running to auto blacklist attacks. You could probably use Chef or SaltStack if really wanted to avoid a remote shell but if you're not already running config management then you have to ask yourself if you're over-engineering a solution. An alternative solution would be to run an OpenVPN tunnel and then you can SSH to your hearts content. But even here, unless you have multiple machines you want to connect to, I can't help thinking you're just making life harder for yourself without getting any realistic gains. This is all based on the very high level spec provided so I accept there might be some currently undisclosed detail that renders the above suggestions moot.
- anonymousiam 6y agoNFS works great over the open Internet, as long as you do it through a secure tunnel. I've been doing this for years as a way of increasing the size of the available storage in a VPS.
- joshbaptiste 6y agoYou run NFS within which secure tunneler?
- namibj 6y agoJust use Wireguard, tbh.
- hnlmorg 6y agoIt's really no different to running any other service over VPN or SSH tunnel. It does work well with NFSv3 but never tried with NFSv4.
- anonymousiam 6y agoI've used both vtun and WireGuard for remote NFS. On a good day, I get 100MB/s to the NFS filesystem on my San Jose AWS instance (from Vegas) via WireGuard. Note: That was before CoVid-19. CenturyLink/Quest has since (stealthily) throttled my bandwidth down from 1Gbps to ~750Mbps.
- calvinmorrison 6y agoYou would use rcpu(1) which essentially imports the remote cpu. In other words the remote processor "sees" the local file structure/namespace. 9front is full of "free carrots". If you want a remote vpn, just import another servers /net ontop of your process namespace and you're all set.