You have a small cluster and you need cluster software for that. Look into Slurm (https://slurm.schedmd.com/ https://slurm.schedmd.com/) which is pretty standar

You have a small cluster and you need cluster software for that. Look into Slurm (https://slurm.schedmd.com/ https://slurm.schedmd.com/) which is pretty standard in HPC world. It allows you to manage ressources (like your individual GPUs), schedule jobs, copy codes and data to the particular machine when the job starts, and much more. It is pretty customizable. Having said that, of course a shared filesystem (NFS, better something like https://www.beegfs.io/ https://www.beegfs.io/ or https://ceph.com/ceph-storage/file-system/ https://ceph.com/ceph-storage/file-system/) together with a good network interconnect (Infiniband cards are expensive but worth their money if you transfer a lot of data) can save a lot of headache.