Ask HN: What tools do you use to do GPU pooling

At my work we have three servers with four GPU’s each at the moment and we’re soon getting a fourth one. The problem we have now is that when we want to do experiments, we choose a server, download the datasets that are needed and then we perform the experiments. Recently we’ve run into some problems as some people have downloaded multiple GB’s of data on one server and once they’re ready to do the experiments all the GPU’s will be used on that machine. At the same time on other servers there are GPU’s being idle. What is being done now is just copying all your data from one server to the other (which might take days) and then hope you can use a GPU. Clearly this is not scaleable and we are interested to know if there are any ways to make a GPU pool where you can launch a job and the system also takes care of data management. Any links to tutorials are much appreciated.