4 ms·
We've put rsync in a HPC scheduler and used it, or some tooling ontop of it really, to copy billions of files for a large-ish HPC compute cluster with many P of
by SEJeff 5y ago
We've put rsync in a HPC scheduler and used it, or some tooling ontop of it really, to copy billions of files for a large-ish HPC compute cluster with many P of data.
- gerdesj 5y agoThat's my experience - millions or billions of files - who cares? rsync is time served. It just works. I'm sure there are other funky solutions but they are not proven over decades.
- johnthescott 5y agowe have had problems with millions of rsunk files.