4 ms·
They're using pigz on [1] so they are compressing? I thought it wasn't a great idea. OK for an on-the-fly solution to the problem but bittorrent or multicast w
by ralph 14y ago
They're using pigz on [1] so they are compressing?
I thought it wasn't a great idea. OK for an on-the-fly solution to the problem but bittorrent or multicast would seem better; the serial route between machines isn't very fault tolerant requiring a start-from-scratch on failure.
socat > nc BTW, and does multicast.
As for ssh, it's a shame the "no encryption" option was removed.
- iuguy 14y agoWow, I completely missed pigz in there for some reason and had to look it up. I agree with you on all counts though!
- evanelias 14y agore: fault tolerance, it's a fair point. Although in practice I've never had this fail part-way on me, and I've used it a couple hundred times with >600GB transfers. We usually use this to copy to 2 or 3 machines at once; it's rare that we'd need to bring up 4+ slaves simultaneously, or split a shard into 4+ pieces. Most Linux distributions already have all the software needed except pigz, which is tiny and available in several packaging systems. I'll definitely give socat a look though, thanks for the tip.
- ralph 14y agoOh, OK, I agree, for 2-3 machines reliability isn't an issue, I was thinking more dozens.