3 ms·

It looks interesting but one small thing. I looked into the transferring large files quickly link[1] and saw that they were using netcat and tar to transfer fil

by iuguy 14y ago

It looks interesting but one small thing. I looked into the transferring large files quickly link[1] and saw that they were using netcat and tar to transfer files. This is not necessarily optimal[2], and applying some compression can go a long way, although this will be dependent on the use case. Compression also has the added bonus of transferring less data across the network and (if you don't uncompress) less space at the other end. SSH is also an option (a slow option, but an option) that provides certain things (encryption, authentication) that make it ideally suited for transfers across network boundaries. [1] - http://engineering.tumblr.com/post/7658008285/efficiently-copying-files-to-multiple-destinations http://engineering.tumblr.com/post/7658008285/efficiently-co... [2] - http://www.ndchost.com/wiki/server-administration/netcat-over-ssh http://www.ndchost.com/wiki/server-administration/netcat-ove...

ralph 14y ago
They're using pigz on [1] so they are compressing? I thought it wasn't a great idea. OK for an on-the-fly solution to the problem but bittorrent or multicast would seem better; the serial route between machines isn't very fault tolerant requiring a start-from-scratch on failure. socat > nc BTW, and does multicast. As for ssh, it's a shame the "no encryption" option was removed.
- iuguy 14y ago
  Wow, I completely missed pigz in there for some reason and had to look it up. I agree with you on all counts though!
- evanelias 14y ago
  re: fault tolerance, it's a fair point. Although in practice I've never had this fail part-way on me, and I've used it a couple hundred times with >600GB transfers. We usually use this to copy to 2 or 3 machines at once; it's rare that we'd need to bring up 4+ slaves simultaneously, or split a shard into 4+ pieces. Most Linux distributions already have all the software needed except pigz, which is tiny and available in several packaging systems. I'll definitely give socat a look though, thanks for the tip.
  - ralph 14y ago
    Oh, OK, I agree, for 2-3 machines reliability isn't an issue, I was thinking more dozens.
deleted 14y ago
[deleted]

by iuguy 14y ago

ralph 14y ago

They're using pigz on [1] so they are compressing? I thought it wasn't a great idea. OK for an on-the-fly solution to the problem but bittorrent or multicast would seem better; the serial route between machines isn't very fault tolerant requiring a start-from-scratch on failure. socat > nc BTW, and does multicast. As for ssh, it's a shame the "no encryption" option was removed.

iuguy 14y ago
Wow, I completely missed pigz in there for some reason and had to look it up. I agree with you on all counts though!
evanelias 14y ago
re: fault tolerance, it's a fair point. Although in practice I've never had this fail part-way on me, and I've used it a couple hundred times with >600GB transfers. We usually use this to copy to 2 or 3 machines at once; it's rare that we'd need to bring up 4+ slaves simultaneously, or split a shard into 4+ pieces. Most Linux distributions already have all the software needed except pigz, which is tiny and available in several packaging systems. I'll definitely give socat a look though, thanks for the tip.
- ralph 14y ago
  Oh, OK, I agree, for 2-3 machines reliability isn't an issue, I was thinking more dozens.

deleted 14y ago

[deleted]