7 ms·

I like it! But for me ssh already provides a simple and secure way to move files from one place or another.. tar -cf - ./files.txt ./orDir/ | ssh host "(cd

by ktm5j 8y ago

I like it! But for me ssh already provides a simple and secure way to move files from one place or another.. tar -cf - ./files.txt ./orDir/ | ssh host "(cd /dest/dir; tar -xf -)" Given that ssh it so ubiquitous I think this will always be my go to.

Giroflex 8y ago
What about good old scp? scp file.txt user@host:/dest/dir/
- alex_young 8y ago
  Or rsync? rsync -zvh file.txt user@host:/dest/dir/ So many ways to do this :)
  - coatmatter 8y ago
    rsync has always been my favourite because it makes the most sense to me (and the --help/man page is easy to read). rsync -n -avh --progress source destination:~/asdf/ for a dry run followed by ctrl-p, ctrl-a, alt-f, alt-d, alt-d to remove the -n flag and then execute that for the real thing. Occasionally though, I'll also use sftp if I'm just pulling one thing - perhaps even after sshing to the remote machine. For all of these, SSH keys should be set up (and desktop logins secured) to make life easier. As for Android, adb push and adb pull -a seems to work better than mtp:// or AirDroid in my experience.
    - LukeShu 8y ago
      After all these years, I still can't keep straight when I need a trailing slash in rsync, and when I need to not have it.
      - coatmatter 8y ago
        If you think of it in terms of archives and whether you want to "extract" into the current directory, or a new directory within the current one; that might help. rsync source destination will plonk the entire source directory and put it inside destination as a neat bundle. rsync source/ destination will take the contents of source (but not the directory source itself) and plonk it in destination I found the info page a little dry but it does describe it succintly: rsync -av /src/foo /dest rsync -av /src/foo/ /dest/foo For some reason though, my head freaks out when it sees "foo" and "bar", but all they're saying is that it does the same thing. If in doubt though, just chuck everything into the destination ~/temp/ or ~/asdf/ and sort it out later. To be honest though, most of the time I just use fish shell's autosuggestions to guide me along.
- ktm5j 8y ago
  I usually go straight to tar & ssh over scp since you can do directories where you can't with scp. But yeah, if you just have some files scp works.
  - mijamo 8y ago
    You can obviously send directories with scp, just use -r. Or am I missing something?
    - charlchi 8y ago
      Recently used it to copy half a terabyte of stuff on my home network. Unsure about the exact specification but it supported the same flags as cp as far as I could tell.
    - codegeek 8y ago
      Yes -r works for copying directories. I do it all the time
    - ktm5j 8y ago
      Well I'll be damned.. not sure what lead me to think this.
- LukeShu 8y ago
  scp performs very slowly with directories containing many small files; "tar|ssh tar" does fine with that.
- zymhan 8y ago
  What if you don't have network and login access to the other person's computer?
reacweb 8y ago
If you can trust to have gnu tar on both side, you can reduce typing. I usually do: tar c files.txt orDir | ssh -C host tar x -C /dest/dir
- coatmatter 8y ago
  What about just typing rsync -havz --progress source destination? -h for human-readable numbers -a for archive mode -v and --progress for verbose info -z for compression during transfer Add a -n for a dry run if required. https://linux.die.net/man/1/rsync https://linux.die.net/man/1/rsync One of the people who came up with it - Andrew Tridgell - was more or less "responsible" for Linux and BitKeeper parting ways, which in turn ultimately lead to the creation of Git. I think it's a fascinating story. Excellent tools. The progress / speed display is the main thing that keeps me coming back to rsync even though other tools might manage the same job - not seeing the progress of a copy is what had me searching for that solution in the very first place.
  - phre4k 8y ago
    Yo dawg, I heard you like progress, so check out rsync's --info=progress2 You can use it since rsync version 3.1.0.
    - coatmatter 8y ago
      Hah I've seen that in the man a few times but always feel too lazy to call it up. Next time I'll give it another go!
apocalypstyx 8y ago
Reminds me of how they left recursions out of the 'cp' command on plan9, so the recommended way to copy directories is: @{cd fromdir && tar cp .} | @{cd todir && tar xT}
clord 8y ago
This particular tool seems useful for sharing files with people who might not be comfortable with the command-line, or where you don't have an account on the receiver's computer, since it produces a regular http link to download the data.
- ktm5j 8y ago
  That makes sense, I hadn't thought about that.
palebluedot 8y ago
> tar -cf - ./files.txt ./orDir/ | ssh host "(cd /dest/dir; tar -xf -)" You should use '&&' instead of ';' on the host side. That way you don't accidentally dump your transfer contents into a wrong directory if the existing host dir doesn't exist. e.g.: tar cf - stuff | ssh host "(cd /dest/dir && tar xf -)"
- mnarayan01 8y ago
  Or just skip the shell entirely: tar -C /dest/dir -xf -
  - ktm5j 8y ago
    Yeah that's the right way to do it, but I always forget which flag it is for tar and this is faster than pulling up the manpages ;)
  - reacharavindh 8y ago
    Could you give a full example of how this would work? Receiver and sender..
    - int0x80 8y ago
      tar cf - src/ | ssh $host "tar -C /dest/dir -xf -"
      - reacharavindh 8y ago
        Do you have any idea about a crazy fast way to transfer files between two machines on the internal network? No encryption necessary No integrity checks No resume support Just plain and as fast it could get.. I need to try the netcat version but I'm hoping someone could show me a concurrent version of it that is mad fast.
        int0x80 8y ago
        You can use nc (netcat). If you have a 10G ethernet you will not saturate it because you will be limited by disk IO (I get 1Gb/s disk reads for a M.2 SSD). I you can read from disk faster, netcat alone will not saturate a 10G link, probably will hit a 3Gb/s limit (depending on your hw). You will need parallel transfers (xargs, a bit of scripting etc). You can also try rsync, maybe it is good enough for you.
welly 8y ago
> ssh already provides a simple way :-D > tar -cf - ./files.txt ./orDir/ | ssh host "(cd /dest/dir; tar -xf -)" :-O
- bencoder 8y ago
  reminds me of this gem https://news.ycombinator.com/item?id=9224 https://news.ycombinator.com/item?id=9224
  - djsumdog 8y ago
    2007. Wow. That's a lot in Internet time.
    - lou1306 8y ago
      It is a lot in computer time as well. The other gem in that thread IMHO is how they marketed Dropbox as a replacement for USB thumb drives (which it also is, but it just goes to show how niche the concept of cloud was 11 years ago).
    - smellf 8y ago
      Yeah, and that guy is actually still active on HN and is always decent whenever someone calls him out about his ancient Dropbox comment.
djsumdog 8y ago
I have a lot of friends who don't know what the fuck SSH is. For them, these services are great at sending them files.
Sir_Cmpwn 8y ago
I use netcat a lot with my friends, we don't even have to do the SSH dance. On the receiving end: nc -vll 0.0.0.0 12345 | pv | tar xv On the sending end: tar cv files... | pv | nc -N "destination IP" 12345 pv is a nice tool that just reports on the rate and number of bytes moving through the pipe, optional if you don't have it. Throw in gpg --symmetric + gpg --decrypt for good measure if you want to encrypt it on the wire with a password.
- zymhan 8y ago
  That works assuming the client has a public IP address or port forwarding set up, or you're on a local network together.
- yjftsjthsd-h 8y ago
  That throws your files over the wire with no encryption, signing, or integrity verification. It might well be fine for your use, but I don't think I'd ever be comfortable with it.
  - Sir_Cmpwn 8y ago
    Did you miss the part of my comment where I explained how to add encryption?
    - yjftsjthsd-h 8y ago
      ...yep, I only read the command itself and skimmed right over that. My bad.
vram22 8y ago
>Ha ha, I remember writing a command like that (made it into a script for repeated use, with args) when one of our HP-UX servers in the company where I worked then, had a DAT drive failure. The script allowed us to take backups of the source code and data on one such box from another box (connected on the same network), until the bad drive was replaced a few days later. Didn't know about nc or rysnc at the time. Good to see those other solutions, and good that there are many ways to do it, with different pros and cons.

by ktm5j 8y ago

Giroflex 8y ago

What about good old scp? scp file.txt user@host:/dest/dir/

alex_young 8y ago
Or rsync? rsync -zvh file.txt user@host:/dest/dir/ So many ways to do this :)
- coatmatter 8y ago
  rsync has always been my favourite because it makes the most sense to me (and the --help/man page is easy to read). rsync -n -avh --progress source destination:~/asdf/ for a dry run followed by ctrl-p, ctrl-a, alt-f, alt-d, alt-d to remove the -n flag and then execute that for the real thing. Occasionally though, I'll also use sftp if I'm just pulling one thing - perhaps even after sshing to the remote machine. For all of these, SSH keys should be set up (and desktop logins secured) to make life easier. As for Android, adb push and adb pull -a seems to work better than mtp:// or AirDroid in my experience.
  - LukeShu 8y ago
    After all these years, I still can't keep straight when I need a trailing slash in rsync, and when I need to not have it.
    - coatmatter 8y ago
      If you think of it in terms of archives and whether you want to "extract" into the current directory, or a new directory within the current one; that might help. rsync source destination will plonk the entire source directory and put it inside destination as a neat bundle. rsync source/ destination will take the contents of source (but not the directory source itself) and plonk it in destination I found the info page a little dry but it does describe it succintly: rsync -av /src/foo /dest rsync -av /src/foo/ /dest/foo For some reason though, my head freaks out when it sees "foo" and "bar", but all they're saying is that it does the same thing. If in doubt though, just chuck everything into the destination ~/temp/ or ~/asdf/ and sort it out later. To be honest though, most of the time I just use fish shell's autosuggestions to guide me along.
ktm5j 8y ago
I usually go straight to tar & ssh over scp since you can do directories where you can't with scp. But yeah, if you just have some files scp works.
- mijamo 8y ago
  You can obviously send directories with scp, just use -r. Or am I missing something?
  - charlchi 8y ago
    Recently used it to copy half a terabyte of stuff on my home network. Unsure about the exact specification but it supported the same flags as cp as far as I could tell.
  - codegeek 8y ago
    Yes -r works for copying directories. I do it all the time
  - ktm5j 8y ago
    Well I'll be damned.. not sure what lead me to think this.
LukeShu 8y ago
scp performs very slowly with directories containing many small files; "tar|ssh tar" does fine with that.
zymhan 8y ago
What if you don't have network and login access to the other person's computer?

reacweb 8y ago

If you can trust to have gnu tar on both side, you can reduce typing. I usually do: tar c files.txt orDir | ssh -C host tar x -C /dest/dir

coatmatter 8y ago
What about just typing rsync -havz --progress source destination? -h for human-readable numbers -a for archive mode -v and --progress for verbose info -z for compression during transfer Add a -n for a dry run if required. https://linux.die.net/man/1/rsync https://linux.die.net/man/1/rsync One of the people who came up with it - Andrew Tridgell - was more or less "responsible" for Linux and BitKeeper parting ways, which in turn ultimately lead to the creation of Git. I think it's a fascinating story. Excellent tools. The progress / speed display is the main thing that keeps me coming back to rsync even though other tools might manage the same job - not seeing the progress of a copy is what had me searching for that solution in the very first place.
- phre4k 8y ago
  Yo dawg, I heard you like progress, so check out rsync's --info=progress2 You can use it since rsync version 3.1.0.
  - coatmatter 8y ago
    Hah I've seen that in the man a few times but always feel too lazy to call it up. Next time I'll give it another go!

apocalypstyx 8y ago

Reminds me of how they left recursions out of the 'cp' command on plan9, so the recommended way to copy directories is: @{cd fromdir && tar cp .} | @{cd todir && tar xT}

clord 8y ago

This particular tool seems useful for sharing files with people who might not be comfortable with the command-line, or where you don't have an account on the receiver's computer, since it produces a regular http link to download the data.

ktm5j 8y ago
That makes sense, I hadn't thought about that.

palebluedot 8y ago

> tar -cf - ./files.txt ./orDir/ | ssh host "(cd /dest/dir; tar -xf -)" You should use '&&' instead of ';' on the host side. That way you don't accidentally dump your transfer contents into a wrong directory if the existing host dir doesn't exist. e.g.: tar cf - stuff | ssh host "(cd /dest/dir && tar xf -)"

mnarayan01 8y ago
Or just skip the shell entirely: tar -C /dest/dir -xf -
- ktm5j 8y ago
  Yeah that's the right way to do it, but I always forget which flag it is for tar and this is faster than pulling up the manpages ;)
- reacharavindh 8y ago
  Could you give a full example of how this would work? Receiver and sender..
  - int0x80 8y ago
    tar cf - src/ | ssh $host "tar -C /dest/dir -xf -"
    - reacharavindh 8y ago
      Do you have any idea about a crazy fast way to transfer files between two machines on the internal network? No encryption necessary No integrity checks No resume support Just plain and as fast it could get.. I need to try the netcat version but I'm hoping someone could show me a concurrent version of it that is mad fast.
      - int0x80 8y ago
        You can use nc (netcat). If you have a 10G ethernet you will not saturate it because you will be limited by disk IO (I get 1Gb/s disk reads for a M.2 SSD). I you can read from disk faster, netcat alone will not saturate a 10G link, probably will hit a 3Gb/s limit (depending on your hw). You will need parallel transfers (xargs, a bit of scripting etc). You can also try rsync, maybe it is good enough for you.

welly 8y ago

> ssh already provides a simple way :-D > tar -cf - ./files.txt ./orDir/ | ssh host "(cd /dest/dir; tar -xf -)" :-O

bencoder 8y ago
reminds me of this gem https://news.ycombinator.com/item?id=9224 https://news.ycombinator.com/item?id=9224
- djsumdog 8y ago
  2007. Wow. That's a lot in Internet time.
  - lou1306 8y ago
    It is a lot in computer time as well. The other gem in that thread IMHO is how they marketed Dropbox as a replacement for USB thumb drives (which it also is, but it just goes to show how niche the concept of cloud was 11 years ago).
  - smellf 8y ago
    Yeah, and that guy is actually still active on HN and is always decent whenever someone calls him out about his ancient Dropbox comment.

djsumdog 8y ago

I have a lot of friends who don't know what the fuck SSH is. For them, these services are great at sending them files.

Sir_Cmpwn 8y ago

I use netcat a lot with my friends, we don't even have to do the SSH dance. On the receiving end: nc -vll 0.0.0.0 12345 | pv | tar xv On the sending end: tar cv files... | pv | nc -N "destination IP" 12345 pv is a nice tool that just reports on the rate and number of bytes moving through the pipe, optional if you don't have it. Throw in gpg --symmetric + gpg --decrypt for good measure if you want to encrypt it on the wire with a password.

zymhan 8y ago
That works assuming the client has a public IP address or port forwarding set up, or you're on a local network together.
yjftsjthsd-h 8y ago
That throws your files over the wire with no encryption, signing, or integrity verification. It might well be fine for your use, but I don't think I'd ever be comfortable with it.
- Sir_Cmpwn 8y ago
  Did you miss the part of my comment where I explained how to add encryption?
  - yjftsjthsd-h 8y ago
    ...yep, I only read the command itself and skimmed right over that. My bad.

vram22 8y ago

>Ha ha, I remember writing a command like that (made it into a script for repeated use, with args) when one of our HP-UX servers in the company where I worked then, had a DAT drive failure. The script allowed us to take backups of the source code and data on one such box from another box (connected on the same network), until the bad drive was replaced a few days later. Didn't know about nc or rysnc at the time. Good to see those other solutions, and good that there are many ways to do it, with different pros and cons.