4 ms·

My solution in these cases: ssh hostname tar cvjf - /path/to/folder | tar xjf - Basically I ask ssh to execute tar on the remote host to create a compress

My solution in these cases: ssh hostname tar cvjf - /path/to/folder | tar xjf - Basically I ask ssh to execute tar on the remote host to create a compressed archive. ssh will output the archive contents on the local host; this data is then passed on to a local tar for extraction.

susam 6y ago
I do mention in my blog post that the SSH gateway in between forbids execution of remote commands without a login shell, so this rules this out as the solution. Otherwise, yes, this would have been a fine solution.
- lmilcin 6y ago
  Honestly, I find it distasteful to have to spend time working around somebody's incompetence at securing systems. Doing it on your time means you delay delivering on your project and you let whoever did this get away with wasting everybody elses time.
  - JadeNB 6y ago
    > Honestly, I find it distasteful to have to spend time working around somebody's incompetence at securing systems. > Doing it on your time means you delay delivering on your project and you let whoever did this get away with wasting everybody elses time. While it may be distasteful, what's the alternative? Refusing on principle to use a system configured in a way you don't like is way more likely to hurt you than it is to hurt anyone else, especially the person who configured the system. Even if you are doing work for someone else (which, I think, is not indicated in the post), so that that person will be affected by your principled refusal, there's no guarantee that they're the ones who misconfigured the environment in which you're operating.
- JadeNB 6y ago
  Your patience in (re-)explaining this constraint to everyone in this thread who thinks you don't know your way around standard Unix tools is impressive. :-)
- AnotherGoodName 6y ago
  I went around a very similar issue by doing it in reverse. Rather than pull via SSH I login to the remote interactive shell and dump the file onto my own server via SSH from that remote shell.
- tssva 6y ago
  Would this make it through the SSH gateway? It creates a pseudo TTY and executes bash as a login shell within it. ssh hostname -qt 'bash -lc "tar czf - /path/to/folder 2>/dev/null | base64"' | base64 -id | tar xvzf -
  - wahern 6y ago
    Probably want to prepend a prefix so you can extract only those lines and filter banners, motd's, etc, like: "... base64 | sed -e 's/^/~/'" | sed -ne 's/^~//p' | base64 ...` BTW, base64 isn't a standard command (nonexistent on OpenBSD), and the options can vary based on implementation even when it exists (decode is -d for GNU coreutils base64, but -D for the macOS command). You can whip up an alternative using od(1) and printf(1): #!/bin/sh # encode as backslash-escaped octal (\0NNN) export LC_ALL=C od -An -to1 -v \ | sed -ne 's/$[01234567][01234567]*$/\\0\1/gp' \ | tr -cd '\\01234567\n' #!/bin/sh # decode backslash-escaped octal export LC_ALL=C while read -r L; do printf "%b" "$L" done It might not be strictly POSIX compliant, but I don't think there are any environments where printf isn't 8-bit clean in the C locale. I've tested something similar (see below) on various versions of bash and d?ash, as well as /bin/sh on AIX, FreeBSD, macOS, OpenBSD, NetBSD, and Solaris. You can implement base64 as a shell script using the above techniques, and do so without relying on awk, which isn't always available. If od(1) isn't available you can use dd(1) to extract stdin byte-by-byte and convert with printf(1), but that gets really slow. I've written an implementation that only needs an 8-bit-clean but otherwise POSIX compliant shell, printf, and either dd or od. It doesn't need tr, but it's even slower without it. It just converts the bytes to decimal and uses plain shell arithmetic for the primitive encoding and decoding, similar to how it would be done in any language.
    - tssva 6y ago
      It certainly isn't a complete cross platform solution but to address the inconsistent options problem and some of the availability issues openssl could be used instead. ... | openssl enc -base64" | openssl -base64 -d | ...
      - tssva 6y ago

susam 6y ago

I do mention in my blog post that the SSH gateway in between forbids execution of remote commands without a login shell, so this rules this out as the solution. Otherwise, yes, this would have been a fine solution.

lmilcin 6y ago
Honestly, I find it distasteful to have to spend time working around somebody's incompetence at securing systems. Doing it on your time means you delay delivering on your project and you let whoever did this get away with wasting everybody elses time.
- JadeNB 6y ago
  > Honestly, I find it distasteful to have to spend time working around somebody's incompetence at securing systems. > Doing it on your time means you delay delivering on your project and you let whoever did this get away with wasting everybody elses time. While it may be distasteful, what's the alternative? Refusing on principle to use a system configured in a way you don't like is way more likely to hurt you than it is to hurt anyone else, especially the person who configured the system. Even if you are doing work for someone else (which, I think, is not indicated in the post), so that that person will be affected by your principled refusal, there's no guarantee that they're the ones who misconfigured the environment in which you're operating.
JadeNB 6y ago
Your patience in (re-)explaining this constraint to everyone in this thread who thinks you don't know your way around standard Unix tools is impressive. :-)
AnotherGoodName 6y ago
I went around a very similar issue by doing it in reverse. Rather than pull via SSH I login to the remote interactive shell and dump the file onto my own server via SSH from that remote shell.
tssva 6y ago
Would this make it through the SSH gateway? It creates a pseudo TTY and executes bash as a login shell within it. ssh hostname -qt 'bash -lc "tar czf - /path/to/folder 2>/dev/null | base64"' | base64 -id | tar xvzf -
- wahern 6y ago
  Probably want to prepend a prefix so you can extract only those lines and filter banners, motd's, etc, like: "... base64 | sed -e 's/^/~/'" | sed -ne 's/^~//p' | base64 ...` BTW, base64 isn't a standard command (nonexistent on OpenBSD), and the options can vary based on implementation even when it exists (decode is -d for GNU coreutils base64, but -D for the macOS command). You can whip up an alternative using od(1) and printf(1): #!/bin/sh # encode as backslash-escaped octal (\0NNN) export LC_ALL=C od -An -to1 -v \ | sed -ne 's/$[01234567][01234567]*$/\\0\1/gp' \ | tr -cd '\\01234567\n' #!/bin/sh # decode backslash-escaped octal export LC_ALL=C while read -r L; do printf "%b" "$L" done It might not be strictly POSIX compliant, but I don't think there are any environments where printf isn't 8-bit clean in the C locale. I've tested something similar (see below) on various versions of bash and d?ash, as well as /bin/sh on AIX, FreeBSD, macOS, OpenBSD, NetBSD, and Solaris. You can implement base64 as a shell script using the above techniques, and do so without relying on awk, which isn't always available. If od(1) isn't available you can use dd(1) to extract stdin byte-by-byte and convert with printf(1), but that gets really slow. I've written an implementation that only needs an 8-bit-clean but otherwise POSIX compliant shell, printf, and either dd or od. It doesn't need tr, but it's even slower without it. It just converts the bytes to decimal and uses plain shell arithmetic for the primitive encoding and decoding, similar to how it would be done in any language.
  - tssva 6y ago
    It certainly isn't a complete cross platform solution but to address the inconsistent options problem and some of the availability issues openssl could be used instead. ... | openssl enc -base64" | openssl -base64 -d | ...
    - tssva 6y ago