9 ms·
File Transfer with SSH, Tee, and Base64
- rsync 6y agoThis is a problem I used to solve with 'zmodem' which you can install on both ends and use to perform text transfers, solely over the interactive channel. I think the binaries were named 'zs' and 'zr'. It's been a long time since I needed to do this anywhere...
- Fnoord 6y agoZmodem, xmodem, kermit. Don't know which one's better etc. Last time I used one of these (don't remember which) was when my ISP booted me off cause of not paying bill, but minicom still worked, and gave me a shell. This was around 2005.
- JadeNB 6y agoBased on an offhand comment at https://www.extraputty.com/features/zmodem.html https://www.extraputty.com/features/zmodem.html, they were `sz` and `rz`. Wikipedia (https://en.wikipedia.org/wiki/ZMODEM https://en.wikipedia.org/wiki/ZMODEM) says that this came out of the BBS days. Boy, that takes me back.
- EvanAnderson 6y agoYou're close on the names of the binaries. They're 'sz' and 'rz'. I used the crap out of them dialing-in to systems back in the day. I had TELNET and FTP access to the Internet out of a dial-in to a shell on a Unix host in the mid 90s (no SLIP, PPP, or SLiRP from the provider at the time. It was great to be able to ftp down files to my home directory then 'sz' them back home.
- wahern 6y agoI wish terminal emulators still supported zmodem natively like some of the better emulators many years ago. It's not too difficult to configure iTerm2 to invoke rz when detecting the escape sequence[1], but it's still too much friction. sz/rz is something I would be using everyday, but I prefer sticking to Terminal.app on macOS--I prefer zero-config environments because of all the various systems, local and remote, I interact with on a daily basis. scp and sftp are great, but the nice thing about in-band file transfers is that your shell session already has the current context for simplifying the transfer. For example, you often want to send to or receive from the current working directory, or something nearby, and interactive file globbing is just a single tab key away, without having to switch sessions, login all over again, copy paths, etc. [1] I don't miss automatic printing without confirmation, though. The prank of `cat /vmlinuz | write user` wasn't that funny when the service desk hands you an entire ream of garbage along with a reprimand.
- ryancnelson 6y agoGnu screen supports zmodem send and receive. I use it frequently to get files into old retro computers I’m playing with. I’ve used it in datacenters where I’m ssh’d through three intermediate hosts, or when I’m inception-ing down into containers and vm’s Tweeted about zmodem just last night! https://twitter.com/ryancnelson/status/1306444057602711553 https://twitter.com/ryancnelson/status/1306444057602711553
- nrclark 6y agoThe instructions in this article should work, but they're more complicated than necessary maybe. If you don't have scp, you can usually just do something like this: ssh user@host cat remote_file > local_file to copy the contents of "remote_file" on the server to "local_file" on your local machine.
- Fnoord 6y ago> If you don't have scp [...] scp is deprecated [1]; use sftp instead. At the very least, sftp should be your preference over scp. [1] https://unix.stackexchange.com/questions/571293/is-scp-unsafe-should-it-be-replaced-with-sftp https://unix.stackexchange.com/questions/571293/is-scp-unsaf...
- nrclark 6y agoScp works fine, I use it all the time. It's less complicated than sftp, and works basically just like the regular 'cp' command. It also doesn't require an sftp server to be present on the remote machine. I'm not 100% sure I buy those "it's less secure" arguments in that Stack-exchange post, but even if true - lots of people use SSH to move stuff around inside an intranet, where both ends of the connection are trusted. In that kind of context, scp works great.
- tialaramex 6y agoWhich parts of the facts listed do you not "buy" ? The fact that a remote server gets to silently respond to the request "Hey, I'd like funny-cat-noises.mp3" by writing two files, funny-cat-noises.mp3 and also .security/overrides.conf ? Maybe your understanding of the protocol was that you allow a remote server to write whatever files it wants to your disk. That seems like a pretty crazy thing to want, but OK. Most people assume it does what it seems like it does. You name one or more files on one machine, and they're copied to the other machine. SFTP can deliver without nasty surprises, but SCP can't really do so, although the OpenSSH team have worked pretty hard to make the worst surprises less likely to bite you.
- waterhouse 6y agoIf you can SSH directly into both systems, then you don't need to print anything directly to the terminal. You can go: ssh system1 'cat payload' > temp cat temp | ssh system2 'cat > payload' Or cut out the middleman, as long as getting two password prompts isn't too confusing: ssh system1 'cat payload' | ssh system2 'cat > payload' I've found the base64/copy-paste trick useful when I can't do a conventional SSH. Usually this is when one of the machines is accessed via a serial console. In that scenario, it can help to compress the data before base64-ing it: machine1$ cat payload | gzip | base64 <output> machine2$ base64 -d | gzip -d > payload <paste output>
- vlovich123 6y agoThey mention that you can’t run commands without an interactive shell. I believe that knocks out your proposal because then rsync+ssh would work without all the complexity you have in your post with all the same benefits. Could be wrong though.
- waterhouse 6y agoI see, I did miss that. I do wonder if there's a workaround for that, though. "bash" has the "-l" option for "Make bash act as if it had been invoked as a login shell", and also "-i" forces interactive mode. For ssh, "-t" is "Force pseudo-terminal allocation"; would that work?
- deleted 6y ago[deleted]
- 6y ago
- deleted 6y ago[deleted]
- silviot 6y agoMy solution in these cases: ssh hostname tar cvjf - /path/to/folder | tar xjf - Basically I ask ssh to execute tar on the remote host to create a compressed archive. ssh will output the archive contents on the local host; this data is then passed on to a local tar for extraction.
- susam 6y agoI do mention in my blog post that the SSH gateway in between forbids execution of remote commands without a login shell, so this rules this out as the solution. Otherwise, yes, this would have been a fine solution.
- lmilcin 6y agoHonestly, I find it distasteful to have to spend time working around somebody's incompetence at securing systems. Doing it on your time means you delay delivering on your project and you let whoever did this get away with wasting everybody elses time.
- JadeNB 6y ago> Honestly, I find it distasteful to have to spend time working around somebody's incompetence at securing systems. > Doing it on your time means you delay delivering on your project and you let whoever did this get away with wasting everybody elses time. While it may be distasteful, what's the alternative? Refusing on principle to use a system configured in a way you don't like is way more likely to hurt you than it is to hurt anyone else, especially the person who configured the system. Even if you are doing work for someone else (which, I think, is not indicated in the post), so that that person will be affected by your principled refusal, there's no guarantee that they're the ones who misconfigured the environment in which you're operating.
- JadeNB 6y agoYour patience in (re-)explaining this constraint to everyone in this thread who thinks you don't know your way around standard Unix tools is impressive. :-)
- djent 6y agoThis seems like a CTF trick. If I were faced with this problem, I would do an SSH local port forward, and then just wget the file, hosted locally with python -m SimpleHTTPServer
- AnotherGoodName 6y agoOr you can open SSH back to your local port forwarded server from the remote SSH shell. At that point you can pipe whatever you want to/from your local server using the standard SSH commands.
- livre 6y agoI think we can assume that if they are restricting things as the article describes they also disabled port forwarding.
- AnotherGoodName 6y agoPort forwarding your local machine is always an option.
- susam 6y agoThat's right. See https://news.ycombinator.com/item?id=24507146 https://news.ycombinator.com/item?id=24507146 for example.
- susam 6y agoI mentioned in my blog post that there is an SSH gateway in between. I tried this on my local system: ssh -L 9000:127.0.0.1:8000 user@host But when I visit http://127.0.0.1:9000/ http://127.0.0.1:9000/ on my local system, I get a connection reset by peer error. At the same time, in the terminal where I have set up the port forward with the ssh command above, I get this error: open failed: unknown channel type The SSH gateway is forbidding port forwarding as well.
- lathiat 6y agoI know we're getting into the weeds here but for the benefit of people learning random things that work slightly differently (tm) A possible workaround for this specific scenario would be to use 'sshutle' which runs a python script on the remote end to terminate the traffic. Except in your specific scenario you'd have to somehow trick it to execute using a login shell like one of the other comments shows. Having said all of that, I would just do what you did :)
- anderiv 6y agoI just confirmed this can also work using AWS Session Manager (a "secure" and easily-auditable way to provide shell access to EC2 instances): $ aws ssm start-session --target i-XXXXXX | tee base64.txt
- jxy 6y agoIf the ssh gateway OP mentions is just a ssh server, a ProxyCommand should be used, and the gateway should be transparent after that. Host rcpu-tun User YOURLOGIN IdentityFile ~/.ssh/YOURSECRET ProxyCommand ssh -q -W HOSTBEHINDFW:22 -l PROXYLOGIN SSHGATEWAY HostName HOSTBEHINDFW
- deleted 6y ago[deleted]
- wahern 6y agossh -W requires that the gateway support port forwarding, which is probably disabled if its enforcing interactive logins. (-W also implicitly sets -T, though I don't know if that would fail the login or not.) Likewise, -J is shorthand for a ProxyCommand that runs nc on the gateway, which also won't work if the gateway requires interactive (i.e. TTY interposed) logins. (EDIT: Looking at the OpenSSH source code, -J is shorthand for generating a ProxyCommand using ssh -W. Either it was refactored to use -W after -W was added, or I'm misremembering how -J used to work.)
- jwilk 6y ago> sed '1,/$ base64/d;/$ exit/,$d' ssh.txt Huh. I'm surpised this works, because $ is not escaped.
- formerly_proven 6y agoSingle quotes.
- jwilk 6y agoNo, I mean $ usually means end-of-string (or end-of-line) in regexps. But TIL in BREs, $ means literal character in the middle of an expression: https://pubs.opengroup.org/onlinepubs/007904975/basedefs/xbd_chap09.html#tag_09_03_08 https://pubs.opengroup.org/onlinepubs/007904975/basedefs/xbd...
- formerly_proven 6y agoOh I see now, I never thought about $ being usable anywhere except at the end as an anchor!
- Johnny555 6y agoI do something similar to move small files via copy-and-paste to/from hosts that I have to reach through an intermediate jump host. Rather than copying the file to the jump host, then to the remote host, I'll just gzip and base64 encode it, then copy and paste it to the remote host.
- EvanAnderson 6y agoI've done this to get files into VMs that have poor or no network connectivity, too. If I can get a console session w/ the ability to paste keystrokes I'm golden... >smile<
- johnisgood 6y agoHeh, thought I was not alone! Although I do not use SSH, I just use netcat. Shared directories are a hassle. :D
- EvanAnderson 6y agoThat's a litmus test for me when working with other technicians. If they need to transfer a file to me across a shared LAN and they offer to pipe the file over Netcat (and they just expect that I know what they mean) then they "rate". If I suggest Netcat and they understand they "rate". If they have no idea what I'm talking about or, worse, suggest a "cloud" service, then I file them appropriately. Edit: Just to be clear, I'm talking about transient situations-- like when I'm working with another contractor on a day-long gig, etc. Obviously, for an established LAN piping files to each other over isn't a sensible or secure file transfer method.
- johnisgood 6y agoI do not know why you are being down-voted. I would expect people to know what netcat is, and what it is used for. If all someone can come up with is related to cloud, then I would dismiss them, too. These "problems" have been solved a very long time ago, and we have great alternatives. For myself I have written two shell functions; nc_send and nc_recv. It uses Nmap's ncat with SSL. It uses GPG and to verify data integrity, I use b2sum. Simple, yet perfect for the task. I use it often besides scp, but of course sometimes netcat is all I need.
- deleted 6y ago[deleted]
- katari_v 6y agoscp is so slow, does anyone want to improve this?