4 ms·

Recently I tried to use an sftp script created with expect that ran fine on the command line, but the same script failed to run under cron. I made sure that all

Recently I tried to use an sftp script created with expect that ran fine on the command line, but the same script failed to run under cron. I made sure that all environment variables were properly set, but sftp didn't ask me for a password. I think it might have been an issue with the absence of a tty sshpass didn't work, ended up rewriting the whole thing in Paramiko ... only to find out it doesn't respect the http_proxy environment variable. That was not a good day

rubicks 3y ago
Working with less-than-stellar tools --- ahem ROS --- has taught me how to placate commands that assume interactivity and/or a tty. To wrap up the offending command with a "fake" `tty`, I do script -qfec "mycommand" /dev/null If the thing insists on interactive input, then I break out the big guns: https://manpages.debian.org/bookworm/expect/expect.1.en.html https://manpages.debian.org/bookworm/expect/expect.1.en.html
- brewedDoritos 3y ago
  I've actually tried script and expect, but they didn't work. I had enabled the debug option on expect and I couldn't see the password prompt when the program ran under cron (i was redirecting the script output to a log file). It did appear when running on the prompt though. I couldn't figure how the sftp program was determining that it was running under cron. I suspect that it was inspecting if stdin was connected to a terminal or not, but I gave up around 4 am.
  - MobiusHorizons 3y ago
    Strace is your friend in such circumstances.
  - noAnswer 3y ago
    The other month I had to wrap a commercial server program with 'screen'. 'script' wasn't enough. If you would run it in the background "./program &" and then exited ssh it would also exit. It wouldn't run with cron until: export TERM=vt100 script -c "screen /foo/bar/program" /dev/null All the program did on the console was print to stdout. How are you even program shit like this!?
    - brewedDoritos 3y ago
      > If you would run it in the background "./program &" and then exited ssh it would also exit. Just wondering... have you tried running it with nohup and in the background?
      - noAnswer 3y ago
        No, I haven't tried that.
  - oasisaimlessly 3y ago
    It was probably checking for a 'controlling terminal' by e.g. opening /dev/tty.
    - fulafel 3y ago
      Script and expect provide the program a controlling terminal, using the same pty functionality that tmux and X11/Wayland terminal emulators use.
xorcist 3y ago
Avoid sftp and use rsync instead (much better support for more types of metadata, more robust error handling). Avoid passwords and use keys instead (easier to distribute, easier to generate, lets you lock them down to a single command). The above avoids much unnecessary thinking. If you really have to, there's always sshpass. And ssh -t to allocate a tty even if you are running without one. But this is seldom really necessary, first try harder do it the easy way.
- brewedDoritos 3y ago
  > Avoid sftp and use rsync instead I should have explored rsync better. I asked chatgpt some use cases and it made seem to be a bad fit because I needed to also delete some files on the destination machine. My prompt fu was probably a bit bad at the time. > Avoid passwords and use keys instead . The vendor only provides password authentication in this case > If you really have to, there's always sshpass. I´ve tried it, but sftp, when running under a cron script, detects that it is not running in interactive mode and does not issue the password prompt. The problem might have been caused by the TERM environment variable not being set as another reader suggested. > And ssh -t to allocate a tty even if you are running without one. But this is seldom really necessary I've used -o RequestTTY=force, but it also didn't work. Granted, it was close to 4 am and I might have missed a key aspect. > first try harder do it the easy way. Paramiko endep up being easier once I understood how to use the proxy command to interact with the company's http proxy.
  - xorcist 3y ago
    Read the man page instead of spending time on text generating tools. There is a batch mode in sftp which you should use. And sshpass will absolutely solve the lack of a missing tty in sftp. (Missing TERM is only a problem when you have a real tty, which you haven't.) Then again, don't do this. Use rsync. It is more robust and will avoid other problems in the future. And don't accept that a vendor only supports password-interactive authentication, it is unlikely to be the case, nobody implements their own ssh and every standard implementation of ssh accepts more ways of authentication. Paramiko is a perfectly workable solution, but it's way more complex and requires more maintenance for your eventual successor. Please don't be that guy.

rubicks 3y ago

Working with less-than-stellar tools --- ahem ROS --- has taught me how to placate commands that assume interactivity and/or a tty. To wrap up the offending command with a "fake" `tty`, I do script -qfec "mycommand" /dev/null If the thing insists on interactive input, then I break out the big guns: https://manpages.debian.org/bookworm/expect/expect.1.en.html https://manpages.debian.org/bookworm/expect/expect.1.en.html

brewedDoritos 3y ago
I've actually tried script and expect, but they didn't work. I had enabled the debug option on expect and I couldn't see the password prompt when the program ran under cron (i was redirecting the script output to a log file). It did appear when running on the prompt though. I couldn't figure how the sftp program was determining that it was running under cron. I suspect that it was inspecting if stdin was connected to a terminal or not, but I gave up around 4 am.
- MobiusHorizons 3y ago
  Strace is your friend in such circumstances.
- noAnswer 3y ago
  The other month I had to wrap a commercial server program with 'screen'. 'script' wasn't enough. If you would run it in the background "./program &" and then exited ssh it would also exit. It wouldn't run with cron until: export TERM=vt100 script -c "screen /foo/bar/program" /dev/null All the program did on the console was print to stdout. How are you even program shit like this!?
  - brewedDoritos 3y ago
    > If you would run it in the background "./program &" and then exited ssh it would also exit. Just wondering... have you tried running it with nohup and in the background?
    - noAnswer 3y ago
      No, I haven't tried that.
- oasisaimlessly 3y ago
  It was probably checking for a 'controlling terminal' by e.g. opening /dev/tty.
  - fulafel 3y ago
    Script and expect provide the program a controlling terminal, using the same pty functionality that tmux and X11/Wayland terminal emulators use.

xorcist 3y ago

Avoid sftp and use rsync instead (much better support for more types of metadata, more robust error handling). Avoid passwords and use keys instead (easier to distribute, easier to generate, lets you lock them down to a single command). The above avoids much unnecessary thinking. If you really have to, there's always sshpass. And ssh -t to allocate a tty even if you are running without one. But this is seldom really necessary, first try harder do it the easy way.

brewedDoritos 3y ago
> Avoid sftp and use rsync instead I should have explored rsync better. I asked chatgpt some use cases and it made seem to be a bad fit because I needed to also delete some files on the destination machine. My prompt fu was probably a bit bad at the time. > Avoid passwords and use keys instead . The vendor only provides password authentication in this case > If you really have to, there's always sshpass. I´ve tried it, but sftp, when running under a cron script, detects that it is not running in interactive mode and does not issue the password prompt. The problem might have been caused by the TERM environment variable not being set as another reader suggested. > And ssh -t to allocate a tty even if you are running without one. But this is seldom really necessary I've used -o RequestTTY=force, but it also didn't work. Granted, it was close to 4 am and I might have missed a key aspect. > first try harder do it the easy way. Paramiko endep up being easier once I understood how to use the proxy command to interact with the company's http proxy.
- xorcist 3y ago
  Read the man page instead of spending time on text generating tools. There is a batch mode in sftp which you should use. And sshpass will absolutely solve the lack of a missing tty in sftp. (Missing TERM is only a problem when you have a real tty, which you haven't.) Then again, don't do this. Use rsync. It is more robust and will avoid other problems in the future. And don't accept that a vendor only supports password-interactive authentication, it is unlikely to be the case, nobody implements their own ssh and every standard implementation of ssh accepts more ways of authentication. Paramiko is a perfectly workable solution, but it's way more complex and requires more maintenance for your eventual successor. Please don't be that guy.