7 ms·
Expect – Linux tool for automating interactive programs
- actinium226 2y agoclaude.ai just suggested this as a tool for automating a workflow that involves ssh'ing into another machine and running some commands as sudo. I'm surprised I've never heard of this tool before. It works great for this scenario and the script can be used a reference to go in manually.
- lo0dot0 2y agoWhat the hell. If the device you are logging into supports sudo then it probably supports bash or python too. In that case a better solution is to create a script on the target device and then call that script over ssh, using key-based authentication. You do not need expect. Expect is mainly used on devices with a rigid terminal UI that do not support scripting and require interactive password authentication.
- actinium226 2y agoThat's definitely do-able, but I didn't want to do that because it adds a level of indirection. When I put my relevant commands in a script, I have to start thinking about what directory it's in and how that affects its operations. I also have to think about how to handle errors for failures of the various commands, and I can only handle those errors on the target machine. If I'd doing expect and a command has an error, I can handle it on my local machine which might have more info (maybe I can set it up to switch to a backup machine or something). Are these solvable problems? Sure, for me it's a matter of taste. Also, your comment would read almost identically and perhaps even better without the first three words.
- gregjor 2y agoGood example of an LLM suggesting something useful, even if just RTFM for forgotten tools. O’Reilly put out a book about expect back in the old days (1995). I used expect a few times to automate remote admin, and even to scrape terminal-based applications.
- anthk 2y agoWhat? It's a manual page.
- actinium226 2y agoOP here, you must have missed my original comment where I said that claude.ai suggested this to me.
- thesuperbigfrog 2y agoMany years ago I used the Perl version of Expect (https://metacpan.org/pod/Expect https://metacpan.org/pod/Expect) to automate several workflows that wanted manual inputs. I thought that it was far better than the TCL Expect because I could also use other CPAN modules and have Perl's regular expressions to parse the system's output. Expect is a great tool to have in your toolkit.
- creer 2y ago> Perl version of Expect (https://metacpan.org/pod/Expect https://metacpan.org/pod/Expect) Still a great answer and latest update in March this year.
- overanalytcl 2y agoArguably, Tcl can do GUIs much easily and can integrate C less awkwardly than Perl. Both have their use cases, at any rate, and more people should use Tcl, Perl and any version of Expect. Nowadays it falls into the category of tools that were once somewhat used, but now are obscure, like fold/fmt, nl, comm and pr.
- thesuperbigfrog 2y ago>> Arguably, Tcl can do GUIs much easily and can integrate C less awkwardly than Perl. We used Perl/Tk for GUIs on that old system. It brimgs the ease of Tk GUIs to Perl: https://metacpan.org/dist/Tk/view/pod/UserGuide.pod https://metacpan.org/dist/Tk/view/pod/UserGuide.pod >> Nowadays it falls into the category of tools that were once somewhat used, but now are obscure To be sure, but a lot of it still works without changes!
- kevin_thibedeau 2y agoPerl/Tk has always been a red headed stepchild since it is a fork attempting to implement a C API that doesn't require Tcl to be linked. Because of that, it has long missed out on significant improvements to Tk over the years. Python Tkinter avoids that by just wrapping around a Tcl interpreter.
- pram 2y agoI used expect a ton years ago, for AIX system janitoring. A lot of things required telnet’ing into proto-ncurses mainframe applications and going through menus, typing and selecting stuff etc. The actual productive workflow is to do your task with autoexpect, and then refine the generated output: https://linux.die.net/man/1/autoexpect https://linux.die.net/man/1/autoexpect
- AtlasBarfed 2y agoI really rail to understand why modern unix OS scripting has not made expect obsolete. But then again bash is state of the art, right? Meta shells and meta meta shells should be an assumed feature by now, not a hit or miss of terminal program feature sets, and not tied to UI programs.
- bjt12345 2y agoTcl/Tk and Expect were popular back in the day because TCL could perform asynchronous programming most easily.
- firesteelrain 2y agoWe used pexpect to automate deployment from various servers and it would prompt users for passwords then they could login to perform deployment tasks for our software. It powered our entire deployment arch at that time. But it would frequently drop off text or prompts if you started doing nested logins. We rewrote our deployment arch since then
- jeffbee 2y agoExpect predates Linux by years. It is an old school portable program.
- lizknope 2y agoIt looks like Expect was first released in 1990 and Linux in 1991 although Linux wasn't really that usable until around 1993.
- jeffbee 2y agoI don't think it's unfair to say that they iterated on Linux until it could run expect :-)
- lizknope 2y agoYeah, I remember reading some posts around 1993 where someone wanted to port some existing Unix program to Linux. Instead of modifying the program they added the missing functionality to Linux. I think it was about polling and signal handling or something like that.
- foresto 2y agoLinux – UNIX-like tool for running expect and other portable software ;)
- shrubble 2y agoOne of the early papers by Don Libes, author of Expect, explaining how to use it; published 1990 - thus predating Linux: https://www.nist.gov/publications/using-expect-automate-system-administration-tasks https://www.nist.gov/publications/using-expect-automate-syst...
- philxor 2y agoLinux tool? Hate when everything Unix automatically gets categorized as Linux. Have used Expect for many years for many use cases from automating servers to networking use cases as well.
- deleted 2y ago[deleted]
- pokstad 2y agoI’ve used Expect on Windows systems. Active State provided commercial distributions of TCL for Windows.
- lizknope 2y agoI used Tcl and Expect on Solaris at a job in 1995
- Borborygymus 2y agoI used to use it on HP-UX in the 90s.
- muterad_murilax 2y agoPoor soul.
- dredmorbius 2y ago<https://news.ycombinator.com/item?id=41272562 https://news.ycombinator.com/item?id=41272562>
- 1vuio0pswjnm7 2y agoFor example, expect(1) is described in the 1989 Don Libes book "Life with UNIX". Libes is the author of Expect. A title such as "Life with Linux" would probably have made little sense at the time.
- 1vuio0pswjnm7 2y agoCorrection: I went back and looked at the book and it does not mention expect. According to Wikipedia expect was first released the following year, in 1990. Apologies for the inadvertence. Nevethless I think it is safe to say that expect was not written using or for Linux as Linux was not released until 1991.
- smitty1e 2y agoAs a default, I'd keep everything in one language: https://pypi.org/project/pyexpect/ https://pypi.org/project/pyexpect/
- synergy20 2y agopexpect seems to be much more popular though
- firesteelrain 2y agoI don’t think you understand nor used the program expect before Pyexpect is in another class of software. It is a Python testing utility that provides expressive assertions for unit testing. On the other hand, the Unix expect program, and its Python equivalent pexpect, are used for automating interactive applications.
- actinium226 2y agoI suppose he/she meant pexpect https://pypi.org/project/pexpect/ https://pypi.org/project/pexpect/
- smitty1e 2y agoThank you
- anotherpaulg 2y agoI just used python pexpect [0] for a fun project. I believe pexpect was inspired by the original Unix expect command. I used it to let aider suggest and run shell commands [1]. This is handy, so aider can offer to run the code it just wrote for you, run migrations, install dependencies, etc. Using pexepect makes it easy for aider to run interactive shell commands and capture their output. This allows aider to fix bugs or suggest next steps based on the command’s output, error messages, etc. [0] https://pexpect.readthedocs.io/en/stable/ https://pexpect.readthedocs.io/en/stable/ [1] https://x.com/paulgauthier/status/1827012590406607040 https://x.com/paulgauthier/status/1827012590406607040
- geekodour 2y agowait so if i understand you correctly, 1. aider makes some change 2. aider then tells itself to use pexpect to check what changes it made 3. pexpect runs, opens browser somehow captures error messages if any and feeds that back? i am pretty sure i misunderstood something
- anotherpaulg 2y agoThere’s a little demo video in the Twitter link that might be worth checking out. But ya, aider can: code something for you, run it, capture the terminal output with pexpect and iterate further based on that.
- firesteelrain 2y agoThe only problem we had with pexpect (maybe expect) was that we started to have prompts disappear or text not appear/be presented to our python script. Could have been an issue of buffers not flushing. It’s been several years but this one detail I remember because we had several workarounds for it.
- neoyagami 2y agoOne of the greatest tool for the shell, I womder how many jobs this tool help to keep
- nrclark 2y agoI use pexpect frequently for automating serial-port communications. It’s very useful for implementing test/flashing automation that needs some kind of a serial-port step (such as asking u-boot to run a few commands).
- PhilipRoman 2y agoI do something very similar, how do you deal with random serial port data loss (dropped bytes)? Maybe it is specific to my environment, but I get mangled commands about 1% of time, which really sucks for automated workflows. I mostly countered this by writing commands to a tmp file, verifying md5sum and then executing the file on the device side.
- orev 2y agoSerial is slow but typically very reliable. If you’re seeing random dropped bytes, I’d take a look at your wiring and PC interface UART which might be low quality. FTDI UARTs in particular are often counterfeit (the real ones are good).
- nrclark 2y agoAcross a large enough sample set, I definitely see occasional serial errors (although they're rare). For me, I detect them by checking the exit-code on each command that I run, and also by looking for expected output in response to the commands. Corrective action is usually a full-stage retry.
- broodbucket 2y agoexpect is one of those true love/hate things. Probably because the situations in which you need to use it are likely cursed. My time spent dealing with expect scripts is always miserable, and yet I don't know what I'd do without it
- Marthinwurer 2y agoI've mostly used it for testing and scripting CLIs. Are there any better ways to do that?
- photonthug 2y agotry using charm bracelets vhs tool to script the tests, then get the demo video as an animated gif for free.
- Borborygymus 2y ago> the situations in which you need to use it are likely cursed. So true, and yet so common.
- declan_roberts 2y agoWhat until you learn about the command "yes" -- now you're really automating with gas!
- nunez 2y agoLegendary tool. I used it once to automate an integration between Sailpoint IIQ and VMware's vRealize Automation.
- mongol 2y agoExpect used to be one of the "big" tools at one point. O'Reilly books written about it and so on. But then it went to obscurity. Kind of strange.
- mhitza 2y agoIt's a rare thing nowadays to find a terminal command that is interactive and doesn't provide alternative flags for batch processing. Expect would still be useful for someone scripting a debugger (for example), but Tcl falls under the close to isoteric language nowadays and it takes less time to just repeat the commands in a readline capable interactive program than find just the right way to script it. Last time I've tried to use expect was when I attempted to build a Tk frontend to the Haskell GHCi debugger (similar to the DDD application), but I got tripped up, from vague memory, on terminal ansi codes, and random lock ups I couldn't pinpoint.
- orev 2y agoSome tools like ssh and gpg go out of their way to prevent easy use via command line flags. They often have a good reason for doing this, but there are also sometimes good reasons where this type of protection causes problems. Since the implementers won’t budge on their stance, you need workarounds like expect.
- rwmj 2y agohttps://github.com/rwmjones/miniexpect https://github.com/rwmjones/miniexpect Miniexpect, a tiny C library that does something similar (for programs, not scripts).
- hiAndrewQuinn 2y agoMy go-to example for `expect` is password-based SSH. Since we're all enlightened beings these days who surely never use this anymore, there's no risk to it: #!/bin/bash expect -c " spawn ssh user@10.0.0.1 expect \"password:\" send \"12345\r\" interact " `interact` hands you back control to do whatever you want, but you don't actually need to do that. You can have `expect` run every command, fill out every field. The tool you need when you're in environments you don't deserve.
- quesera 2y agoFor this particular use case, I'd suggest sshpass instead. https://linux.die.net/man/1/sshpass ... Operations folks today might not understand -- everything just works nowadays, which is slightly boring. And boring is good! But there's something unique about the feeling of making things work even though they are not designed to work, and really have no right to work. Expect has been a lifesaver for some of the most critical systems at some of the biggest companies in the world. No aspersions cast by me!
- hiAndrewQuinn 2y agoDefinitely, `sshpass` is the more better keyhole tool for this job. I picked it as the most common example I can imagine everyone going "ugh, I have to do this by hand again?" at.
- oxygen_crisis 2y agosshpass is a travesty of security. It has all the pitfalls of putting your front door key under your doormat. Except you're doing it in a world where everyone can materialize a unique key out of thin air, and you can always instantly tell your door which of those keys should or shouldn't be allowed to open it. sshpass is a curse, even calling it a crutch would be improper flattery. It's a dangerous cheat that accomplishes nothing except impeding people from otherwise spending the 20 minutes it takes to figure out SSH keys.
- 2y ago
- noisy_boy 2y agoIs there an alternative to expect that I can just compile locally without having to deal with dependencies?
- bolapara 2y agoExpect is such a great tool for a specific class of problem. Decades ago I worked as a test engineer for a big hardware vendor. We had maybe a couple hundred raid controllers that we were testing and to address some issues we found, the supplier needed us to change a bunch of configuration variables that were only accessible via a text menu system over a serial connection to these devices. After painful manual reconfiguration of a handful of these things, I instead wrote a fairly complex expect script to make the appropriate changes and also validate all the settings on the rest of the raid controllers in a fraction of the time it would have taken to do it manually. It saved me so much hassle and got me some pretty good kudos and respect from my direct leads. It's one of those tools that I kind of wish I would need to use again!
- i_am_a_peasant 2y agothis is the sort of context I've also seen expect use. hardware test automation in manufacturing xD
- deleted 2y ago[deleted]
- ponow 2y agoNow add cross-platform GUI interaction for apps and web apps. [Ducks]
- i_am_a_peasant 2y agothe windows gui equivalent is AutoHotkey https://www.autohotkey.com/ https://www.autohotkey.com/
- nasso_dev 2y ago> Cause your computer to dial you back, so that you can login without paying for the call. oh. that's an OLD old tool..
- jdblair 2y agoexpect predates Linux!
- politelemon 2y ago> Cause your computer to dial you back, so that you can login without paying for the call. I don't understand what this means, but it sounds like an interesting piece of 'how we used to do things'?
- shrubble 2y agoIn the days of landline service, you have to pay for long distance calls; so it mattered which side made the call, as the originator of the call was charged. So you make a quick call to have the computer call you back and stay on for a long time; the number associated with the computer location is billed.
- totalhack 2y agoExpect was the first tool that really hooked me at my first job at Sun. I had to write a tool to automate verification of the boot process during firmware dev, handling variety of odd/unexpected behaviors, paths, and prompts. Many years later I heard they were still relying on that tool!
- sgarland 2y agoVery handy tool. I use it to partially automate logins on a CLI tool that have 2FA. Expect hooks op (1Pass’ CLI) to send the password, then tells the program what method of 2FA I want to use (Okta push to my phone).
- hiyer 2y agoSame here - fetch the password from mac's "security" command, and then select the 2fa method.
- coxley 2y agoExpect helped launch my early career in programming. Automating tedious tasks as a network engineer and sysadmin motivated me to learn proper software development. If gluing together TCL/tk scripts could make my life easier, imagine what larger programs would do. The good days.
- monetus 2y agoWhere the Tool Command Language really shines is this, commanding tools. TCL is criminally underrated for that use case.
- sctb 2y agoMacPorts is a noteworthy project that seems to use Tcl in its sweet spot.
- deleted 2y ago[deleted]
- 1vuio0pswjnm7 2y agoI find I am able to duplicate basic expect functionality by running tmux on the server and using tmux commands, namely tmux send and tmux capture.
- lcall 2y agoDejagnu, written in expect, is also useful for testing. I have used it in my TUI personal organizer project. https://en.wikipedia.org/wiki/Dejagnu https://en.wikipedia.org/wiki/Dejagnu