6 ms·

Don Libes' Expect: A Surprisingly Underappreciated Unix Automation Tool (2016)

104 points by dvfjsdhgfv 7y ago

Riverheart 7y ago
Expect is a great idea. Shout-out for the python version https://github.com/pexpect/pexpect https://github.com/pexpect/pexpect
- jmmcd 7y ago
  `pexpect` is used by Jupyter Notebook, and gets some maintainence from the Jupyter people. So, like the article argues, it is likely to be stable for the long term.
pmoriarty 7y ago
I've recently become a fan of expect-lite[1], which has a much simpler syntax. Here are some examples: Check the IP address of the localhost >ifconfig <192\.168\.1\.1 >ip -6 addr show <2001:db8::f00d ssh to a remote host >ssh root@host-021 <assword: >>secret_password # issue a command once logged in >ls >exit [1] - http://expect-lite.sourceforge.net/ http://expect-lite.sourceforge.net/
- erredois 7y ago
  Thank you for this, looks amazing to create simple tests
- ohithereyou 7y ago
  Inputting an SSH password using an expect script seems like an anti-pattern to me. In general, why can't you use a password-less key on the client and restrict what commands it can execute in the authorized_keys file on the server?
  - ebcode 7y ago
    ... because you don't control the server?
pram 7y ago
I used expect a ton a decade+ ago to script a bunch of ncurses-esque menus over a telnet connection. It was an obscure tool even then. I can’t think of much I’d use it for today. Ansible is far better. It was mostly helpful when you had to deal with a mainframe terminal program that was otherwise impossible to automate.
- vandal_at_your 7y ago
  Also surprised to see this at HN. The approach to automating interactive applications based on pty superpose (and I/O direction) coupled to a pattern->action model is an extremely old fashioned approach. 20 years ago expect was an invaluable tool though. The model for developing cli tools has changed so much in reaction that very few use cases for 'vanilla' expect remain. Not to mention that TCL is not a very popular PL and unlikely to make a comeback.
  - robertelder 7y ago
    'expect' is definitely for edge-case automation, but in the few cases where it is useful, it is extremely useful. Despite the fact that it's quite 'old fashioned', it is a dependency of DejaGnu which, itself, is a dependency of the test cases for a huge number of GNU projects. This leads me to believe that it'll still be around for a very long time and will continue to work the same as it does now.
    - shantly 7y ago
      I used it as recently as a couple years ago to make short work of some build + deployment stuff on some slightly unusual platform. Might've been Garmin wearables? Something like that. When you need "expect", it's wonderful.
    - theamk 7y ago
      Sure, but is it worth learning TCL for a few edge-cases? I like python, so I used "pexpect" in the past. I am sure other languages have such libraries too. And if you are going for solid production, you probably want to re-implement "expect" anyway -- so all input is whitelisted and unexpected messages are flagged. You probably don't what to break your expensive 1990's industrial device because you were driving it with expect script, and it cheerfully ignored "WARNING RESERVE BATTERY DEAD, REPLACE BEFORE POWER OFF" messages.
polm23 7y ago
I once used expect to brute force a puzzle in an interactive fiction game.
Stratoscope 7y ago
Fun fact: Don is the son of microcomputing pioneer Sol Libes: https://www.google.com/search?q=sol+libes https://www.google.com/search?q=sol+libes
- watersb 7y ago
  Thanks. I saw the name and immediately suspect that was so. Good times.
- balnaphone 7y ago
  Don Libes is also the author of "Obfuscated C Code and Other Mysteries", which is the first book I ever read that really explores all the dark corners of programming in C; it's a really great book if you use the C language. I used Expectk extensively in 1993, but I can't say I really enjoyed it. It was a crufty and error-prone undertaking, and in my opinion is best for testing, or as a last resort for talking to black-box code with an interactive cli.
thesuperbigfrog 7y ago
I use Perl for a lot of my scripts, so the Expect Perl module is great for automation without having to switch over to TCL. https://metacpan.org/pod/distribution/Expect/Expect.pod https://metacpan.org/pod/distribution/Expect/Expect.pod The regex matching features of the Perl version make it extremely useful to match and extract interactive results.
dang 7y ago
Discussed at the time: https://news.ycombinator.com/item?id=13130218 https://news.ycombinator.com/item?id=13130218
lilyball 7y ago
expect is used to run a bunch of interactive tests for the fish shell (https://github.com/fish-shell/fish-shell/tree/master/tests https://github.com/fish-shell/fish-shell/tree/master/tests, all of the files ending in .expect, where the file interactive.expect.rc defines a bunch of utilities).
zaroth 7y ago
Years ago I was automating control of 3rd party WiFi devices, many of which presented a console which would be a much more reliable means of controlling the device than trying to hack the web UIs of the time. I was using Tcl mostly running on Windows, and the Expect library for some reason had tremendous lag, or maybe it was a paid extension from ActiveState, I can’t remember the exact reason for not using it... So I decided to just reimplement my own ‘expect’ command using native Tcl and a socket with just enough knowledge of Telnet control codes to get a prompt to appear. What amazed me was how few lines of code it took to implement a fully functional Expect, and how useful it was as a means of structuring the automation, specifically around timing when to send inputs, parsing outputs, and confirming an action completed. I’ll try to dig up the old code and post it on Github...
davidgerard 7y ago
+1. expect is awesome and so useful.
- theamk 7y ago
  I am curious, what are you using it for?
amelius 7y ago
Some things that would be nice to have: support for mouse events, OCR of graphical applications (so e.g. you can program it to click on a text even if the location/font of the text isn't fixed), and support for DOM tree navigation in case you're interacting with a web page.
deleted 7y ago
[deleted]
theamk 7y ago
The reason that "expect" is underappreciated is right there in the text: > most of the things that you could use expect for, are usually done much better using another method. Outside of it's very narrow use case (driving un-scriptable CLIs and automated tests for command-line tools, and maybe some exotic CLI-centered workflows), trying to use "expect" will often result in fragile code with surprising failure mode. In 2nd example, script will fail if user has non-default top's config. It also messes up terminal on exit. Use top's "personal configuration file" instead. In the 4th example, you are not just trapping Ctrl-D in bash, you are trapping it in all bash-spawned commands (like "cat >file"). I am going to guess that bash's native IGNOREEOF option was what the author wanted. In the 9th example (md5sum) , expect'll work fine -- but learning standard unix commands like "grep -m1" and/or "tee /dev/stderr" will allow many things that expect cannot, like doing an md5sum on millions of small files.
Rich_Morin 7y ago
While running scripted `apt-get -y install ...` commands, I occasionally encounter packages that require answers to configuration questions. In some cases, the "-y" option handles the problem, but in others it does not. I've thought about going back to the developers of the packages, asking them to provide workarounds based on environment variables, etc. I may still do this, but it seems like Expect might provide a handy workaround in the meanwhile. However, I'm concerned that the mock-GUI dialogs that these packages use (using terminal graphics) might make it hard for Expect to interact with them. For example, some of these dialogs want the user to navigate through a menu, select an item, etc. Suggestions, anyone?
- jfim 7y ago
  I believe those are using whiptail, so maybe a modified version of it that allows scripting would do the trick.

104 points by dvfjsdhgfv 7y ago

Expect is a great idea. Shout-out for the python version https://github.com/pexpect/pexpect https://github.com/pexpect/pexpect

jmmcd 7y ago
`pexpect` is used by Jupyter Notebook, and gets some maintainence from the Jupyter people. So, like the article argues, it is likely to be stable for the long term.

pmoriarty 7y ago

I've recently become a fan of expect-lite[1], which has a much simpler syntax. Here are some examples: Check the IP address of the localhost >ifconfig <192\.168\.1\.1 >ip -6 addr show <2001:db8::f00d ssh to a remote host >ssh root@host-021 <assword: >>secret_password # issue a command once logged in >ls >exit [1] - http://expect-lite.sourceforge.net/ http://expect-lite.sourceforge.net/

erredois 7y ago
Thank you for this, looks amazing to create simple tests
ohithereyou 7y ago
Inputting an SSH password using an expect script seems like an anti-pattern to me. In general, why can't you use a password-less key on the client and restrict what commands it can execute in the authorized_keys file on the server?
- ebcode 7y ago
  ... because you don't control the server?

pram 7y ago

I used expect a ton a decade+ ago to script a bunch of ncurses-esque menus over a telnet connection. It was an obscure tool even then. I can’t think of much I’d use it for today. Ansible is far better. It was mostly helpful when you had to deal with a mainframe terminal program that was otherwise impossible to automate.

vandal_at_your 7y ago
Also surprised to see this at HN. The approach to automating interactive applications based on pty superpose (and I/O direction) coupled to a pattern->action model is an extremely old fashioned approach. 20 years ago expect was an invaluable tool though. The model for developing cli tools has changed so much in reaction that very few use cases for 'vanilla' expect remain. Not to mention that TCL is not a very popular PL and unlikely to make a comeback.
- robertelder 7y ago
  'expect' is definitely for edge-case automation, but in the few cases where it is useful, it is extremely useful. Despite the fact that it's quite 'old fashioned', it is a dependency of DejaGnu which, itself, is a dependency of the test cases for a huge number of GNU projects. This leads me to believe that it'll still be around for a very long time and will continue to work the same as it does now.
  - shantly 7y ago
    I used it as recently as a couple years ago to make short work of some build + deployment stuff on some slightly unusual platform. Might've been Garmin wearables? Something like that. When you need "expect", it's wonderful.
  - theamk 7y ago
    Sure, but is it worth learning TCL for a few edge-cases? I like python, so I used "pexpect" in the past. I am sure other languages have such libraries too. And if you are going for solid production, you probably want to re-implement "expect" anyway -- so all input is whitelisted and unexpected messages are flagged. You probably don't what to break your expensive 1990's industrial device because you were driving it with expect script, and it cheerfully ignored "WARNING RESERVE BATTERY DEAD, REPLACE BEFORE POWER OFF" messages.

polm23 7y ago

I once used expect to brute force a puzzle in an interactive fiction game.

Stratoscope 7y ago

Fun fact: Don is the son of microcomputing pioneer Sol Libes: https://www.google.com/search?q=sol+libes https://www.google.com/search?q=sol+libes

watersb 7y ago
Thanks. I saw the name and immediately suspect that was so. Good times.
balnaphone 7y ago
Don Libes is also the author of "Obfuscated C Code and Other Mysteries", which is the first book I ever read that really explores all the dark corners of programming in C; it's a really great book if you use the C language. I used Expectk extensively in 1993, but I can't say I really enjoyed it. It was a crufty and error-prone undertaking, and in my opinion is best for testing, or as a last resort for talking to black-box code with an interactive cli.

thesuperbigfrog 7y ago

I use Perl for a lot of my scripts, so the Expect Perl module is great for automation without having to switch over to TCL. https://metacpan.org/pod/distribution/Expect/Expect.pod https://metacpan.org/pod/distribution/Expect/Expect.pod The regex matching features of the Perl version make it extremely useful to match and extract interactive results.

dang 7y ago

Discussed at the time: https://news.ycombinator.com/item?id=13130218 https://news.ycombinator.com/item?id=13130218

lilyball 7y ago

expect is used to run a bunch of interactive tests for the fish shell (https://github.com/fish-shell/fish-shell/tree/master/tests https://github.com/fish-shell/fish-shell/tree/master/tests, all of the files ending in .expect, where the file interactive.expect.rc defines a bunch of utilities).

zaroth 7y ago

Years ago I was automating control of 3rd party WiFi devices, many of which presented a console which would be a much more reliable means of controlling the device than trying to hack the web UIs of the time. I was using Tcl mostly running on Windows, and the Expect library for some reason had tremendous lag, or maybe it was a paid extension from ActiveState, I can’t remember the exact reason for not using it... So I decided to just reimplement my own ‘expect’ command using native Tcl and a socket with just enough knowledge of Telnet control codes to get a prompt to appear. What amazed me was how few lines of code it took to implement a fully functional Expect, and how useful it was as a means of structuring the automation, specifically around timing when to send inputs, parsing outputs, and confirming an action completed. I’ll try to dig up the old code and post it on Github...

davidgerard 7y ago

+1. expect is awesome and so useful.

theamk 7y ago
I am curious, what are you using it for?

amelius 7y ago

Some things that would be nice to have: support for mouse events, OCR of graphical applications (so e.g. you can program it to click on a text even if the location/font of the text isn't fixed), and support for DOM tree navigation in case you're interacting with a web page.

deleted 7y ago

[deleted]

theamk 7y ago

The reason that "expect" is underappreciated is right there in the text: > most of the things that you could use expect for, are usually done much better using another method. Outside of it's very narrow use case (driving un-scriptable CLIs and automated tests for command-line tools, and maybe some exotic CLI-centered workflows), trying to use "expect" will often result in fragile code with surprising failure mode. In 2nd example, script will fail if user has non-default top's config. It also messes up terminal on exit. Use top's "personal configuration file" instead. In the 4th example, you are not just trapping Ctrl-D in bash, you are trapping it in all bash-spawned commands (like "cat >file"). I am going to guess that bash's native IGNOREEOF option was what the author wanted. In the 9th example (md5sum) , expect'll work fine -- but learning standard unix commands like "grep -m1" and/or "tee /dev/stderr" will allow many things that expect cannot, like doing an md5sum on millions of small files.

Rich_Morin 7y ago

While running scripted `apt-get -y install ...` commands, I occasionally encounter packages that require answers to configuration questions. In some cases, the "-y" option handles the problem, but in others it does not. I've thought about going back to the developers of the packages, asking them to provide workarounds based on environment variables, etc. I may still do this, but it seems like Expect might provide a handy workaround in the meanwhile. However, I'm concerned that the mock-GUI dialogs that these packages use (using terminal graphics) might make it hard for Expect to interact with them. For example, some of these dialogs want the user to navigate through a menu, select an item, etc. Suggestions, anyone?

jfim 7y ago
I believe those are using whiptail, so maybe a modified version of it that allows scripting would do the trick.