9 ms·

Expect – A tool for automating interactive applications

104 points by networked 12y ago

rikkus 12y ago
I remember this from writing chat(8) scripts. Pretty nice little language!
- lowlevel 12y ago
  Yeah, me too. I don't think I've needed anything like this since... I don't understand why it's news all of a sudden.
agumonkey 12y ago
I remember seeing an elisp equivalent used to write ftp clients in emacs. Can't find it again, but anyway, I never got to see the original expect.
hcal 12y ago
I used to use expect to automate a bunch of business reports and system functions. I switched to a python work-alike called pexpect that I still use and love. It feels a little hacky but it works and saves a huge amount of time over the course of a year.
iaskwhy 12y ago
I use it to install MySQL with some non default values. I'm not that good with bash and install scripts and expect was the only way around some issues I was having. secure_mysql_script=$(expect -c ' spawn sudo mysql_secure_installation expect "Enter current password for root (enter for none):" send "\r" expect "Change the root password?" send "n\r" expect "Remove anonymous users?" send "y\r" expect "Disallow root login remotely?" send "y\r" expect "Remove test database and access to it?" send "y\r" expect "Reload privilege tables now?" send "y\r" expect eof ') echo "$secure_mysql_script"
ff_ 12y ago
It's wonderful for rapidly SSHing into a bunch of machines for doing automated things. It's a fast hacking provisioning tool, I really like it for this purpose.
thyrsus 12y ago
Be sure to check out "autoexpect": it starts an interactive session, and generates an expect script based on that. Then you typically need to edit it for generalization (e.g., skip dates and time strings; often skip version strings, etc.).
- zackmorris 12y ago
  How did I not know about this? Autoexpect is one of the thousand ways of interacting with computers that we USED to do back in the 80s and 90s that have since fallen out of favor. Macros, modem dialers, folder watchers, etc.. Remember trying the record button in Applecript Editor, only to find that it didn't actually work like it did in Macro Maker? The fact that autoexpect isn't installed by default on Mac OS blows my mind. Edit: still trying to install autoexpect on Mac OS 10.9.2 and having no luck. Tried ActiveTcl at http://www.activestate.com/activetcl/downloads http://www.activestate.com/activetcl/downloads but it doesn't seem to have autoexpect. Tried brew install autoexpect, tried port install autoexpect, nothing. Sometimes unix reminds me of looking up hints in video game magazines upon finding oneself stuck, so if someone knows how to get autoexpect on Mac, please let us know because I'm throwing in the towel.
  - deleted 12y ago
    [deleted]
  - networked 12y ago
    Autoexpect comes bundled with Expect (it is an Expect script itself). Try brew install expect
jpgvm 12y ago
I have been using a Ruby expect-like called Greenletters by Avdi Grimm, introduced on his blog here: http://devblog.avdi.org/2010/07/19/greenletters-painless-automation-and-testing-for-command-line-applications/ http://devblog.avdi.org/2010/07/19/greenletters-painless-aut... It's available via rubygems or github: https://github.com/avdi/greenletters https://github.com/avdi/greenletters
nnnnni 12y ago
expect is one of the most useful system administration tools out there! I've used it to batch-modify settings on 100 machines where I couldn't just scp in a new file due to each one needing small variations. If you don't know expect (and you're a sysadmin), learn it.
waterside81 12y ago
I love expect. We use it in upstart scripts. Little known gem of the unix world.
philbarr 12y ago
Has anyone used Expect for Windows? Any good experiences to share? I'm wondering if it's something I should take the time to learn and use.
- jasomill 12y ago
  Expect for Windows has been finicky for me every time I've tried using it, but Cygwin Expect works well, both for interacting with other Cygwin processes (not arbitrary Win32 console apps!), and for interacting with remote machines via (Cygwin) Telnet and SSH.
danielweber 12y ago
Wow, a blast from the past. I still have some expect tools I wrote long ago that I use and maintain to this day. Not so much new stuff since scripting languages have gotten better at doing what expect does -- often calling their modules "expect" and using the exact same syntax. I once wrote a webserver in Expect. No, it's not a good language for writing a webserver.
- davidw 12y ago
  Tcl is a perfectly fine language for writing web servers with, actually. It was possible to do a nice, lightweight, evented web server years before Node.js showed up. AOLserver is also done with Tcl, and for a while in the 90ies, was one of the best systems for doing dynamic web sites.
  - mcguire 12y ago
    My understanding was that AOLserver was the NCSA (pre-Apache httpd) server, with TCL as an extension language, not that AOLserver was written in TCL. Weird.
    - davidw 12y ago
      Yes, it's a mixture of the two, which makes eminent sense if you consider the speed of computers 20 years ago. It's Tcl, by the way, not TCL.
      - VBprogrammer 12y ago
        Having spent time working on a million line Tcl codebase (for my sins) I can assure you that it is possible, but not in the least bit pleasant!
        davidw 12y ago
        That I can see. But I can also see a lot of today's hip things, like Node.js, not being terribly fun either at that kind of size.
throwaway_3424 12y ago
It's very frustrating to see something like this that claims to be very useful, but shows no examples on the page. >Expect is a tool for automating interactive applications such as telnet, ftp, passwd, fsck, rlogin, tip, etc. Expect really makes this stuff trivial. How? >Expect can make easy all sorts of tasks that are prohibitively difficult with anything else. You will find that Expect is an absolutely invaluable tool - using it, you will be able to automate tasks that you've never even thought of before - and you'll be able to do this automation quickly and easily. HOW? Ah, examples... >http://expect.sourceforge.net/#examples http://expect.sourceforge.net/#examples It's just a giant README. This is frustrating.
- klibertp 12y ago
  There are nice examples on wikipedia: http://en.wikipedia.org/wiki/Expect http://en.wikipedia.org/wiki/Expect.
- nfriedly 12y ago
  Here's an example: https://github.com/nfriedly/Meteor-ODB-II/blob/master/deploy/deploy.sh https://github.com/nfriedly/Meteor-ODB-II/blob/master/deploy... This script runs the meteor deploy command as part of my CI loop every time new code gets pushed to github. The deploy command expects me to type in a bunch of information, and expect does that for me.
pielud 12y ago
For python, there's pexpect: http://pexpect.readthedocs.org/en/latest/ http://pexpect.readthedocs.org/en/latest/
- zwischenzug 12y ago
  I used pexpect as a base technology for ShutIt: http://ianmiell.github.io/shutit/ http://ianmiell.github.io/shutit/ it really is 95% of tcl's expect functionality, much easier to debug and understand. I'm grateful it exists, and it's much under-used I think.
  - emmelaich 12y ago
    Nice. I have a somewhat similar tool but it doesn't do much besides check for password expiry and do password changes. It uses pexpect but also multiprocessing and multiprocessing.Queue. I built most of it before we started using Ansible at work, but it is still useful in those places where Ansible is clumsy.
    - zwischenzug 12y ago
      Interesting - where is Ansible clumsy?
      - emmelaich 12y ago
        (just noticed this) I was just sort of echoing your annoyed by the obfuscation and indirection of ... Ansible's prime raison d'être is not remote execution but configuration management.
  - rPawel 12y ago
    It was great to see ShutIt during the last Docker London!
    - zwischenzug 12y ago
      Thanks! A previous similar talk is the first link here: https://www.youtube.com/user/ianmiell https://www.youtube.com/user/ianmiell
seqizz 12y ago
Here is an alternative I use. http://empty.sourceforge.net/ http://empty.sourceforge.net/
SEJeff 12y ago
Oh expect. I've got a crappy little script I wrote when I was a sysadmin at Ticketmaster used to change a lot of our out of band management card passwords: http://www.digitalprognosis.com/opensource/scripts/change_pass.exp http://www.digitalprognosis.com/opensource/scripts/change_pa... It might be useful to someone else.
jlturner 12y ago
We used Expect recently to automate iOS runtime hacking (ssh to jailbroken iOS device, run cycript to inject code, profit). One of those old Unix tools (built on tcl) that's very handy for automation.
sseal 12y ago
Love Expect, simple yet so functional.
fintler 12y ago
Here's a neat integration test suite written in expect for an HPC job launcher. Diving into this was my introduction to expect. It's a very practical little tool. https://github.com/SchedMD/slurm/tree/master/testsuite/expect https://github.com/SchedMD/slurm/tree/master/testsuite/expec...
cordite 12y ago
I found the disclaimer interesting.
daveloyall 12y ago
Latest changes: May 1st, 2014 according to the changelog: http://expect.cvs.sourceforge.net/viewvc/expect/expect/ChangeLog?revision=5.126 http://expect.cvs.sourceforge.net/viewvc/expect/expect/Chang... Also, you can apt-get install `expect`.
spektom 12y ago
Oh, this is great old tool! :) Here's PHP extension for Expect that I've created back in 2005: http://pecl.php.net/package/expect http://pecl.php.net/package/expect
npsimons 12y ago
Expect is one of the few reliable ways I've found to automate crufty, closed source things. Say, like a perforce client pull that uses a caching proxy so that the proxy is up to date for actual human pulls.
younata 12y ago
I once wrote a very minimal IRC client by essentially wrapping telnet around expect. All it did was automate sending out PINGs, but it worked.
Crito 12y ago
Expect is great, however if you ever determine that using it is the best way to solve your problem, then you are probably approaching the problem the wrong way in the first place (the best way to proceed from that point may very well be to continue using expect though). I used it a few years ago to create a test suite for some dedicated VoIP hardware (controlling the boxes over telnet). It worked great, but if I had had more time, that definitely would not have been my approach.
ZoFreX 12y ago
I first discovered this about two months ago. It may be old, but it solved a problem we had and nothing else quite fit the job. We're using it to run a flaky test suite and count how many times it succeeds, fails, or times out (yes, times out - yay Android!). Very important when you're trying to determine if potential stability improvements actually make a difference or not, and you need to run the 3 minute suite ten times at least to get something statistically significant.

104 points by networked 12y ago

rikkus 12y ago

I remember this from writing chat(8) scripts. Pretty nice little language!

lowlevel 12y ago
Yeah, me too. I don't think I've needed anything like this since... I don't understand why it's news all of a sudden.

agumonkey 12y ago

I remember seeing an elisp equivalent used to write ftp clients in emacs. Can't find it again, but anyway, I never got to see the original expect.

hcal 12y ago

I used to use expect to automate a bunch of business reports and system functions. I switched to a python work-alike called pexpect that I still use and love. It feels a little hacky but it works and saves a huge amount of time over the course of a year.

iaskwhy 12y ago

I use it to install MySQL with some non default values. I'm not that good with bash and install scripts and expect was the only way around some issues I was having. secure_mysql_script=$(expect -c ' spawn sudo mysql_secure_installation expect "Enter current password for root (enter for none):" send "\r" expect "Change the root password?" send "n\r" expect "Remove anonymous users?" send "y\r" expect "Disallow root login remotely?" send "y\r" expect "Remove test database and access to it?" send "y\r" expect "Reload privilege tables now?" send "y\r" expect eof ') echo "$secure_mysql_script"

ff_ 12y ago

It's wonderful for rapidly SSHing into a bunch of machines for doing automated things. It's a fast hacking provisioning tool, I really like it for this purpose.

thyrsus 12y ago

Be sure to check out "autoexpect": it starts an interactive session, and generates an expect script based on that. Then you typically need to edit it for generalization (e.g., skip dates and time strings; often skip version strings, etc.).

zackmorris 12y ago
How did I not know about this? Autoexpect is one of the thousand ways of interacting with computers that we USED to do back in the 80s and 90s that have since fallen out of favor. Macros, modem dialers, folder watchers, etc.. Remember trying the record button in Applecript Editor, only to find that it didn't actually work like it did in Macro Maker? The fact that autoexpect isn't installed by default on Mac OS blows my mind. Edit: still trying to install autoexpect on Mac OS 10.9.2 and having no luck. Tried ActiveTcl at http://www.activestate.com/activetcl/downloads http://www.activestate.com/activetcl/downloads but it doesn't seem to have autoexpect. Tried brew install autoexpect, tried port install autoexpect, nothing. Sometimes unix reminds me of looking up hints in video game magazines upon finding oneself stuck, so if someone knows how to get autoexpect on Mac, please let us know because I'm throwing in the towel.
- deleted 12y ago
  [deleted]
- networked 12y ago
  Autoexpect comes bundled with Expect (it is an Expect script itself). Try brew install expect

jpgvm 12y ago

I have been using a Ruby expect-like called Greenletters by Avdi Grimm, introduced on his blog here: http://devblog.avdi.org/2010/07/19/greenletters-painless-automation-and-testing-for-command-line-applications/ http://devblog.avdi.org/2010/07/19/greenletters-painless-aut... It's available via rubygems or github: https://github.com/avdi/greenletters https://github.com/avdi/greenletters

nnnnni 12y ago

expect is one of the most useful system administration tools out there! I've used it to batch-modify settings on 100 machines where I couldn't just scp in a new file due to each one needing small variations. If you don't know expect (and you're a sysadmin), learn it.

waterside81 12y ago

I love expect. We use it in upstart scripts. Little known gem of the unix world.

philbarr 12y ago

Has anyone used Expect for Windows? Any good experiences to share? I'm wondering if it's something I should take the time to learn and use.

jasomill 12y ago
Expect for Windows has been finicky for me every time I've tried using it, but Cygwin Expect works well, both for interacting with other Cygwin processes (not arbitrary Win32 console apps!), and for interacting with remote machines via (Cygwin) Telnet and SSH.

danielweber 12y ago

Wow, a blast from the past. I still have some expect tools I wrote long ago that I use and maintain to this day. Not so much new stuff since scripting languages have gotten better at doing what expect does -- often calling their modules "expect" and using the exact same syntax. I once wrote a webserver in Expect. No, it's not a good language for writing a webserver.

davidw 12y ago
Tcl is a perfectly fine language for writing web servers with, actually. It was possible to do a nice, lightweight, evented web server years before Node.js showed up. AOLserver is also done with Tcl, and for a while in the 90ies, was one of the best systems for doing dynamic web sites.
- mcguire 12y ago
  My understanding was that AOLserver was the NCSA (pre-Apache httpd) server, with TCL as an extension language, not that AOLserver was written in TCL. Weird.
  - davidw 12y ago
    Yes, it's a mixture of the two, which makes eminent sense if you consider the speed of computers 20 years ago. It's Tcl, by the way, not TCL.
    - VBprogrammer 12y ago
      Having spent time working on a million line Tcl codebase (for my sins) I can assure you that it is possible, but not in the least bit pleasant!
      - davidw 12y ago
        That I can see. But I can also see a lot of today's hip things, like Node.js, not being terribly fun either at that kind of size.

throwaway_3424 12y ago

It's very frustrating to see something like this that claims to be very useful, but shows no examples on the page. >Expect is a tool for automating interactive applications such as telnet, ftp, passwd, fsck, rlogin, tip, etc. Expect really makes this stuff trivial. How? >Expect can make easy all sorts of tasks that are prohibitively difficult with anything else. You will find that Expect is an absolutely invaluable tool - using it, you will be able to automate tasks that you've never even thought of before - and you'll be able to do this automation quickly and easily. HOW? Ah, examples... >http://expect.sourceforge.net/#examples http://expect.sourceforge.net/#examples It's just a giant README. This is frustrating.

klibertp 12y ago
There are nice examples on wikipedia: http://en.wikipedia.org/wiki/Expect http://en.wikipedia.org/wiki/Expect.
nfriedly 12y ago
Here's an example: https://github.com/nfriedly/Meteor-ODB-II/blob/master/deploy/deploy.sh https://github.com/nfriedly/Meteor-ODB-II/blob/master/deploy... This script runs the meteor deploy command as part of my CI loop every time new code gets pushed to github. The deploy command expects me to type in a bunch of information, and expect does that for me.

pielud 12y ago

For python, there's pexpect: http://pexpect.readthedocs.org/en/latest/ http://pexpect.readthedocs.org/en/latest/

zwischenzug 12y ago
I used pexpect as a base technology for ShutIt: http://ianmiell.github.io/shutit/ http://ianmiell.github.io/shutit/ it really is 95% of tcl's expect functionality, much easier to debug and understand. I'm grateful it exists, and it's much under-used I think.
- emmelaich 12y ago
  Nice. I have a somewhat similar tool but it doesn't do much besides check for password expiry and do password changes. It uses pexpect but also multiprocessing and multiprocessing.Queue. I built most of it before we started using Ansible at work, but it is still useful in those places where Ansible is clumsy.
  - zwischenzug 12y ago
    Interesting - where is Ansible clumsy?
    - emmelaich 12y ago
      (just noticed this) I was just sort of echoing your annoyed by the obfuscation and indirection of ... Ansible's prime raison d'être is not remote execution but configuration management.
- rPawel 12y ago
  It was great to see ShutIt during the last Docker London!
  - zwischenzug 12y ago
    Thanks! A previous similar talk is the first link here: https://www.youtube.com/user/ianmiell https://www.youtube.com/user/ianmiell

seqizz 12y ago

Here is an alternative I use. http://empty.sourceforge.net/ http://empty.sourceforge.net/

SEJeff 12y ago

Oh expect. I've got a crappy little script I wrote when I was a sysadmin at Ticketmaster used to change a lot of our out of band management card passwords: http://www.digitalprognosis.com/opensource/scripts/change_pass.exp http://www.digitalprognosis.com/opensource/scripts/change_pa... It might be useful to someone else.

jlturner 12y ago

We used Expect recently to automate iOS runtime hacking (ssh to jailbroken iOS device, run cycript to inject code, profit). One of those old Unix tools (built on tcl) that's very handy for automation.

sseal 12y ago

Love Expect, simple yet so functional.

fintler 12y ago

Here's a neat integration test suite written in expect for an HPC job launcher. Diving into this was my introduction to expect. It's a very practical little tool. https://github.com/SchedMD/slurm/tree/master/testsuite/expect https://github.com/SchedMD/slurm/tree/master/testsuite/expec...

cordite 12y ago

I found the disclaimer interesting.

daveloyall 12y ago

Latest changes: May 1st, 2014 according to the changelog: http://expect.cvs.sourceforge.net/viewvc/expect/expect/ChangeLog?revision=5.126 http://expect.cvs.sourceforge.net/viewvc/expect/expect/Chang... Also, you can apt-get install `expect`.

spektom 12y ago

Oh, this is great old tool! :) Here's PHP extension for Expect that I've created back in 2005: http://pecl.php.net/package/expect http://pecl.php.net/package/expect

npsimons 12y ago

Expect is one of the few reliable ways I've found to automate crufty, closed source things. Say, like a perforce client pull that uses a caching proxy so that the proxy is up to date for actual human pulls.

younata 12y ago

I once wrote a very minimal IRC client by essentially wrapping telnet around expect. All it did was automate sending out PINGs, but it worked.

Crito 12y ago

Expect is great, however if you ever determine that using it is the best way to solve your problem, then you are probably approaching the problem the wrong way in the first place (the best way to proceed from that point may very well be to continue using expect though). I used it a few years ago to create a test suite for some dedicated VoIP hardware (controlling the boxes over telnet). It worked great, but if I had had more time, that definitely would not have been my approach.

ZoFreX 12y ago

I first discovered this about two months ago. It may be old, but it solved a problem we had and nothing else quite fit the job. We're using it to run a flaky test suite and count how many times it succeeds, fails, or times out (yes, times out - yay Android!). Very important when you're trying to determine if potential stability improvements actually make a difference or not, and you need to run the 3 minute suite ten times at least to get something statistically significant.