7 ms·

Expect is cool, I have used it in the past as well as the expect perl module, but in my opinion is always a suboptimal way to automate things. It's always bette

Expect is cool, I have used it in the past as well as the expect perl module, but in my opinion is always a suboptimal way to automate things. It's always better and more robust to have a proper interface to script things instead of relying on matching brittle text/regex responses and timers and such. Of course sometimes it's not possible, so expect comes in handy.

wyldfire 10y ago
I 100% agree. If I can use paramiko/fabric or something similar, I prefer it way more than expect. If you have a slave device over RS232, expect is probably your only option. But if that remote item is running sshd, you are way better off without expect.
- crististm 10y ago
  what's wrong with expect spawning ssh??
  - jstimpfle 10y ago
    I think expect is often used to have it enter username and password "interactively". But it's easier and more robust to just use the public-key auth facility that is built-in to ssh.
    - Karunamon 10y ago
      Not always possible. The problem is that ssh lives in the mythical land of perfect security, where people never need to do unsafe things in order to get their jobs done. Hence keys silently not working without "correct" permissions, passwords not passable on the CLI, inconsistent type-yes-to-override behavior, and other user-hostile design choices. I love openssh, but it's a real pain in the ass sometimes.
      - jstimpfle 10y ago
        I don't have any of these problems. Just don't use passwords non-interactively, but use public key auth. Also, as any sane program, (open)SSH does not ask you to type anything when running non-interactively in a script (more precisely, when no terminal is present).
        Karunamon 10y ago
        If you don't have any of those problems, you're very lucky. Again, ssh is designed with the assumption that people never have a reason to do unsafe things.
      - seiferteric 10y ago
        there is also sshpass for supplying password, as well as using a languages library like paramiko for python which I believe allows you to supply a password. It would take more than password auth to force me to use expect these days :)
    - crististm 10y ago
      This isn't expects' fault. If you did not want to record the password in the script then ssh manually and then continue the script from there (e.g. with 'interact' command)
  - wyldfire 10y ago
    expect scripts in general are brittle IMO. The general design is: "look for this and then that and then this. If at some point you stall along this path for more than X seconds, something must have gone wrong, abort." Well, I maintain that the characters in the stream aren't as good success criterion as the stack of exit codes among the tasks. So if I "ssh remotefoo /tests/the_test.sh" it is the_test.sh's job to return 0 if it succeeded and nonzero otherwise. ssh will return 0 if the remote script returns zero, so I can count on that. But if the host is busy sometimes and the_test.sh takes 91 seconds on a heavily-loaded day but 37-45 seconds normally, I might encode the wrong timeout in my expect script. Things like /etc/issue or /etc/motd or PS1 should have no bearing on my test results. But if someone writes the text "Password:" or "Username:" or some other similar content in there, my expect script might choke on it.
    - zwischenzug 10y ago
      Indeed, which is why in ShutIt I used some tricks with the prompt to determine whether a command was finished, and then (optional, but defaulting) checks on exit codes to determine success/failure. It seems to work pretty well, and no need to specify what it expected. It results in code like this: https://github.com/ianmiell/shutit-chef-env/blob/master/shutit_chef_env.py#L83-L88 https://github.com/ianmiell/shutit-chef-env/blob/master/shut... which sets up a chef server.
      - wyldfire 10y ago
        Aside: the bare except clause used here [1] is a bad habit. You should try to avoid it. Python raises things like NameError and TypeError for simple design problems like mistyping variable names. It's extremely rare that you can handle errors like that. Your "except:" would attempt to handle and effectively mask bugs like this. [1] https://github.com/ianmiell/shutit-chef-env/blob/master/shutit_chef_env.py#L56 https://github.com/ianmiell/shutit-chef-env/blob/master/shut...
        zwischenzug 10y ago
        Thanks for the tip. Laziness is a sin and a virtue.
  - mypalmike 10y ago
    Why would you mix expect with ssh? Most of the time, if you can ssh, you can use an exec channel ("ssh somehost somecommand"), so you shouldn't need to pretend to be typing characters and waiting for prompts.
- seanp2k2 10y ago
  I've used paramiko + expect to automate distributed benchmarking + results gathering. I can say that while it ultimately worked, Curse That Thing In Particular. It was so so so fiddly to get everything working, and very fragile + output-dependent. https://github.com/ip2k/Mad-Science?files=1 https://github.com/ip2k/Mad-Science?files=1 I wrote this a long time ago and it's ugly. It totally worked and let our company find more optimal hosting configs.
  - lamontcg 10y ago
    > while it ultimately worked, Curse That Thing In Particular "Things Only 90s SysAdmins Will Remember" Expect is mostly a tool of last resort when a utility has silly prompts that you can't bypass by hitting it with command line options like `-y` and stuff.
  - knownothing 10y ago
    What does the `?files=1` bit do?
    - wyldfire 10y ago
      It appears to omit the repo's description and some of the buttons.
- pmoriarty 10y ago
  What's the advantage of using paramiko/fabric vs something like Salt, Ansible, Chef, or Puppet?
  - dozzie 10y ago
    Paramiko is a Python API for SSH client, so it's not comparable to either of the rest. Fabric itself is a poor man's badly implemented RPC for sysadmins: you send commands and receive output and result codes. Fabric wraps this in a Python API, so you can react to how your operations run. Salt and Ansible are similar to Fabric (and "poor man's badly implemented RPC" comment applies to them as well), but they also define some higher level tasks, like "have X package installed". All three have the operations defined at the call initiator and return synchronously, when all the work is done. Then there are CFEngine, Puppet, and Chef. Every managed server runs an agent that periodically executes some rules. CFEngine and Puppet have their own DSLs, and Chef uses general purpose language (Ruby), what I consider a bad fit. These work asynchronously, as you don't push the rules to the servers, but change them in their distribution point and wait for servers to pick up the update.
stcredzero 10y ago
Expect is cool, I have used it in the past as well as the expect perl module, but in my opinion is always a suboptimal way to automate things. When I was a graduate Comp Sci TA in the 1990's, it was the perfect way to automate grading the Unix shell that the Systems Software class was supposed to write. Effectively, I was also writing a regression/functional testing system for the class. I explained that the Expect/Tcl script was going to be used to grade their project, and I provided the code. One of my students said I was the best TA ever! There were a few students who complained that they shouldn't be marked off for a feature breaking, because it had worked a week or two ago. There were a few students who never figured out that the script was going to be used to grade their project, even though I provided weekly email updates explaining this.
- vram22 10y ago
  Nice use of expect. I expect (pun intended, though realized after the fact) that you used it to simulate sending user input to their shell? and checking that the shell's response was as expected? (not doing the second on purpose either, honest, it just seems the right word to use).
- schoen 10y ago
  > I explained that the Expect/Tcl script was going to be used to grade their project, and I provided the code. If it's deterministic, is there a risk of someone just writing a second Expect script (instead of a solution to the programming exercise) that interacts with yours and gives the replies that you want?
  - stcredzero 10y ago
    Sure, but I didn't necessarily say that I'd give them the final version of the script and all of the tests.
    - schoen 10y ago
      Oh, I see!
  - 10y ago
vram22 10y ago
Yes, expect is really cool. I haven't used it heavily, but had used it some earlier, to automate small command-line tasks to good effect. Also: 1. I had read much of Don Libes' Expect book. He had a unique style of writing in that book. Hard to describe, partly since it was some years ago, and partly because I can't find the words for it. But I think anyone reading the book would notice the style after reading a few pages. It was not a bad style by any means, just ... something different. 2. I've read that expect was (maybe still is) used heavily in the electronics/EDA industry, though I am either not sure or don't remember how exactly they use it, or rather, for what. And also maybe it was used heavily by GNU software for testing. Maybe someone here can confirm.
flukus 10y ago
Is it intended to automate things? It looks like it's meant to save typing with interactive things.

wyldfire 10y ago

I 100% agree. If I can use paramiko/fabric or something similar, I prefer it way more than expect. If you have a slave device over RS232, expect is probably your only option. But if that remote item is running sshd, you are way better off without expect.

crististm 10y ago
what's wrong with expect spawning ssh??
- jstimpfle 10y ago
  I think expect is often used to have it enter username and password "interactively". But it's easier and more robust to just use the public-key auth facility that is built-in to ssh.
  - Karunamon 10y ago
    Not always possible. The problem is that ssh lives in the mythical land of perfect security, where people never need to do unsafe things in order to get their jobs done. Hence keys silently not working without "correct" permissions, passwords not passable on the CLI, inconsistent type-yes-to-override behavior, and other user-hostile design choices. I love openssh, but it's a real pain in the ass sometimes.
    - jstimpfle 10y ago
      I don't have any of these problems. Just don't use passwords non-interactively, but use public key auth. Also, as any sane program, (open)SSH does not ask you to type anything when running non-interactively in a script (more precisely, when no terminal is present).
      - Karunamon 10y ago
        If you don't have any of those problems, you're very lucky. Again, ssh is designed with the assumption that people never have a reason to do unsafe things.
    - seiferteric 10y ago
      there is also sshpass for supplying password, as well as using a languages library like paramiko for python which I believe allows you to supply a password. It would take more than password auth to force me to use expect these days :)
  - crististm 10y ago
    This isn't expects' fault. If you did not want to record the password in the script then ssh manually and then continue the script from there (e.g. with 'interact' command)
- wyldfire 10y ago
  expect scripts in general are brittle IMO. The general design is: "look for this and then that and then this. If at some point you stall along this path for more than X seconds, something must have gone wrong, abort." Well, I maintain that the characters in the stream aren't as good success criterion as the stack of exit codes among the tasks. So if I "ssh remotefoo /tests/the_test.sh" it is the_test.sh's job to return 0 if it succeeded and nonzero otherwise. ssh will return 0 if the remote script returns zero, so I can count on that. But if the host is busy sometimes and the_test.sh takes 91 seconds on a heavily-loaded day but 37-45 seconds normally, I might encode the wrong timeout in my expect script. Things like /etc/issue or /etc/motd or PS1 should have no bearing on my test results. But if someone writes the text "Password:" or "Username:" or some other similar content in there, my expect script might choke on it.
  - zwischenzug 10y ago
    Indeed, which is why in ShutIt I used some tricks with the prompt to determine whether a command was finished, and then (optional, but defaulting) checks on exit codes to determine success/failure. It seems to work pretty well, and no need to specify what it expected. It results in code like this: https://github.com/ianmiell/shutit-chef-env/blob/master/shutit_chef_env.py#L83-L88 https://github.com/ianmiell/shutit-chef-env/blob/master/shut... which sets up a chef server.
    - wyldfire 10y ago
      Aside: the bare except clause used here [1] is a bad habit. You should try to avoid it. Python raises things like NameError and TypeError for simple design problems like mistyping variable names. It's extremely rare that you can handle errors like that. Your "except:" would attempt to handle and effectively mask bugs like this. [1] https://github.com/ianmiell/shutit-chef-env/blob/master/shutit_chef_env.py#L56 https://github.com/ianmiell/shutit-chef-env/blob/master/shut...
      - zwischenzug 10y ago
        Thanks for the tip. Laziness is a sin and a virtue.
- mypalmike 10y ago
  Why would you mix expect with ssh? Most of the time, if you can ssh, you can use an exec channel ("ssh somehost somecommand"), so you shouldn't need to pretend to be typing characters and waiting for prompts.
seanp2k2 10y ago
I've used paramiko + expect to automate distributed benchmarking + results gathering. I can say that while it ultimately worked, Curse That Thing In Particular. It was so so so fiddly to get everything working, and very fragile + output-dependent. https://github.com/ip2k/Mad-Science?files=1 https://github.com/ip2k/Mad-Science?files=1 I wrote this a long time ago and it's ugly. It totally worked and let our company find more optimal hosting configs.
- lamontcg 10y ago
  > while it ultimately worked, Curse That Thing In Particular "Things Only 90s SysAdmins Will Remember" Expect is mostly a tool of last resort when a utility has silly prompts that you can't bypass by hitting it with command line options like `-y` and stuff.
- knownothing 10y ago
  What does the `?files=1` bit do?
  - wyldfire 10y ago
    It appears to omit the repo's description and some of the buttons.
pmoriarty 10y ago
What's the advantage of using paramiko/fabric vs something like Salt, Ansible, Chef, or Puppet?
- dozzie 10y ago
  Paramiko is a Python API for SSH client, so it's not comparable to either of the rest. Fabric itself is a poor man's badly implemented RPC for sysadmins: you send commands and receive output and result codes. Fabric wraps this in a Python API, so you can react to how your operations run. Salt and Ansible are similar to Fabric (and "poor man's badly implemented RPC" comment applies to them as well), but they also define some higher level tasks, like "have X package installed". All three have the operations defined at the call initiator and return synchronously, when all the work is done. Then there are CFEngine, Puppet, and Chef. Every managed server runs an agent that periodically executes some rules. CFEngine and Puppet have their own DSLs, and Chef uses general purpose language (Ruby), what I consider a bad fit. These work asynchronously, as you don't push the rules to the servers, but change them in their distribution point and wait for servers to pick up the update.

stcredzero 10y ago

Expect is cool, I have used it in the past as well as the expect perl module, but in my opinion is always a suboptimal way to automate things. When I was a graduate Comp Sci TA in the 1990's, it was the perfect way to automate grading the Unix shell that the Systems Software class was supposed to write. Effectively, I was also writing a regression/functional testing system for the class. I explained that the Expect/Tcl script was going to be used to grade their project, and I provided the code. One of my students said I was the best TA ever! There were a few students who complained that they shouldn't be marked off for a feature breaking, because it had worked a week or two ago. There were a few students who never figured out that the script was going to be used to grade their project, even though I provided weekly email updates explaining this.

vram22 10y ago
Nice use of expect. I expect (pun intended, though realized after the fact) that you used it to simulate sending user input to their shell? and checking that the shell's response was as expected? (not doing the second on purpose either, honest, it just seems the right word to use).
schoen 10y ago
> I explained that the Expect/Tcl script was going to be used to grade their project, and I provided the code. If it's deterministic, is there a risk of someone just writing a second Expect script (instead of a solution to the programming exercise) that interacts with yours and gives the replies that you want?
- stcredzero 10y ago
  Sure, but I didn't necessarily say that I'd give them the final version of the script and all of the tests.
  - schoen 10y ago
    Oh, I see!
- 10y ago

vram22 10y ago

Yes, expect is really cool. I haven't used it heavily, but had used it some earlier, to automate small command-line tasks to good effect. Also: 1. I had read much of Don Libes' Expect book. He had a unique style of writing in that book. Hard to describe, partly since it was some years ago, and partly because I can't find the words for it. But I think anyone reading the book would notice the style after reading a few pages. It was not a bad style by any means, just ... something different. 2. I've read that expect was (maybe still is) used heavily in the electronics/EDA industry, though I am either not sure or don't remember how exactly they use it, or rather, for what. And also maybe it was used heavily by GNU software for testing. Maybe someone here can confirm.

flukus 10y ago

Is it intended to automate things? It looks like it's meant to save typing with interactive things.