3 ms·

I ran about 30k machines with sshpass and it worked just fine. Easy to script. I don’t understand at all why we need something else here.

vbernat 3y ago
What about: this is now somewhat builtin into SSH, so you don't need an additional tool (or more accurately, this additional tool could be far simpler and more robust).
- latchkey 3y ago
  Ok. That is helpful, but ssh server or client? No way I would bother upgrading the server on all those boxes just to get to a point that is already working with sshpass.
  - vbernat 3y ago
    Client. OpenSSH 8.4 has SSH_ASKPASS_REQUIRE that allows one to always invoke SSH_ASKPASS, notably when run from a terminal (previously, it was only invoked when there was no terminal to prompt the password).
    - latchkey 3y ago
      Interesting then! Makes sense. Thanks for the explanation.
    - gnufx 3y ago
      That doesn't help with 2FA, though, or on, say, EL8.
MikePlacid 3y ago
> I ran about 30k machines with sshpass and it worked just fine. Easy to script. I don’t understand at all why we need something else here. Do you keep all your passwords in one directory or in something more complicated?
- latchkey 3y ago
  User: user Pass: 1 Lol
  - gossamer 3y ago
    I kinda like the way ansible does it. There is a concept of a vault. You put all the passwords in that file and they are all encrypted. You use one password when running the command or playbook and all of the keys are decrypted as needed. I don't know if that is efficient for 30K machines though.
    - latchkey 3y ago
      It isnt. I ended up building a small golang binary that ran as root and I could hit it with http calls to execute whatever I wanted. Built a message queue that would work through all the machines for eventual consistency. Worked great.
      - xorcist 3y ago
        I've used both puppet and ansible at that scale and it's not only workable, it is a good idea.
        latchkey 3y ago
        How did you deal with machines randomly crashing and ensuring everything was consistent at all times across multiple physical datacenters? I was also solving more problems than just ansible/puppet solve... like monitoring of specific systems on the machines. Regardless, more than one way to skin a cat.
        xorcist 3y ago
        That's the whole idea of using a system based on declarative state. As soon as the system is back up, the agent can resolve state again. You also keep a central copy of the state of every agent. You can absolutely do this by writing your own agent (or by writing a family of bash script, but they tend to grow pretty complex over time), ansible is just a framework to write that in a standardized way. It will also out of the box handle a number of common system state such as running services and sysctl triggers. There are a number of similar systems such as puppet or salt, which are all variations of the same basic idea. 30k hosts are a lot, and will need sizing the system appropriately, but it's not an unusual configuration by any means.

vbernat 3y ago

What about: this is now somewhat builtin into SSH, so you don't need an additional tool (or more accurately, this additional tool could be far simpler and more robust).

latchkey 3y ago
Ok. That is helpful, but ssh server or client? No way I would bother upgrading the server on all those boxes just to get to a point that is already working with sshpass.
- vbernat 3y ago
  Client. OpenSSH 8.4 has SSH_ASKPASS_REQUIRE that allows one to always invoke SSH_ASKPASS, notably when run from a terminal (previously, it was only invoked when there was no terminal to prompt the password).
  - latchkey 3y ago
    Interesting then! Makes sense. Thanks for the explanation.
  - gnufx 3y ago
    That doesn't help with 2FA, though, or on, say, EL8.

MikePlacid 3y ago

> I ran about 30k machines with sshpass and it worked just fine. Easy to script. I don’t understand at all why we need something else here. Do you keep all your passwords in one directory or in something more complicated?

latchkey 3y ago
User: user Pass: 1 Lol
- gossamer 3y ago
  I kinda like the way ansible does it. There is a concept of a vault. You put all the passwords in that file and they are all encrypted. You use one password when running the command or playbook and all of the keys are decrypted as needed. I don't know if that is efficient for 30K machines though.
  - latchkey 3y ago
    It isnt. I ended up building a small golang binary that ran as root and I could hit it with http calls to execute whatever I wanted. Built a message queue that would work through all the machines for eventual consistency. Worked great.
    - xorcist 3y ago
      I've used both puppet and ansible at that scale and it's not only workable, it is a good idea.
      - latchkey 3y ago
        How did you deal with machines randomly crashing and ensuring everything was consistent at all times across multiple physical datacenters? I was also solving more problems than just ansible/puppet solve... like monitoring of specific systems on the machines. Regardless, more than one way to skin a cat.
        xorcist 3y ago
        That's the whole idea of using a system based on declarative state. As soon as the system is back up, the agent can resolve state again. You also keep a central copy of the state of every agent. You can absolutely do this by writing your own agent (or by writing a family of bash script, but they tend to grow pretty complex over time), ansible is just a framework to write that in a standardized way. It will also out of the box handle a number of common system state such as running services and sysctl triggers. There are a number of similar systems such as puppet or salt, which are all variations of the same basic idea. 30k hosts are a lot, and will need sizing the system appropriately, but it's not an unusual configuration by any means.