5 ms·

Show HN: Automation the KISS way. No YAML involved

5 points by linkdd 5y ago

Hello HN, I like Ansible as a tool for automation to perform tasks on multiple remote hosts. But like many of you, I'm sick of the custom YAML DSL that many of this kind of tools provide. How many times have we said "just give me a true scripting language!" ? That's why I started this project `tricorder` (as in Star Trek's tricorder, a simple device to do pretty much anything required by the plot). For now, it's only a tool to execute a command on multiple hosts (like ansible, but without the YAML) and returning the outputs as JSON so you can query it with other tools like `jq` in your bash scripts. But with time, I intend to add other tools to provide the following features: - mimic the "gather facts" feature from ansible - upload/download files to/from remote hosts - Rust API to include in your projects - bindings to other languages like Python/TypeScript/Go I'd be happy to have some feedback on the source code (as Rust is not my main language) or on what features you'd like to see implemented. Thank you :) Link to the Github repository: https://github.com/linkdd/tricorder

linkdd 5y ago
Clickable link: https://github.com/linkdd/tricorder https://github.com/linkdd/tricorder
ciprian_craciun 5y ago
I couldn't agree more with you about Ansible... I've started using Ansible in two projects, and I think it was enough for me... Ansible is perhaps great when you have granular tasks, that maybe don't need to communicate between them, or perhaps that don't have too much logic (or worse loops). However, as soon as your Ansible "scripts" start becoming actual "scripts", everything gets out of hand... (Also the performance is terrible due to the fact that each task is "bundled" on the host, copied remotely, extracted, executed, deleted, repeat for each and every instance of the same task...) ---- Now about your `tricorder`, by looking at the readme, I gather it just allows one to run the same command over multiple hosts (filtered by name or tag), but nothing beyond that? How could one run for example a pipeline of tools on the remote hosts? (I assume `bash -c 'tool-1 | tool-2'`, but with careful escaping?) Perhaps a first step towards actually being an "Ansible replacement" would be to bring back in some "scripting" ability. For example instead of giving each command as arguments, there could be a "library of scripts", and the user can choose one of those to run on each remote host. Then those scripts could be written in whatever language the user chooses (`bash`, Python, Ruby, etc.) Also, with regard to the "library of scripts", one thing that Ansible got right is the possibility to put multiple different tasks in the same file, thus one doesn't need 20 files for 20 small tasks, but instead these can be put inside the same YAML file. ---- For example this is how I did it in my own "Ansible replacement", <https://github.com/volution/z-run https://github.com/volution/z-run>, like for example, one can write something like these snippets: <https://github.com/volution/z-run/blob/development/examples/ssh.z-run https://github.com/volution/z-run/blob/development/examples/...> (see `ssh / launch / simple` that runs locally which just calls `z-run ssh "${_target}" ':: ssh / remote / main' "${@}"`, which is in itself a non trivial `bash` script, one that will run remotely.) In the end, what I think we need, is a tool to easily run "scripts" on a remote system, but make it as easy to use as it would be on the local machine. (Sort of like the RPC but across SSH and for scripts.)
- gilcot 5y ago
  At first look, your z-run remind me a helper[1] I use sometimes. For massive remote executions, I often use dssh[2] program or similar. I think that tricorder belongs to that later category and isn't a true library of scripts dispatcher. [1] https://github.com/lucaswerkmeister/cheats https://github.com/lucaswerkmeister/cheats [2] http://sigpipe.io/dssh http://sigpipe.io/dssh
- linkdd 5y ago
  Thank you for your feedback. > I gather it just allows one to run the same command over multiple hosts (filtered by name or tag), but nothing beyond that? For now. I intend to carefully extend it to other features, like in said in the original post: - mimic the "gather facts" feature from ansible so you can query it with `jq` - provide an API in a few programming languages so that you can choose whatever fits your needs to "script" the logic (and with Rust/Go, distributing a single binary as a 'setup.exe' should be plenty enough) - probably a plugin system similar to kubectl (as in subcommands look in PATH for a binary named `tricorder-subcommand`) I'd like to see it becoming a real replacement for YAML based workflows, but I started small because I don't want to repeat the same mistakes. I'd like to see how people would use this before building on it.
- gilcot 5y ago
  > For example instead of giving each command as arguments, there could be a "library of scripts", and the user can choose one of those to run on each remote host. Then those scripts could be written in whatever language the user chooses (`bash`, Python, Ruby, etc.) For running scripts from a library, I've used ByJG's automate.sh[1] that call'em recipes. Unlike z-run, they're bash only and there's no select menu and looks simpler. [1] https://github.com/byjg/automate https://github.com/byjg/automate
stephenr 5y ago
This is interesting. Is the tag mode an or match or an and match? Ie does “server,backend” in the example match those with both tags or either tag?
- linkdd 5y ago
  Thank you for the feedback. This is an OR, if either tag is present, it will be selected. I'm not set on this, maybe I will add a simple parser for something like `(A|B)&(~C)`. What do you think ?
  - stephenr 5y ago
    I think the ability to target more restrictively would be a fantastic addition, largely because I could imagine the tags being used in a way the example doesn't cover: DC/AZ/Region tags. I don't know that a full parser for Boolean logic is necessary - even having a complimentary `-T` option that takes mandatory tag names might be enough. I'm curious about the ability to upload stuff too. Right now I mostly use a tool I wrote in shell several years ago (unimaginatively called `remote-install.sh`) that basically tars up a directory (minus exclude rules), sends it to the remote host, untars it to a tmp dir, and then runs the specified command (via sudo if specified). But it's very much a one-at-a-time process. Meanwhile a deployment helper script I wrote is very much a multiple-at-once process, and so it's multiple background SSH processes. I could see both of these likely being candidates for using tricorder. One interesting thing I don't see on your roadmap would be some way to have per-host data sent (i.e. a way to perhaps set some args to a command/script on a per-host basis, or perhaps a way to define environment variables to set per-host, and the script invocation could reference those)... Maybe this is part of the 'send data' stage, and it just has the ability to substitute a few vars when determining what so send. So e.g. `--send-file "%id%.env" --send-file "script.sh --send-file "data.foo"` would interpolate the `%id%` substring with the target's ID from the TOML file, allowing to send a customised file per-host.. Anyway. Just a thought!
    - linkdd 5y ago
      > largely because I could imagine the tags being used in a way the example doesn't cover: DC/AZ/Region tags. I'll add this feature to the roadmap then :) > I don't know that a full parser for Boolean logic is necessary - even having a complimentary `-T` option that takes mandatory tag names might be enough. I truly dislike command lines with hundreds of arguments. IMHO, A boolean logic would be clearer on the intent than a combination of -t -T -not-T etc... > so it's multiple background SSH processes. I'm not sure yet if parallelization should be the default or not. Also I've not yet learned Tokio and Async programming in Rust :P > some way to have per-host data sent Yes, just like Ansible evaluates very variable per host. At the moment you can do it manually: res=$(tricorder -i inventory -- echo "available") host_ids=$(echo $res | jq "some jq black magic") for host_id in $host_ids do tricorder -i inventory -H $host_id -- echo "specific data sent to $host_id" done Maybe i'll make use of the `tinytemplate` crate and add a -D option for data that would be similar to the inventory (TOML static file or program/script outputting JSON). I'm not sure yet on how it should look. I mean to keep tricorder KISS so it can be integrated easily in a bash script as a standalone command, and not a full-on folder structure like ansible playbooks.
cdaringe 5y ago
Keep working on this. It’s a splendid idea. Would be cool to eventually have a recipe where you import the crate, then program the deploy in rust, invoking your tool which builds my-build.rs, or a proper cargo build. The following not true apples to apples, but pretty close. Rather than another crummy stringy DSL, the ocaml community said “we want to program our CI pipelines in OCaml”. So, they created https://github.com/ocurrent/ocurrent https://github.com/ocurrent/ocurrent. You build your pipeline in an incredible language—then the CI server simply invokes your pipeline code, assuming you’ve implemented a basic interface.
- linkdd 5y ago
  Thank you for your feedback! :) > Would be cool to eventually have a recipe where you import the crate, then program the deploy in rust This is actually one of the goals and already somewhat possible since v0.5.0 (i'm at v0.8.0 now, working on the test suite to reach 1.0.0) if you don't mind writing shell commands as strings in Rust :P The currently available tasks are: - info: to get what's in the inventory (and later infos about the remote host) - exec: to run a (templated) shell command - upload: to upload files or render templates - download: to download files With this, you can already upload archives, extract them, create systemd files, start the service, etc... It's a bit more manual than with ansible/puppet/... but it works :) I think, at the moment, a more correct comparison would be with Fabric[0] in Python. I have some ideas for post-1.0.0 relying on TCP/IP tunneling through the SSH session (docker builds on remotes, querying remote services hiding behind an SSH bastion, ...), I'd love to hear more ideas though :) [0] - https://www.fabfile.org/

5 points by linkdd 5y ago

linkdd 5y ago

Clickable link: https://github.com/linkdd/tricorder https://github.com/linkdd/tricorder

ciprian_craciun 5y ago

I couldn't agree more with you about Ansible... I've started using Ansible in two projects, and I think it was enough for me... Ansible is perhaps great when you have granular tasks, that maybe don't need to communicate between them, or perhaps that don't have too much logic (or worse loops). However, as soon as your Ansible "scripts" start becoming actual "scripts", everything gets out of hand... (Also the performance is terrible due to the fact that each task is "bundled" on the host, copied remotely, extracted, executed, deleted, repeat for each and every instance of the same task...) ---- Now about your `tricorder`, by looking at the readme, I gather it just allows one to run the same command over multiple hosts (filtered by name or tag), but nothing beyond that? How could one run for example a pipeline of tools on the remote hosts? (I assume `bash -c 'tool-1 | tool-2'`, but with careful escaping?) Perhaps a first step towards actually being an "Ansible replacement" would be to bring back in some "scripting" ability. For example instead of giving each command as arguments, there could be a "library of scripts", and the user can choose one of those to run on each remote host. Then those scripts could be written in whatever language the user chooses (`bash`, Python, Ruby, etc.) Also, with regard to the "library of scripts", one thing that Ansible got right is the possibility to put multiple different tasks in the same file, thus one doesn't need 20 files for 20 small tasks, but instead these can be put inside the same YAML file. ---- For example this is how I did it in my own "Ansible replacement", <https://github.com/volution/z-run https://github.com/volution/z-run>, like for example, one can write something like these snippets: <https://github.com/volution/z-run/blob/development/examples/ssh.z-run https://github.com/volution/z-run/blob/development/examples/...> (see `ssh / launch / simple` that runs locally which just calls `z-run ssh "${_target}" ':: ssh / remote / main' "${@}"`, which is in itself a non trivial `bash` script, one that will run remotely.) In the end, what I think we need, is a tool to easily run "scripts" on a remote system, but make it as easy to use as it would be on the local machine. (Sort of like the RPC but across SSH and for scripts.)

gilcot 5y ago
At first look, your z-run remind me a helper[1] I use sometimes. For massive remote executions, I often use dssh[2] program or similar. I think that tricorder belongs to that later category and isn't a true library of scripts dispatcher. [1] https://github.com/lucaswerkmeister/cheats https://github.com/lucaswerkmeister/cheats [2] http://sigpipe.io/dssh http://sigpipe.io/dssh
linkdd 5y ago
Thank you for your feedback. > I gather it just allows one to run the same command over multiple hosts (filtered by name or tag), but nothing beyond that? For now. I intend to carefully extend it to other features, like in said in the original post: - mimic the "gather facts" feature from ansible so you can query it with `jq` - provide an API in a few programming languages so that you can choose whatever fits your needs to "script" the logic (and with Rust/Go, distributing a single binary as a 'setup.exe' should be plenty enough) - probably a plugin system similar to kubectl (as in subcommands look in PATH for a binary named `tricorder-subcommand`) I'd like to see it becoming a real replacement for YAML based workflows, but I started small because I don't want to repeat the same mistakes. I'd like to see how people would use this before building on it.
gilcot 5y ago
> For example instead of giving each command as arguments, there could be a "library of scripts", and the user can choose one of those to run on each remote host. Then those scripts could be written in whatever language the user chooses (`bash`, Python, Ruby, etc.) For running scripts from a library, I've used ByJG's automate.sh[1] that call'em recipes. Unlike z-run, they're bash only and there's no select menu and looks simpler. [1] https://github.com/byjg/automate https://github.com/byjg/automate

stephenr 5y ago

This is interesting. Is the tag mode an or match or an and match? Ie does “server,backend” in the example match those with both tags or either tag?

linkdd 5y ago
Thank you for the feedback. This is an OR, if either tag is present, it will be selected. I'm not set on this, maybe I will add a simple parser for something like `(A|B)&(~C)`. What do you think ?
- stephenr 5y ago
  I think the ability to target more restrictively would be a fantastic addition, largely because I could imagine the tags being used in a way the example doesn't cover: DC/AZ/Region tags. I don't know that a full parser for Boolean logic is necessary - even having a complimentary `-T` option that takes mandatory tag names might be enough. I'm curious about the ability to upload stuff too. Right now I mostly use a tool I wrote in shell several years ago (unimaginatively called `remote-install.sh`) that basically tars up a directory (minus exclude rules), sends it to the remote host, untars it to a tmp dir, and then runs the specified command (via sudo if specified). But it's very much a one-at-a-time process. Meanwhile a deployment helper script I wrote is very much a multiple-at-once process, and so it's multiple background SSH processes. I could see both of these likely being candidates for using tricorder. One interesting thing I don't see on your roadmap would be some way to have per-host data sent (i.e. a way to perhaps set some args to a command/script on a per-host basis, or perhaps a way to define environment variables to set per-host, and the script invocation could reference those)... Maybe this is part of the 'send data' stage, and it just has the ability to substitute a few vars when determining what so send. So e.g. `--send-file "%id%.env" --send-file "script.sh --send-file "data.foo"` would interpolate the `%id%` substring with the target's ID from the TOML file, allowing to send a customised file per-host.. Anyway. Just a thought!
  - linkdd 5y ago
    > largely because I could imagine the tags being used in a way the example doesn't cover: DC/AZ/Region tags. I'll add this feature to the roadmap then :) > I don't know that a full parser for Boolean logic is necessary - even having a complimentary `-T` option that takes mandatory tag names might be enough. I truly dislike command lines with hundreds of arguments. IMHO, A boolean logic would be clearer on the intent than a combination of -t -T -not-T etc... > so it's multiple background SSH processes. I'm not sure yet if parallelization should be the default or not. Also I've not yet learned Tokio and Async programming in Rust :P > some way to have per-host data sent Yes, just like Ansible evaluates very variable per host. At the moment you can do it manually: res=$(tricorder -i inventory -- echo "available") host_ids=$(echo $res | jq "some jq black magic") for host_id in $host_ids do tricorder -i inventory -H $host_id -- echo "specific data sent to $host_id" done Maybe i'll make use of the `tinytemplate` crate and add a -D option for data that would be similar to the inventory (TOML static file or program/script outputting JSON). I'm not sure yet on how it should look. I mean to keep tricorder KISS so it can be integrated easily in a bash script as a standalone command, and not a full-on folder structure like ansible playbooks.

cdaringe 5y ago

Keep working on this. It’s a splendid idea. Would be cool to eventually have a recipe where you import the crate, then program the deploy in rust, invoking your tool which builds my-build.rs, or a proper cargo build. The following not true apples to apples, but pretty close. Rather than another crummy stringy DSL, the ocaml community said “we want to program our CI pipelines in OCaml”. So, they created https://github.com/ocurrent/ocurrent https://github.com/ocurrent/ocurrent. You build your pipeline in an incredible language—then the CI server simply invokes your pipeline code, assuming you’ve implemented a basic interface.

linkdd 5y ago
Thank you for your feedback! :) > Would be cool to eventually have a recipe where you import the crate, then program the deploy in rust This is actually one of the goals and already somewhat possible since v0.5.0 (i'm at v0.8.0 now, working on the test suite to reach 1.0.0) if you don't mind writing shell commands as strings in Rust :P The currently available tasks are: - info: to get what's in the inventory (and later infos about the remote host) - exec: to run a (templated) shell command - upload: to upload files or render templates - download: to download files With this, you can already upload archives, extract them, create systemd files, start the service, etc... It's a bit more manual than with ansible/puppet/... but it works :) I think, at the moment, a more correct comparison would be with Fabric[0] in Python. I have some ideas for post-1.0.0 relying on TCP/IP tunneling through the SSH session (docker builds on remotes, querying remote services hiding behind an SSH bastion, ...), I'd love to hear more ideas though :) [0] - https://www.fabfile.org/