5 ms·

This might be off-topic (or it might be a breath of fresh air if you're tired of configuration managers) -- I've been toying with the idea of making a trolling

This might be off-topic (or it might be a breath of fresh air if you're tired of configuration managers) -- I've been toying with the idea of making a trolling-but-no-really deployment framework called tarpipe, and all it does is take some files on your host, get em to a $place in one step, and run hook.sh. Oooooptionally, do some dir moves and symlnks to keep a prior state backed up, and service stop|start on either side of the mv/ln, to minimize downtime. Usage could be `tarpipe ssh user@host` or `tarpipe <(echo "cd keke && bash -c")` just as easily. It goes without saying that this simply wouldn't be comparable to Ansible, Chef, or other CM because it's too simple. It doesn't help manage state if it escapes $cwd. But if your application can curb its enthusiasm to a directory... boy is it simple if that's true. I already do this on the daily to crashland my bashrc and dotfiles on any new remote host. Maybe this kind of explicitly zero-dep deploy would be useful for more situations. Would anyone have a use for that? --- EDIT: What the heck: I prototyped it: https://gist.github.com/heavenlyhash/b575092aa84ce9f3e1d2 https://gist.github.com/heavenlyhash/b575092aa84ce9f3e1d2

mattzito 11y ago
I'm sure it'll work for some people, but configuration management is one of those things where the devil isn't IN the details, the devil IS the details. What happens if you have binary files to move around that are big? How do you insure atomicity? What happens when a run dies midway through? Can you make sure operations are idempotent? That's even before we get into variables, state, coordination/orchestration, and so on. I spent a lot of time (a troubling amount, really) in the automation space, and while it's true that simple problems are easy to solve, simple problems quickly become hard problems. Then you're building state machines, binary distribution systems, and now all of a sudden you have an enterprise workflow/configuration management system. I swear, half the products in this space started out exactly as yours, and the author just kept finding edge cases to fix until someone gave them $10m in series A funding and it was suddenly a business.
- heavenlyhash 11y ago
  Absolutely. But there's a really wildly underserved niche where I have less computers than netflix and I really just wanna push, not have a push framework engine manager tower of turtles. Sometimes I need kubernetes and orchestration and teleporting state snapshots and oh my. Sometimes I just need three files on the dang VPS box. (I see your point, of course. I wouldn't want to build orchestration in bash. Rather, I'd really like to see at least one tool that looks at all this, and goes "huh. We're gonna KISS. And no, it's never going to orchestrate 200 machines. That's okay.")
  - empthought 11y ago
    That's Ansible as far as I can tell. Roles, group_vars, dynamic inventory, etc. are all optional; pushing a few files to some VPSes is just a few lines in a YAML file.
    - heavenlyhash 11y ago
      And a sprawling python dependency on each side? You can play the "python's everywhere" card now if you like, but I'll look at you funny. How many megs, exactly, is the smallest thing you can have ansible push to an empty slave server with no prior contact?
      - empthought 11y ago
        I thought you were talking about simplicity/ease of use, not network bandwidth efficiency. If you're counting megabytes in a land where low-end mobile phones have 16 gigabytes of storage and 100Mbps wireless, then you probably have a boutique enough use case that you should roll your own.
  - mattzito 11y ago
    As the other respondent to this post said - ansible does this use case very well. Maybe the best if you don't already know puppet.
  - debaserab2 11y ago
krakensden 11y ago
I mean, this is basically how Ansible works under the hood, only with standalone scripts instead of arbitrary tarballs. You could probably implement it as an action plugin, and then keep using Ansible for all the hard annoying bits- the local execution framework, the ssh wrapper with all the edges sanded down, the 10k handled edge cases around running 'sudo' on arbitrary hosts...
kodablah 11y ago
I have been toying with something similar. Basically you write Go code and my prog is similar to "go run" except it cross-compiles it for the target server, ships itself over via ssh/sftp, runs itself, and deletes itself. I already use this like fabric in some places and have even built in some early conf mgmt with Go templates. Haven't really open sourced it yet except for the scaffolding at https://github.com/cretz/systrument https://github.com/cretz/systrument
- deleted 11y ago
  [deleted]
majewsky 11y ago
If you're looking for a simple/minimalistic configuration management, I'll shamelessly plug my own: http://holocm.org http://holocm.org
s_kilk 11y ago
> making a trolling-but-no-really deployment framework called tarpipe... I ended up writing something _very_ like this at a previous job, where we were building out a lot of new systems but the existing Ops department were still living in the Solaris days and wouldn't let anything like Chef/Ansible into their world. It actually worked really well, you got simple one-command deploys, with rollback and so on, which (seemed to) always work without a hitch. The whole thing was maybe a few hundred lines of bash. That was a few years ago, and apparently that hacky little tool is still in use for all deployments.
pas 11y ago
https://github.com/pinterest/teletraan https://github.com/pinterest/teletraan ? Uses simple scripts for the "logic", uses a tarball for "distribution".

mattzito 11y ago

I'm sure it'll work for some people, but configuration management is one of those things where the devil isn't IN the details, the devil IS the details. What happens if you have binary files to move around that are big? How do you insure atomicity? What happens when a run dies midway through? Can you make sure operations are idempotent? That's even before we get into variables, state, coordination/orchestration, and so on. I spent a lot of time (a troubling amount, really) in the automation space, and while it's true that simple problems are easy to solve, simple problems quickly become hard problems. Then you're building state machines, binary distribution systems, and now all of a sudden you have an enterprise workflow/configuration management system. I swear, half the products in this space started out exactly as yours, and the author just kept finding edge cases to fix until someone gave them $10m in series A funding and it was suddenly a business.

heavenlyhash 11y ago
Absolutely. But there's a really wildly underserved niche where I have less computers than netflix and I really just wanna push, not have a push framework engine manager tower of turtles. Sometimes I need kubernetes and orchestration and teleporting state snapshots and oh my. Sometimes I just need three files on the dang VPS box. (I see your point, of course. I wouldn't want to build orchestration in bash. Rather, I'd really like to see at least one tool that looks at all this, and goes "huh. We're gonna KISS. And no, it's never going to orchestrate 200 machines. That's okay.")
- empthought 11y ago
  That's Ansible as far as I can tell. Roles, group_vars, dynamic inventory, etc. are all optional; pushing a few files to some VPSes is just a few lines in a YAML file.
  - heavenlyhash 11y ago
    And a sprawling python dependency on each side? You can play the "python's everywhere" card now if you like, but I'll look at you funny. How many megs, exactly, is the smallest thing you can have ansible push to an empty slave server with no prior contact?
    - empthought 11y ago
      I thought you were talking about simplicity/ease of use, not network bandwidth efficiency. If you're counting megabytes in a land where low-end mobile phones have 16 gigabytes of storage and 100Mbps wireless, then you probably have a boutique enough use case that you should roll your own.
- mattzito 11y ago
  As the other respondent to this post said - ansible does this use case very well. Maybe the best if you don't already know puppet.
- debaserab2 11y ago

krakensden 11y ago

I mean, this is basically how Ansible works under the hood, only with standalone scripts instead of arbitrary tarballs. You could probably implement it as an action plugin, and then keep using Ansible for all the hard annoying bits- the local execution framework, the ssh wrapper with all the edges sanded down, the 10k handled edge cases around running 'sudo' on arbitrary hosts...

kodablah 11y ago

I have been toying with something similar. Basically you write Go code and my prog is similar to "go run" except it cross-compiles it for the target server, ships itself over via ssh/sftp, runs itself, and deletes itself. I already use this like fabric in some places and have even built in some early conf mgmt with Go templates. Haven't really open sourced it yet except for the scaffolding at https://github.com/cretz/systrument https://github.com/cretz/systrument

deleted 11y ago
[deleted]

majewsky 11y ago

If you're looking for a simple/minimalistic configuration management, I'll shamelessly plug my own: http://holocm.org http://holocm.org

s_kilk 11y ago

> making a trolling-but-no-really deployment framework called tarpipe... I ended up writing something _very_ like this at a previous job, where we were building out a lot of new systems but the existing Ops department were still living in the Solaris days and wouldn't let anything like Chef/Ansible into their world. It actually worked really well, you got simple one-command deploys, with rollback and so on, which (seemed to) always work without a hitch. The whole thing was maybe a few hundred lines of bash. That was a few years ago, and apparently that hacky little tool is still in use for all deployments.

pas 11y ago

https://github.com/pinterest/teletraan https://github.com/pinterest/teletraan ? Uses simple scripts for the "logic", uses a tarball for "distribution".