Orchestration is a tough one because the second you bring it up, it devolves into a bunch of enterprise offerings and datastores and servers and holy wars and e

Orchestration is a tough one because the second you bring it up, it devolves into a bunch of enterprise offerings and datastores and servers and holy wars and evaluations and managed services and horrendously complicated setup. Workflow engines, schedulers, all supporting "standards" but none interop without a million lines of glue code. I've ended up writing basically this: - Groovy for a lot of syntactic power, decent performance, ability to make workflow-ish scripts - GPars for orchestration primitives to do stuff either serially or in parallel. - a configuration service to have configuration scoped at the global/environment/cluster/datacenter/rack/individual node levels - an access layer for executing remote commands and getting stdout/stderr back, and for sending, receiving files. It can be SSH, kubectl, AWS SSM, dockerrun, etc. I've had to rewrite the core framework four times, but it has enabled me to do complicated orchestrations such as spawn clustered databases, spawn fleets of load generators, data loaders, or api servers, tear them down, etc. I can code distributed backups and restores, and other cluster-wide "applications" with this as the basis. I can run things from the IDE. I can run locally or from remote servers. I can CRON them or whatever. I can debug them from IDE. I can do adhoc/arbitrary cluster-wide operations/commands using the Groovy Console. I will probably OSS it at some point, but it's not a finsihed product. But after three years I can't imagine doing what I do with it anywhere else. It could be called by workflow engines, it can call out to terraform and other tools. It can do multicloud (with enough code :-), it can do mixed access / deployments (with enough code :-) I guess the other limitation is the basic problem with a single orchestration node: probably won't easily scale to the 1000s of VM/nodes easily, or as efficiently as some other distributed coordinator could, but those things all come with the one-orchestration/integration/workflow-to-rule-them-all products. I don't want to be that.