4 ms·

I really wish there was a configuration management tool out there with a statically typed API. I've spent countless hours debugging Ansible's YAML which gets re

I really wish there was a configuration management tool out there with a statically typed API. I've spent countless hours debugging Ansible's YAML which gets really unwieldy with hundreds or thousands of tasks.

breatheoften 7y ago
Pulumi comes the closest that I’ve seen — https://www.pulumi.com/ https://www.pulumi.com/
- cheshire_cat 7y ago
  This seems to work only with cloud providers, not with dedi boxes / VPSs?
  - eropple 7y ago
    "Cloud?" It can work with anything with an API. If there's a Terraform provider that they or the community hasn't projected into TypeScript (or whatever) you can do that yourself with minimal effort. Not sure what else you'd want.
    - fock 7y ago
      How about plain old SSH? like all those low-end-boxes, you know ;).
      - eropple 7y ago
        I could write a provider to do so in a couple hours if one doesn't already exist. But frankly, if you're using those boxes you've already decided that your time is worth less than your money and spending time on automation is probably lower ROI than moving to something, anything, with an API. Anything that isn't IBM, anyway.
        fock 7y ago
        Cool, if your provider will have a tested way to install packages, copy/template files and setup services beyond docker. Basically Ansible/Saltstack/Puppet/Chef _are_ the API for plain UNIX. As for the great "time worth less than money"-argument - I host my own nextcloud (for me and my family) on such a VPS for ~USD9. I get some 100GB of storage, ample RAM and CPU; sure that's probably oversubscribed, but after setup I can leave it running for years without worrying to check my bandwidth/storage quota in addition to it being cheaper than any big cloud-provider. Initial setup works with cloudinit and managing the VMs itself has an API. And I suppose for everyone not running a customer-facing business, the ROI of such solutions is basically unparalleled ;).
        eropple 7y ago
        Thanks for the lecture; I didn't spend three years teaching Chef at a Fortune 100, I promise. You get that one can just write a thing to kick off instance-level CM if it doesn't exist, yeah? Like I don't know if it does for Pulumi because Pulumi doesn't seem to care much about trailing-edge systems--for now at least, it seems, they have a big enough market without having to stretch backwards--but it's not complicated; Terraform already has provisioners to do exactly that, so the pattern exists.
        _frkl 7y ago
        > Basically Ansible/Saltstack/Puppet/Chef _are_ the API for plain UNIX. I've never thought about it like that, but I think you're exactly right. That's a really useful way to think about it. To me, anyway.
        _frkl 7y ago
        Dunno, in my experience there is a long tail of use-cases where it is a more practical to work closer to system level (for different reasons: less complexity, political issues, legacy systems, etc.), so I don't think it's as black and white as you make it out to be. But I agree that there is a threshold after which 'everything API' is the way to go. Where that threshold is if often guesswork. Overall, I don't think our tooling is quite there, and we haven't found the right abstractions yet. Terraform and Pulumi are nice enough, but to me it looks like they are only treating the symptoms, instead of looking at the issues on a more holistic level. I don't have any good solution for that, but I think we'll need to try to look at everything we're doing from different perspectives than the ones we are used to. That's hard (at least for me), and it's getting harder the deeper we dive into the current layers of complexity wrapped into more complexity.
snagglegaggle 7y ago
Also, being able to weaken the distinction between configuration and code would be good. Often I need to make small change to code. But is a lot of work.
_frkl 7y ago
It's not yet 100% ready for prime-time, and certainly not for hundreds of thousands of tasks, but I wrote a ...hm... generic framework-wrapping framework (using Ansible as the first and only currently supported 'framework' -- but I'm working on a Terraform adapter, and plain shell-scripts. K8s is also planned), that supports optional typing of arguments. It also allows you to add other metadata to arguments (like documentation, references): https://freckles.io https://freckles.io It can be used directly, using yaml. Or via Python (a bit like Pulumi, just more generic, and also a bit more flexible in other ways). Should be easy enough to also support other languages like JavaScript, although I think I'll write a client-library for a proper functional language first, because that'd be a really good fit, conceptually. Still needs better (and up-to-date) documentation, and I realize I really have to work on my elevator pitch (as is probably quite obvious). But I'm already using it for most of my sysadmin/devops related tasks, and I'm growing more and more confident that the concept behind it is sound. The ultimate goal is to allow users to build 100% composable devops pipelines, so that a complete app- (or whatever else-) infrastructure (including potential bootstrapping of physical machines as well as building container images, etc.) can be described with a single dict-like structure. That's probably a bigger piece of cake than I'll be able to bite off though :-)
TheThickOfIt 7y ago
IMO Ansible is not a good tool for configuration management compared to tools like Puppet and Chef. There's no node graph which mininises how much you can do in parallel and proper chaining of dependencies, and using YAML to describe largely imperative tasks is downright horrible (looking at you with_items). I think it works fine for orchestration, but anything more and it gets uncomfortable.
networked 7y ago
A sibling comment already mentions Pulumi. There is also mgmt, which is a statically-typed, functional, and reactive configuration language. In mgmt the configuration is a function of the system's state. Configuration is automatically recomputed and reapplied when the relevant state changes. This is something I have on many occasions wanted in a configuration management tool. For an introduction to how it works you can read https://purpleidea.com/blog/2018/02/05/mgmt-configuration-language/ https://purpleidea.com/blog/2018/02/05/mgmt-configuration-la.... Mgmt is not yet production-ready. I expect the trend of YAML configuration with templating hacks on top (which comes in at least two varieties: templates producing YAML as text like in Salt and templates inside YAML leaf nodes like in Ansible) to be seen as the early 2010s equivalent of growing an ad hoc programming language on top of XML. It tends to happen, since complex configuration needs to reuse values and contain expressions and it is easy to implement in a backwards-compatible way. The users will appreciate it if they had nothing before, but it locks them into a path of writing code (because it is code) with an ugly error-prone syntax, complex two-layered semantics, and little abstraction. I have got good mileage out of Ansible, but I would enjoy using it a lot more if the playbooks were written in a restricted subset of Python, for example. You can do better without types, too. Writing Scheme, Lua, Tcl, etc. code as configuration even predates the XML craze. (Tcl and Lua code can be sandboxed and limited in the number interpreter instructions and wall-clock run time. Tcl in particular has extensive sandboxing capabilities built in with "safe interpreters".). The appearance of restricted configuration languages like Starlark (https://github.com/google/starlark-go https://github.com/google/starlark-go), Jsonnet (https://jsonnet.org/ https://jsonnet.org/), and Dhall (https://dhall-lang.org/ https://dhall-lang.org/) looks like a reaction to the trend of JSON/YAML templating, and it is a welcome reaction.

breatheoften 7y ago

Pulumi comes the closest that I’ve seen — https://www.pulumi.com/ https://www.pulumi.com/

cheshire_cat 7y ago
This seems to work only with cloud providers, not with dedi boxes / VPSs?
- eropple 7y ago
  "Cloud?" It can work with anything with an API. If there's a Terraform provider that they or the community hasn't projected into TypeScript (or whatever) you can do that yourself with minimal effort. Not sure what else you'd want.
  - fock 7y ago
    How about plain old SSH? like all those low-end-boxes, you know ;).
    - eropple 7y ago
      I could write a provider to do so in a couple hours if one doesn't already exist. But frankly, if you're using those boxes you've already decided that your time is worth less than your money and spending time on automation is probably lower ROI than moving to something, anything, with an API. Anything that isn't IBM, anyway.
      - fock 7y ago
        Cool, if your provider will have a tested way to install packages, copy/template files and setup services beyond docker. Basically Ansible/Saltstack/Puppet/Chef _are_ the API for plain UNIX. As for the great "time worth less than money"-argument - I host my own nextcloud (for me and my family) on such a VPS for ~USD9. I get some 100GB of storage, ample RAM and CPU; sure that's probably oversubscribed, but after setup I can leave it running for years without worrying to check my bandwidth/storage quota in addition to it being cheaper than any big cloud-provider. Initial setup works with cloudinit and managing the VMs itself has an API. And I suppose for everyone not running a customer-facing business, the ROI of such solutions is basically unparalleled ;).
        eropple 7y ago
        Thanks for the lecture; I didn't spend three years teaching Chef at a Fortune 100, I promise. You get that one can just write a thing to kick off instance-level CM if it doesn't exist, yeah? Like I don't know if it does for Pulumi because Pulumi doesn't seem to care much about trailing-edge systems--for now at least, it seems, they have a big enough market without having to stretch backwards--but it's not complicated; Terraform already has provisioners to do exactly that, so the pattern exists.
        _frkl 7y ago
        > Basically Ansible/Saltstack/Puppet/Chef _are_ the API for plain UNIX. I've never thought about it like that, but I think you're exactly right. That's a really useful way to think about it. To me, anyway.
      - _frkl 7y ago
        Dunno, in my experience there is a long tail of use-cases where it is a more practical to work closer to system level (for different reasons: less complexity, political issues, legacy systems, etc.), so I don't think it's as black and white as you make it out to be. But I agree that there is a threshold after which 'everything API' is the way to go. Where that threshold is if often guesswork. Overall, I don't think our tooling is quite there, and we haven't found the right abstractions yet. Terraform and Pulumi are nice enough, but to me it looks like they are only treating the symptoms, instead of looking at the issues on a more holistic level. I don't have any good solution for that, but I think we'll need to try to look at everything we're doing from different perspectives than the ones we are used to. That's hard (at least for me), and it's getting harder the deeper we dive into the current layers of complexity wrapped into more complexity.

snagglegaggle 7y ago

Also, being able to weaken the distinction between configuration and code would be good. Often I need to make small change to code. But is a lot of work.

_frkl 7y ago

It's not yet 100% ready for prime-time, and certainly not for hundreds of thousands of tasks, but I wrote a ...hm... generic framework-wrapping framework (using Ansible as the first and only currently supported 'framework' -- but I'm working on a Terraform adapter, and plain shell-scripts. K8s is also planned), that supports optional typing of arguments. It also allows you to add other metadata to arguments (like documentation, references): https://freckles.io https://freckles.io It can be used directly, using yaml. Or via Python (a bit like Pulumi, just more generic, and also a bit more flexible in other ways). Should be easy enough to also support other languages like JavaScript, although I think I'll write a client-library for a proper functional language first, because that'd be a really good fit, conceptually. Still needs better (and up-to-date) documentation, and I realize I really have to work on my elevator pitch (as is probably quite obvious). But I'm already using it for most of my sysadmin/devops related tasks, and I'm growing more and more confident that the concept behind it is sound. The ultimate goal is to allow users to build 100% composable devops pipelines, so that a complete app- (or whatever else-) infrastructure (including potential bootstrapping of physical machines as well as building container images, etc.) can be described with a single dict-like structure. That's probably a bigger piece of cake than I'll be able to bite off though :-)

TheThickOfIt 7y ago

IMO Ansible is not a good tool for configuration management compared to tools like Puppet and Chef. There's no node graph which mininises how much you can do in parallel and proper chaining of dependencies, and using YAML to describe largely imperative tasks is downright horrible (looking at you with_items). I think it works fine for orchestration, but anything more and it gets uncomfortable.

networked 7y ago

A sibling comment already mentions Pulumi. There is also mgmt, which is a statically-typed, functional, and reactive configuration language. In mgmt the configuration is a function of the system's state. Configuration is automatically recomputed and reapplied when the relevant state changes. This is something I have on many occasions wanted in a configuration management tool. For an introduction to how it works you can read https://purpleidea.com/blog/2018/02/05/mgmt-configuration-language/ https://purpleidea.com/blog/2018/02/05/mgmt-configuration-la.... Mgmt is not yet production-ready. I expect the trend of YAML configuration with templating hacks on top (which comes in at least two varieties: templates producing YAML as text like in Salt and templates inside YAML leaf nodes like in Ansible) to be seen as the early 2010s equivalent of growing an ad hoc programming language on top of XML. It tends to happen, since complex configuration needs to reuse values and contain expressions and it is easy to implement in a backwards-compatible way. The users will appreciate it if they had nothing before, but it locks them into a path of writing code (because it is code) with an ugly error-prone syntax, complex two-layered semantics, and little abstraction. I have got good mileage out of Ansible, but I would enjoy using it a lot more if the playbooks were written in a restricted subset of Python, for example. You can do better without types, too. Writing Scheme, Lua, Tcl, etc. code as configuration even predates the XML craze. (Tcl and Lua code can be sandboxed and limited in the number interpreter instructions and wall-clock run time. Tcl in particular has extensive sandboxing capabilities built in with "safe interpreters".). The appearance of restricted configuration languages like Starlark (https://github.com/google/starlark-go https://github.com/google/starlark-go), Jsonnet (https://jsonnet.org/ https://jsonnet.org/), and Dhall (https://dhall-lang.org/ https://dhall-lang.org/) looks like a reaction to the trend of JSON/YAML templating, and it is a welcome reaction.