9 ms·

Marelle: logic programming for devops

148 points by lars512 13y ago

CraigJPerry 13y ago
Prolog does seem a reasonable fit for the problem space. That said, even CFEngine's long winded syntax (born of its commitment to "promise theory") isn't so bad. In my experience I don't feel expressiveness of the available languages is the problem most I need of solving. The real problems not being tackled by the existing configuration management tools are testing bundles/(play|cook)books/modules, cleanly injecting metadata to facilitate reuse and the bundle debugging workflow.
- atsaloli 13y ago
  What kind of metadata would facilitate reuse? Have you seen the latest verbose mode in CFEngine (as of 3.5, June 2013)? It's more compact to facilitate debugging. How else could debugging workflow be improved?
  - CraigJPerry 13y ago
    Say I have a simple bundle for inserting lines in /etc/security/limits.conf, it'd be good to have the bundle agent / edit_line which describes how to make the change, completely separate from the data that describes user1 gets a hard nofile of 64k. There's lots of ways to achieve that from just doing selective usebundles from various bundle files containing only vars to more flexible approaches; In our deployment we've provided this as part of our bundle framework (we inject the parameter data via modules protocol - so a box in location A applies different data than a box under line of business B), they both reuse the exact same (comprehensivly tested) code. This means no code release for a data change. No need to train everyone in CFE syntax just to allow an app developer to express that user c on box d is allowed more open file handles. The point though is that It'd be good to have as a first class language feature instead of each cfe site reinventing a separate framework for the same functionality. Haven't had any hands on with 3.5 so not familiar with that, but better print debugging isnt where I was going with the idea. I want to be able to step through a bundle, I want to be able to break when global class X is set. I want to then be able to set class X while stepping through. Kinda like making debug logging level interative and with higher signal to noise ratio on screen. If there was language support for unit testing my bundles I could use that to reason about what I expect to happen. Diego Zambonis book proposed one way to do that with the existing facilities but its just a proof of concept, e.g. it cant guarantee environment cleanup after a show stopping error in a test.
zimbatm 13y ago
The meet/met notation instantly made me think of babushka: http://babushka.me/ http://babushka.me/ The base principle seems to be the same except that babushka is a ruby DSL and that the dependency solving is part of the library instead of the language.
- qznc 13y ago
  Babushka seems to be one of the inspirations. The article says "Still, Babushka made me uneasy: all this ceremony and complex templating, just to describe a few facts and simple rules?"
e12e 13y ago
Very interesting. I wonder if it would be viable to build an entire package management system (like apt) in a similar manner -- and if there are any (semi-recent) improvements to prolog-like systems that might improve dependency resolution "for free"?
- X4 13y ago
  These are the package management systems that I use. A new package management systems just does this http://xkcd.com/927/ http://xkcd.com/927/ What we need is a cross-platform cross-applicaton meta package management system which comes as configuration management that evolves to a general purpose package manager without relying on any external package manager. I'm not fanboy enough and would happily replace: emerge, equo, apt-get, aptitude, ipkg, dpkg, zypp rpm, yum, tazpkg, pacman, homebrew tar/gzip, slackpkg (Yeah, you, Slackware, we were friends and you helped me for long enough. You shouldn't do all my work anymore, you're old. Take a rest) pip, cabal, bundler, gem, composer, pear, cpan, ctan, cran, maven, vundle – haven't you also a love hate relationship with these?
e12e 13y ago
Also like the bootstrap.sh - very nicely structured and easier to verify than many others I've seen. I'd still prefer these scripts to output missing dependencies that calling sudo -- but otherwise very nice. That is, rather than: function install_git() { echo 'Trying to install git' case $(uname -s) in #... Linux) if has_exec apt-get; then apt_update sudo apt-get install -y git #... else bail "Unknown linux variant" fi ;; *) bail "Unknown operating system $(uname -s)" ;; esac } Simply aggreagate a list of missing packages, and ouput (based on a similar case-statement for detetining host): echo Please install missing packages: echo sudo apt-get install ${missing-deps} https://github.com/larsyencken/marelle/blob/master/bootstrap.sh https://github.com/larsyencken/marelle/blob/master/bootstrap...
- lars512 13y ago
  Thanks! :) It's a little aggressive in calling sudo mainly because I'm using it to bootstrap new systems non-interactively. I don't want it to ask any questions at all, just to go.
acjohnson55 13y ago
It's been a while since I've studied Prolog, but it's good to see it getting some press. I learned Prolog before ever dabbling in Scheme or ML, and it was just as much of a paradigm shift from imperative style. It's kind of like FP-style pattern matching, but taken to a whole new level, such that the pattern matching itself is pretty much the entire computational engine. Rather than using functions, which take a number of arguments and return a result, you use predicates on a number arguments. You build your program out of clauses, which are conjunctions of predicates. A process called unification [1] finds the argument values (if any) that satisfy an entire clause. By sharing constants and variables between predicates, the valid results that unification can produce are restricted to what the logic of your program allows. So your algorithms end up being, in many ways, description of what a valid results look like, and the Prolog engine goes out and finds them. This paradigm clearly maps nicely to domains with lots of rules, like decision engines, parsers, and such. Sounds awesome, right? One caveat is that, in practice, you as a programmer have to have a pretty good understanding of how unification works in order to write performant algorithms. This is all from several years-old recollections and a little bit of Wikipedia to refresh my memory, so someone please chime in if I'm incorrect someplace or leaving out key concepts. [1] http://en.wikipedia.org/wiki/Unification_%28computer_science%29#Application:_Unification_in_logic_programming http://en.wikipedia.org/wiki/Unification_%28computer_science...
mickeyp 13y ago
Prolog's a surprisingly good fit for many every-day tasks where people otherwise invent DSLs or poorly-defined state machines and pattern matching algorithms. So how come we don't use it more often? Well, the reason is actually quite simple: Prolog's incredibly difficult to reason about once you grow past trivial facts and clauses like in the OP's article -- the examples given by him would've been introduced to you in the first lesson on Prolog in, say, University. The complexities of green/red cuts, backtracking and the procedural nature of Prolog makes it a really difficult language to truly learn -- much less one to specify complex programs in. I've never programmed in a language where 5-6 lines of errant Prolog code could stump myself, a post grad and a professor before until I did Prolog.
- keenerd 13y ago
  It is a shame really. Declarative programming is so much simpler to think about and extremely powerful. I've been working on trying to make declarative programming more accessible and while I personally have seen 60x improvements in ability, I have been unable to convince any of my teammates to give it a whirl.
  - X4 13y ago
    @keenerd try to build a DSL ontop of prolog and they'll come & play.
    - keenerd 13y ago
      It is a library that works with any language, no need to learn a new one. Two people can even work on the same problem in different languages and seamlessly combine their work into one result.
      - X4 13y ago
        interesting! what is it for?
- chubot 13y ago
  So I'm not experienced with Prolog, but the thing that scared me off is that it doesn't appear to have any computational complexity guarantees. It seems like will scale very poorly as your data gets bigger. It's just a notation for exhaustive search as far as I know, and you have to add ad hoc cuts outside the declarative model to get reasonable performance. I have experience with other abstractions which are supposed to be "declarative", but you really have to peek under the hood at the implementation to get them to work in production settings, and I don't like that.
calpaterson 13y ago
I'm skeptical, because there is already a configuration management language that is strongly influenced by Prolog and logic programming - Puppet - and the results are not that great. The secret of configuration management is that ordering NEEDS to be a first class feature and ordered manifests should be the default, not the option. Puppet suffers hugely from the influence from the fact that ordering is unpredictable and that you have to use arrows or requires to add order. People accidentally forget to add ordering, the resulting code works most of the time, and then one day you start running into an issue because some implicit dependency either didn't happen, or happened in the wrong order. In as much as all logic programming involves not explicitly controlling flow, I think they're inappropriate for configuration management. Sadly, I'm still stuck with using Puppet because of the organisation I work in. Having worked with Prolog in the past, I would also suggest that an important feature of a configuration management language is that the syntax make it hard to make typos and other minor errors because the feedback loop of deploying your code against a local vagrant machine is already long enough. I vividly remember the annoyance of having missed the full stop to end a block. I'm not desperate to repeat that experience.
- gwillem 13y ago
  That's the main reason we've switched over from Puppet to Ansible. It has less abstraction layers, but at least its execution order is predictable. The easy learning curve for new colleagues and the compact syntax are also nice benefits.
  - lobster_johnson 13y ago
    That's what I don't understand about Ansible. It's basically -- literally, actually -- Puppet reinvented with a new syntax based on YAML. Puppet works so well, Ansible seems like a typical "not invented here" tragedy that frequently occurs in open source. Rather than coalesce around something that works and has years of experience and knowledge and bug fixing sunk into it, people create something from scratch. To be specific, this is Ansible: - name: Memcached template: src=$repository_basedir/memcached/templates/memcached.conf.j2 dest=/etc/memcached.conf owner=root group=root mode=0644 notify: - memcached-restart The same in Puppet: file { '/etc/memcached.conf': content => template('memcached/memcached.conf'), owner => root, group => root, mode => '0644', notify => Service['memcached']; } It's literally the same data expressed here. Puppet has very little abstraction. But what you do get, which Ansible has not really understood, I think, is a data model. The above results in the graph getting an object File['/etc/memcached.conf'] which can be referred to. I can do this: service { 'memcached': require => File['/etc/memcached.conf']; } This in turn creates the graph node Service['memcached']. It takes part in a larger graph that looks like this: Stage[main]/Node[myserver]/Service['memcached'] In fact, manifests are all compiled into a large graph like this, with interdependencies between the nodes. The logic-based, Prolog-type language in the OP's link is precisely what Puppet builds underneath. Likewise, the graph model is extensible. I can invent a new object like so: define public_key() { file { "/home/$name/.ssh/authorized_keys": content => "$key"; } } public_key { 'someuser': key => 'some public key'; } account { 'someuser': require => PublicKey['someuser']; } Here, the define public_key() actually defines a new type of object that can be instantiated in scripts. When one is instantiated, other objects can be depend on it. This allows me to encapsulate stuff in my scripts; rather than having other declarations depend on the internal details of how a public key is stored (ie., in a home directory), they depend on my definition. Ansible may look simpler on the surface, but I think it's all surface. When you have learned to use Puppet, it becomes simple. Ansible looks to me like the product of a person who looked at Puppet, said something like "ugh Ruby, ew syntax" and made his own thing, without looking deeper. Simialrly, Ansible appeals to people who look at Puppet with the same reaction, again without looking deeper. The irony is that they are buying into something that is arguably poorer. That said, I would argue that part of the problem is Puppet's documentation, which surely must drive people to "poor man's Puppet" solutions. (The cynic in me wonders if Puppet's poor documentation is a deliberate ploy on part of Puppet Labs to boost their consulting business.)
deleted 13y ago
[deleted]

148 points by lars512 13y ago

Prolog does seem a reasonable fit for the problem space. That said, even CFEngine's long winded syntax (born of its commitment to "promise theory") isn't so bad. In my experience I don't feel expressiveness of the available languages is the problem most I need of solving. The real problems not being tackled by the existing configuration management tools are testing bundles/(play|cook)books/modules, cleanly injecting metadata to facilitate reuse and the bundle debugging workflow.

atsaloli 13y ago
What kind of metadata would facilitate reuse? Have you seen the latest verbose mode in CFEngine (as of 3.5, June 2013)? It's more compact to facilitate debugging. How else could debugging workflow be improved?
- CraigJPerry 13y ago
  Say I have a simple bundle for inserting lines in /etc/security/limits.conf, it'd be good to have the bundle agent / edit_line which describes how to make the change, completely separate from the data that describes user1 gets a hard nofile of 64k. There's lots of ways to achieve that from just doing selective usebundles from various bundle files containing only vars to more flexible approaches; In our deployment we've provided this as part of our bundle framework (we inject the parameter data via modules protocol - so a box in location A applies different data than a box under line of business B), they both reuse the exact same (comprehensivly tested) code. This means no code release for a data change. No need to train everyone in CFE syntax just to allow an app developer to express that user c on box d is allowed more open file handles. The point though is that It'd be good to have as a first class language feature instead of each cfe site reinventing a separate framework for the same functionality. Haven't had any hands on with 3.5 so not familiar with that, but better print debugging isnt where I was going with the idea. I want to be able to step through a bundle, I want to be able to break when global class X is set. I want to then be able to set class X while stepping through. Kinda like making debug logging level interative and with higher signal to noise ratio on screen. If there was language support for unit testing my bundles I could use that to reason about what I expect to happen. Diego Zambonis book proposed one way to do that with the existing facilities but its just a proof of concept, e.g. it cant guarantee environment cleanup after a show stopping error in a test.

zimbatm 13y ago

The meet/met notation instantly made me think of babushka: http://babushka.me/ http://babushka.me/ The base principle seems to be the same except that babushka is a ruby DSL and that the dependency solving is part of the library instead of the language.

qznc 13y ago
Babushka seems to be one of the inspirations. The article says "Still, Babushka made me uneasy: all this ceremony and complex templating, just to describe a few facts and simple rules?"

e12e 13y ago

Very interesting. I wonder if it would be viable to build an entire package management system (like apt) in a similar manner -- and if there are any (semi-recent) improvements to prolog-like systems that might improve dependency resolution "for free"?

X4 13y ago
These are the package management systems that I use. A new package management systems just does this http://xkcd.com/927/ http://xkcd.com/927/ What we need is a cross-platform cross-applicaton meta package management system which comes as configuration management that evolves to a general purpose package manager without relying on any external package manager. I'm not fanboy enough and would happily replace: emerge, equo, apt-get, aptitude, ipkg, dpkg, zypp rpm, yum, tazpkg, pacman, homebrew tar/gzip, slackpkg (Yeah, you, Slackware, we were friends and you helped me for long enough. You shouldn't do all my work anymore, you're old. Take a rest) pip, cabal, bundler, gem, composer, pear, cpan, ctan, cran, maven, vundle – haven't you also a love hate relationship with these?

e12e 13y ago

Also like the bootstrap.sh - very nicely structured and easier to verify than many others I've seen. I'd still prefer these scripts to output missing dependencies that calling sudo -- but otherwise very nice. That is, rather than: function install_git() { echo 'Trying to install git' case $(uname -s) in #... Linux) if has_exec apt-get; then apt_update sudo apt-get install -y git #... else bail "Unknown linux variant" fi ;; *) bail "Unknown operating system $(uname -s)" ;; esac } Simply aggreagate a list of missing packages, and ouput (based on a similar case-statement for detetining host): echo Please install missing packages: echo sudo apt-get install ${missing-deps} https://github.com/larsyencken/marelle/blob/master/bootstrap.sh https://github.com/larsyencken/marelle/blob/master/bootstrap...

lars512 13y ago
Thanks! :) It's a little aggressive in calling sudo mainly because I'm using it to bootstrap new systems non-interactively. I don't want it to ask any questions at all, just to go.

acjohnson55 13y ago

It's been a while since I've studied Prolog, but it's good to see it getting some press. I learned Prolog before ever dabbling in Scheme or ML, and it was just as much of a paradigm shift from imperative style. It's kind of like FP-style pattern matching, but taken to a whole new level, such that the pattern matching itself is pretty much the entire computational engine. Rather than using functions, which take a number of arguments and return a result, you use predicates on a number arguments. You build your program out of clauses, which are conjunctions of predicates. A process called unification [1] finds the argument values (if any) that satisfy an entire clause. By sharing constants and variables between predicates, the valid results that unification can produce are restricted to what the logic of your program allows. So your algorithms end up being, in many ways, description of what a valid results look like, and the Prolog engine goes out and finds them. This paradigm clearly maps nicely to domains with lots of rules, like decision engines, parsers, and such. Sounds awesome, right? One caveat is that, in practice, you as a programmer have to have a pretty good understanding of how unification works in order to write performant algorithms. This is all from several years-old recollections and a little bit of Wikipedia to refresh my memory, so someone please chime in if I'm incorrect someplace or leaving out key concepts. [1] http://en.wikipedia.org/wiki/Unification_%28computer_science%29#Application:_Unification_in_logic_programming http://en.wikipedia.org/wiki/Unification_%28computer_science...

mickeyp 13y ago

Prolog's a surprisingly good fit for many every-day tasks where people otherwise invent DSLs or poorly-defined state machines and pattern matching algorithms. So how come we don't use it more often? Well, the reason is actually quite simple: Prolog's incredibly difficult to reason about once you grow past trivial facts and clauses like in the OP's article -- the examples given by him would've been introduced to you in the first lesson on Prolog in, say, University. The complexities of green/red cuts, backtracking and the procedural nature of Prolog makes it a really difficult language to truly learn -- much less one to specify complex programs in. I've never programmed in a language where 5-6 lines of errant Prolog code could stump myself, a post grad and a professor before until I did Prolog.

keenerd 13y ago
It is a shame really. Declarative programming is so much simpler to think about and extremely powerful. I've been working on trying to make declarative programming more accessible and while I personally have seen 60x improvements in ability, I have been unable to convince any of my teammates to give it a whirl.
- X4 13y ago
  @keenerd try to build a DSL ontop of prolog and they'll come & play.
  - keenerd 13y ago
    It is a library that works with any language, no need to learn a new one. Two people can even work on the same problem in different languages and seamlessly combine their work into one result.
    - X4 13y ago
      interesting! what is it for?
chubot 13y ago
So I'm not experienced with Prolog, but the thing that scared me off is that it doesn't appear to have any computational complexity guarantees. It seems like will scale very poorly as your data gets bigger. It's just a notation for exhaustive search as far as I know, and you have to add ad hoc cuts outside the declarative model to get reasonable performance. I have experience with other abstractions which are supposed to be "declarative", but you really have to peek under the hood at the implementation to get them to work in production settings, and I don't like that.

calpaterson 13y ago

I'm skeptical, because there is already a configuration management language that is strongly influenced by Prolog and logic programming - Puppet - and the results are not that great. The secret of configuration management is that ordering NEEDS to be a first class feature and ordered manifests should be the default, not the option. Puppet suffers hugely from the influence from the fact that ordering is unpredictable and that you have to use arrows or requires to add order. People accidentally forget to add ordering, the resulting code works most of the time, and then one day you start running into an issue because some implicit dependency either didn't happen, or happened in the wrong order. In as much as all logic programming involves not explicitly controlling flow, I think they're inappropriate for configuration management. Sadly, I'm still stuck with using Puppet because of the organisation I work in. Having worked with Prolog in the past, I would also suggest that an important feature of a configuration management language is that the syntax make it hard to make typos and other minor errors because the feedback loop of deploying your code against a local vagrant machine is already long enough. I vividly remember the annoyance of having missed the full stop to end a block. I'm not desperate to repeat that experience.

gwillem 13y ago
That's the main reason we've switched over from Puppet to Ansible. It has less abstraction layers, but at least its execution order is predictable. The easy learning curve for new colleagues and the compact syntax are also nice benefits.
- lobster_johnson 13y ago
  That's what I don't understand about Ansible. It's basically -- literally, actually -- Puppet reinvented with a new syntax based on YAML. Puppet works so well, Ansible seems like a typical "not invented here" tragedy that frequently occurs in open source. Rather than coalesce around something that works and has years of experience and knowledge and bug fixing sunk into it, people create something from scratch. To be specific, this is Ansible: - name: Memcached template: src=$repository_basedir/memcached/templates/memcached.conf.j2 dest=/etc/memcached.conf owner=root group=root mode=0644 notify: - memcached-restart The same in Puppet: file { '/etc/memcached.conf': content => template('memcached/memcached.conf'), owner => root, group => root, mode => '0644', notify => Service['memcached']; } It's literally the same data expressed here. Puppet has very little abstraction. But what you do get, which Ansible has not really understood, I think, is a data model. The above results in the graph getting an object File['/etc/memcached.conf'] which can be referred to. I can do this: service { 'memcached': require => File['/etc/memcached.conf']; } This in turn creates the graph node Service['memcached']. It takes part in a larger graph that looks like this: Stage[main]/Node[myserver]/Service['memcached'] In fact, manifests are all compiled into a large graph like this, with interdependencies between the nodes. The logic-based, Prolog-type language in the OP's link is precisely what Puppet builds underneath. Likewise, the graph model is extensible. I can invent a new object like so: define public_key() { file { "/home/$name/.ssh/authorized_keys": content => "$key"; } } public_key { 'someuser': key => 'some public key'; } account { 'someuser': require => PublicKey['someuser']; } Here, the define public_key() actually defines a new type of object that can be instantiated in scripts. When one is instantiated, other objects can be depend on it. This allows me to encapsulate stuff in my scripts; rather than having other declarations depend on the internal details of how a public key is stored (ie., in a home directory), they depend on my definition. Ansible may look simpler on the surface, but I think it's all surface. When you have learned to use Puppet, it becomes simple. Ansible looks to me like the product of a person who looked at Puppet, said something like "ugh Ruby, ew syntax" and made his own thing, without looking deeper. Simialrly, Ansible appeals to people who look at Puppet with the same reaction, again without looking deeper. The irony is that they are buying into something that is arguably poorer. That said, I would argue that part of the problem is Puppet's documentation, which surely must drive people to "poor man's Puppet" solutions. (The cynic in me wonders if Puppet's poor documentation is a deliberate ploy on part of Puppet Labs to boost their consulting business.)

deleted 13y ago

[deleted]