4 ms·

Introducing Consul Template

101 points by Perceptes 12y ago

Oculus 12y ago
Probably the biggest feature over Confd is the ability to run arbitrary commands when a template update completes.
- kelseyhightower 12y ago
  Not to take anything away from consul-template, but that has been a feature of confd for a while: https://github.com/kelseyhightower/confd/blob/master/docs/template-resources.md#example https://github.com/kelseyhightower/confd/blob/master/docs/te...
  - Oculus 12y ago
    My bad, I misspoke.
im_dario 12y ago
It's neat! Where I work we are about to use Consul and this is great news for us as we were looking for something like Consul Template. I really like that they are simple Go templates :)
kelseyhightower 12y ago
confd author here. I'm glad to see this idea validated, and I think consul-template looks pretty awesome and well integrated with consul. consul-template takes advantage of consul specific features such as the consul services API and datacenters, which is really great for consul users. These are things I did not want to add and maintain in confd, since I'm supporting consul, etcd, and environment variables as backends I felt those would be one off features, which really belong in a consul focused tool like this. Some will ask why did Hashicorp not just contribute to confd? Well they did. They added initial support for consul to confd almost a year ago and have been very helpful with my questions around consul ever since. The contributions go both ways too. I've been a contributor to Hashicorp's Packer project -- I added Ansible and GCE support and various bug fixes. This is just how opensource works. I'm actually pretty happy to see one of my projects get so much attention and now inspire another project. That's the whole point right?
- deleted 12y ago
  [deleted]
splitrocket 12y ago
Hashicorp continues to kickass. However, I wonder why this isn't built into consul directly?
kanzure 12y ago
I am not so sure that every possible update can be performed with those go templates, so I wrote a python version of confd (pyconfd) that uses polling and python plugins that do whatever they please: https://github.com/kanzure/pyconfd https://github.com/kanzure/pyconfd What would be really neat is some way of having consul initiate an event instead of having pyconfd (or confd for that matter) sleep and check every n seconds. This may already be implemented in consul upstream but I haven't been paying very close attention... Edit: dry run is very neat :-)
- mitchellh 12y ago
  Consul supports long polling. This is what Consul Template uses. With long polling, the changes are edge triggered so there is very little delay. We also compile and read the template once to determine only the keys you care about from the template to determine what Consul services/keys to look for, so we're not requesting ANY changes.
- deleted 12y ago
  [deleted]
- kelseyhightower 12y ago
  Again not to take anything away from consul-template or pyconfd, but confd supports "real-time" updates from both consul and etcd, the watch support was added about 2 weeks ago on the master branch. confd also has dry run support: https://github.com/kelseyhightower/confd/blob/master/docs/noop-mode.md https://github.com/kelseyhightower/confd/blob/master/docs/no... It was apart of the initial release of confd. These distributed K/V stores are designed to offer a minimal feature set, and the watch support is a perfect way to deliver events to clients. While HTTP call backs would be nice, I think you have to draw the line somewhere. Since these types of systems, consul or etcd, sit at a critical point in the infrastructure lets keep them lean and focused. But that's just my opinion.
  - kanzure 12y ago
    Hey that watch stuff is pretty neat, I might have to switch back in some of my containers :-) https://github.com/kelseyhightower/confd/commit/16d6b347c3d335f6b26ce7422c608a7141f85b86 https://github.com/kelseyhightower/confd/commit/16d6b347c3d3...
maccam94 12y ago
It seems like there are a lot of similar competing projects in the service config space now. I was interested in trying this out last year: https://github.com/coderanger/etcd-chef https://github.com/coderanger/etcd-chef
s0l1dsnak3123 12y ago
I wrote a script to auto-reload HA_proxy if nodes change in consul, now I'm glad I can now retire it and switch to this!
helfire 12y ago
This looks great and glad HashiCorp keeps releasing Opensource software! Anyone have a good end to end example of Consul, Docker, Service Discovery? Nginx example is good to get started, but just need some of the basics to point in the right direction.
- kanzure 12y ago
  > Anyone have a good end to end example of Consul, Docker, Service Discovery? There are a few floating around. Here's mine: https://github.com/kanzure/docker-basenode https://github.com/kanzure/docker-basenode Here's something useful for end-to-end: https://github.com/hashicorp/consul-haproxy https://github.com/hashicorp/consul-haproxy
ende42 12y ago
For scenarios where I felt etcd/gconf/consul is overkill I wrote goconfd (https://github.com/niko/goconfd https://github.com/niko/goconfd). Features: * reads config from a json file * GET requests with deep links into the json tree * POST request with template as body and deep links (using the go template engine, too) * long polling (response has to be triggered explicitly by a PUT request) I also made a simple docker container running HAProxy - configured by goconfd - to orchestrate docker containers.

101 points by Perceptes 12y ago

Oculus 12y ago

Probably the biggest feature over Confd is the ability to run arbitrary commands when a template update completes.

kelseyhightower 12y ago
Not to take anything away from consul-template, but that has been a feature of confd for a while: https://github.com/kelseyhightower/confd/blob/master/docs/template-resources.md#example https://github.com/kelseyhightower/confd/blob/master/docs/te...
- Oculus 12y ago
  My bad, I misspoke.

im_dario 12y ago

It's neat! Where I work we are about to use Consul and this is great news for us as we were looking for something like Consul Template. I really like that they are simple Go templates :)

kelseyhightower 12y ago

confd author here. I'm glad to see this idea validated, and I think consul-template looks pretty awesome and well integrated with consul. consul-template takes advantage of consul specific features such as the consul services API and datacenters, which is really great for consul users. These are things I did not want to add and maintain in confd, since I'm supporting consul, etcd, and environment variables as backends I felt those would be one off features, which really belong in a consul focused tool like this. Some will ask why did Hashicorp not just contribute to confd? Well they did. They added initial support for consul to confd almost a year ago and have been very helpful with my questions around consul ever since. The contributions go both ways too. I've been a contributor to Hashicorp's Packer project -- I added Ansible and GCE support and various bug fixes. This is just how opensource works. I'm actually pretty happy to see one of my projects get so much attention and now inspire another project. That's the whole point right?

deleted 12y ago
[deleted]

splitrocket 12y ago

Hashicorp continues to kickass. However, I wonder why this isn't built into consul directly?

kanzure 12y ago

I am not so sure that every possible update can be performed with those go templates, so I wrote a python version of confd (pyconfd) that uses polling and python plugins that do whatever they please: https://github.com/kanzure/pyconfd https://github.com/kanzure/pyconfd What would be really neat is some way of having consul initiate an event instead of having pyconfd (or confd for that matter) sleep and check every n seconds. This may already be implemented in consul upstream but I haven't been paying very close attention... Edit: dry run is very neat :-)

mitchellh 12y ago
Consul supports long polling. This is what Consul Template uses. With long polling, the changes are edge triggered so there is very little delay. We also compile and read the template once to determine only the keys you care about from the template to determine what Consul services/keys to look for, so we're not requesting ANY changes.
deleted 12y ago
[deleted]
kelseyhightower 12y ago
Again not to take anything away from consul-template or pyconfd, but confd supports "real-time" updates from both consul and etcd, the watch support was added about 2 weeks ago on the master branch. confd also has dry run support: https://github.com/kelseyhightower/confd/blob/master/docs/noop-mode.md https://github.com/kelseyhightower/confd/blob/master/docs/no... It was apart of the initial release of confd. These distributed K/V stores are designed to offer a minimal feature set, and the watch support is a perfect way to deliver events to clients. While HTTP call backs would be nice, I think you have to draw the line somewhere. Since these types of systems, consul or etcd, sit at a critical point in the infrastructure lets keep them lean and focused. But that's just my opinion.
- kanzure 12y ago
  Hey that watch stuff is pretty neat, I might have to switch back in some of my containers :-) https://github.com/kelseyhightower/confd/commit/16d6b347c3d335f6b26ce7422c608a7141f85b86 https://github.com/kelseyhightower/confd/commit/16d6b347c3d3...

maccam94 12y ago

It seems like there are a lot of similar competing projects in the service config space now. I was interested in trying this out last year: https://github.com/coderanger/etcd-chef https://github.com/coderanger/etcd-chef

s0l1dsnak3123 12y ago

I wrote a script to auto-reload HA_proxy if nodes change in consul, now I'm glad I can now retire it and switch to this!

helfire 12y ago

This looks great and glad HashiCorp keeps releasing Opensource software! Anyone have a good end to end example of Consul, Docker, Service Discovery? Nginx example is good to get started, but just need some of the basics to point in the right direction.

kanzure 12y ago
> Anyone have a good end to end example of Consul, Docker, Service Discovery? There are a few floating around. Here's mine: https://github.com/kanzure/docker-basenode https://github.com/kanzure/docker-basenode Here's something useful for end-to-end: https://github.com/hashicorp/consul-haproxy https://github.com/hashicorp/consul-haproxy

ende42 12y ago

For scenarios where I felt etcd/gconf/consul is overkill I wrote goconfd (https://github.com/niko/goconfd https://github.com/niko/goconfd). Features: * reads config from a json file * GET requests with deep links into the json tree * POST request with template as body and deep links (using the go template engine, too) * long polling (response has to be triggered explicitly by a PUT request) I also made a simple docker container running HAProxy - configured by goconfd - to orchestrate docker containers.