3 ms·

I am not so sure that every possible update can be performed with those go templates, so I wrote a python version of confd (pyconfd) that uses polling and pytho

by kanzure 12y ago

I am not so sure that every possible update can be performed with those go templates, so I wrote a python version of confd (pyconfd) that uses polling and python plugins that do whatever they please: https://github.com/kanzure/pyconfd https://github.com/kanzure/pyconfd What would be really neat is some way of having consul initiate an event instead of having pyconfd (or confd for that matter) sleep and check every n seconds. This may already be implemented in consul upstream but I haven't been paying very close attention... Edit: dry run is very neat :-)

mitchellh 12y ago
Consul supports long polling. This is what Consul Template uses. With long polling, the changes are edge triggered so there is very little delay. We also compile and read the template once to determine only the keys you care about from the template to determine what Consul services/keys to look for, so we're not requesting ANY changes.
deleted 12y ago
[deleted]
kelseyhightower 12y ago
Again not to take anything away from consul-template or pyconfd, but confd supports "real-time" updates from both consul and etcd, the watch support was added about 2 weeks ago on the master branch. confd also has dry run support: https://github.com/kelseyhightower/confd/blob/master/docs/noop-mode.md https://github.com/kelseyhightower/confd/blob/master/docs/no... It was apart of the initial release of confd. These distributed K/V stores are designed to offer a minimal feature set, and the watch support is a perfect way to deliver events to clients. While HTTP call backs would be nice, I think you have to draw the line somewhere. Since these types of systems, consul or etcd, sit at a critical point in the infrastructure lets keep them lean and focused. But that's just my opinion.
- kanzure 12y ago
  Hey that watch stuff is pretty neat, I might have to switch back in some of my containers :-) https://github.com/kelseyhightower/confd/commit/16d6b347c3d335f6b26ce7422c608a7141f85b86 https://github.com/kelseyhightower/confd/commit/16d6b347c3d3...
tobz 12y ago
I also wrote something (before confd but I've yet to see anything that replicates it) for treating etcd like a filesystem and doing edge-triggered updates. You form directory structures, and those long keys get mapped to files on disk with their value as the file contents. What really bugs me about these solutions -- confd, consul templates -- is that managing templated configuration files isn't as quick, but has been trivially possible with normal configuration management tools since their inception. When I think of managing HAProxy configurations, for example, I don't see Chef as a bottleneck. I can parallel SSH and run Chef on those hosts in 20 - 30 seconds. I don't care about the configuration being pristine, either.. because that's what backend health checks are for. That extends to a lot of their examples, but it's one of the ones that stuck out to me. The things that I think are hard, or harder, are small bits of data, like database credentials, etc.... where people absolutely don't want to be making an HTTP call for every single thing, or don't want to define a template for hundreds of individual files that might be needed. That's a real bottleneck - you need something on top of these solutions to manage that. So for every file you care about being updated fast... you still need to manage a corresponding file. I don't know necessarily how to solve that -- I treat K/V pairs as file name/contents pairs in my tool and don't think twice -- but it definitely feels a bit wrong.

by kanzure 12y ago

mitchellh 12y ago

Consul supports long polling. This is what Consul Template uses. With long polling, the changes are edge triggered so there is very little delay. We also compile and read the template once to determine only the keys you care about from the template to determine what Consul services/keys to look for, so we're not requesting ANY changes.

deleted 12y ago

[deleted]

kelseyhightower 12y ago

Again not to take anything away from consul-template or pyconfd, but confd supports "real-time" updates from both consul and etcd, the watch support was added about 2 weeks ago on the master branch. confd also has dry run support: https://github.com/kelseyhightower/confd/blob/master/docs/noop-mode.md https://github.com/kelseyhightower/confd/blob/master/docs/no... It was apart of the initial release of confd. These distributed K/V stores are designed to offer a minimal feature set, and the watch support is a perfect way to deliver events to clients. While HTTP call backs would be nice, I think you have to draw the line somewhere. Since these types of systems, consul or etcd, sit at a critical point in the infrastructure lets keep them lean and focused. But that's just my opinion.

kanzure 12y ago
Hey that watch stuff is pretty neat, I might have to switch back in some of my containers :-) https://github.com/kelseyhightower/confd/commit/16d6b347c3d335f6b26ce7422c608a7141f85b86 https://github.com/kelseyhightower/confd/commit/16d6b347c3d3...

tobz 12y ago

I also wrote something (before confd but I've yet to see anything that replicates it) for treating etcd like a filesystem and doing edge-triggered updates. You form directory structures, and those long keys get mapped to files on disk with their value as the file contents. What really bugs me about these solutions -- confd, consul templates -- is that managing templated configuration files isn't as quick, but has been trivially possible with normal configuration management tools since their inception. When I think of managing HAProxy configurations, for example, I don't see Chef as a bottleneck. I can parallel SSH and run Chef on those hosts in 20 - 30 seconds. I don't care about the configuration being pristine, either.. because that's what backend health checks are for. That extends to a lot of their examples, but it's one of the ones that stuck out to me. The things that I think are hard, or harder, are small bits of data, like database credentials, etc.... where people absolutely don't want to be making an HTTP call for every single thing, or don't want to define a template for hundreds of individual files that might be needed. That's a real bottleneck - you need something on top of these solutions to manage that. So for every file you care about being updated fast... you still need to manage a corresponding file. I don't know necessarily how to solve that -- I treat K/V pairs as file name/contents pairs in my tool and don't think twice -- but it definitely feels a bit wrong.