4 ms·

Sorry if this is a dumb question but I've never understood the purpose of these configuration stores. If you're not running in the cloud, but have servers in a

by kbd 13y ago

Sorry if this is a dumb question but I've never understood the purpose of these configuration stores. If you're not running in the cloud, but have servers in a datacenter that all mount an NFS share, is there any benefit of these over simply reading a json/yaml file off of an NFS mount?

yummyfajitas 13y ago
Or if you are running in the cloud, simply reading a json/yaml file off S3?
- cconger 13y ago
  I think there are a few key benefits to a centralized manager over the distributed file listener. 1. coordination between services on reconf. 2. Consistent implementation on what constitutes a change through the API on the central server. By making it API based you can hook into these updates and cause reconfs and coordinated responses (like rolling bounces) through that system as opposed to each system polling the file and hoping that the order comes out in the wash. You could with enough work make it so that a client was aware of how to handle individual diffs from the file and coordinates through the file but at that point your now distributing common parsing logic across multiple systems (and potentially implementations) rather than a central system which sounds awfully un-DRY.
regularfry 13y ago
Resiliency to your NFS server falling over is one. Sane atomic updates and locking is another.
- kbd 13y ago
  Atomic updates on an NFS share can be achieved with renaming changed files into place, no?
  - nknighthb 13y ago
    It sounds like you're expecting NFS to be a reliable, proven, high-performance technology. It's anything but. I've never encountered an implementation without horrible bugs, most commonly involving locking and atomicity issues. Things get oh-so-much worse if you have to mix implementations, and performance is always bad. I would never consider NFS an option in any production system. On a more general note, anything you build that relies on the particular semantics of any sort of traditional filesystem is sure to be wrong, either now or in the future when it needs to be run on a different filesystem. This is an area of software engineering that's a serious pain in the ass. Avoid it whenever you can.
- vidarh 13y ago
  HA NFS is not hard. Using DRDB to replicate the underlying block device, or Gluster to replicate the underlying filesystem, plus IP takeover or e.g. keeaplived is fairly trivial to set up. Running NFS on a single server is a bit like running one of these configuration management systems on a single server - they won't be resilient then either. Using Gluster directly is another easy way of getting resiliency. Atomic updates and locking is another matter, but for a lot of setups it's simply not needed.
  - regularfry 13y ago
    Or you can run a single tool that is designed for this one job. Gluster in particular isn't a panacea for resiliency, you've got to really know where it departs from POSIX to not create problems for yourself.
    - Diederich 13y ago
      Are the GlusterFS people mistaken? http://gluster.org/community/documentation/index.php/GlusterFS_General_FAQ#What_file_system_semantics_does_GlusterFS_Support.3B_is_it_fully_POSIX_compliant.3F http://gluster.org/community/documentation/index.php/Gluster... "GlusterFS is fully POSIX compliant."
      - regularfry 13y ago
        Ooh, news to me. That certainly wasn't the case 2 years ago.
rafekett 13y ago
Leader election in addition to what else has been said
- kbd 13y ago
  Leader election - isn't that just a problem created by using an external cluster of servers for storing config? If you were just reading a file, there's no issue of selecting a master from your NFS share. The two rationales I've heard so far are A. a cluster of one of these config servers will provide better uptime than just an NFS share and B. you can use one of these things as a distributed lock server.

by kbd 13y ago

yummyfajitas 13y ago

Or if you are running in the cloud, simply reading a json/yaml file off S3?

cconger 13y ago
I think there are a few key benefits to a centralized manager over the distributed file listener. 1. coordination between services on reconf. 2. Consistent implementation on what constitutes a change through the API on the central server. By making it API based you can hook into these updates and cause reconfs and coordinated responses (like rolling bounces) through that system as opposed to each system polling the file and hoping that the order comes out in the wash. You could with enough work make it so that a client was aware of how to handle individual diffs from the file and coordinates through the file but at that point your now distributing common parsing logic across multiple systems (and potentially implementations) rather than a central system which sounds awfully un-DRY.

regularfry 13y ago

Resiliency to your NFS server falling over is one. Sane atomic updates and locking is another.

kbd 13y ago
Atomic updates on an NFS share can be achieved with renaming changed files into place, no?
- nknighthb 13y ago
  It sounds like you're expecting NFS to be a reliable, proven, high-performance technology. It's anything but. I've never encountered an implementation without horrible bugs, most commonly involving locking and atomicity issues. Things get oh-so-much worse if you have to mix implementations, and performance is always bad. I would never consider NFS an option in any production system. On a more general note, anything you build that relies on the particular semantics of any sort of traditional filesystem is sure to be wrong, either now or in the future when it needs to be run on a different filesystem. This is an area of software engineering that's a serious pain in the ass. Avoid it whenever you can.
vidarh 13y ago
HA NFS is not hard. Using DRDB to replicate the underlying block device, or Gluster to replicate the underlying filesystem, plus IP takeover or e.g. keeaplived is fairly trivial to set up. Running NFS on a single server is a bit like running one of these configuration management systems on a single server - they won't be resilient then either. Using Gluster directly is another easy way of getting resiliency. Atomic updates and locking is another matter, but for a lot of setups it's simply not needed.
- regularfry 13y ago
  Or you can run a single tool that is designed for this one job. Gluster in particular isn't a panacea for resiliency, you've got to really know where it departs from POSIX to not create problems for yourself.
  - Diederich 13y ago
    Are the GlusterFS people mistaken? http://gluster.org/community/documentation/index.php/GlusterFS_General_FAQ#What_file_system_semantics_does_GlusterFS_Support.3B_is_it_fully_POSIX_compliant.3F http://gluster.org/community/documentation/index.php/Gluster... "GlusterFS is fully POSIX compliant."
    - regularfry 13y ago
      Ooh, news to me. That certainly wasn't the case 2 years ago.

rafekett 13y ago

Leader election in addition to what else has been said

kbd 13y ago
Leader election - isn't that just a problem created by using an external cluster of servers for storing config? If you were just reading a file, there's no issue of selecting a master from your NFS share. The two rationales I've heard so far are A. a cluster of one of these config servers will provide better uptime than just an NFS share and B. you can use one of these things as a distributed lock server.