3 ms·

Etcd v0.1.2 with a new dashboard

62 points by polvi 13y ago

sudhirj 13y ago
Etcd and CoreOS look really great, but I'm still missing a fundamental question. If I've got a set of CoreOS instances running on EC2 with auto scaling on, how does a new instance add itself to the cluster when it starts up? How does it know which other servers to associate itself to, and how do these server know to trust it?
- philips 13y ago
  > How does it know which other servers to associate itself to Right now we have a service, called coordination (c10n), which helps boot strap an example cluster on ec2. The basic idea is that in order to get things going you need to know the IP of at least one other machine and c10n holds on to this information. The root for this bootstrap is a unique token you put in the AWS user data. You either need an external service like c10n with a shared token or a stable hostname that you setup yourself that can distribute the initial seed of information. An external service makes for an easier experience. > and how do these server know to trust it? As we get further along we will be adding a API/dashboard to help with boot strapping on etcd and approving/denying new machines as followers. That is part of why we added client certs to etcd from the very beginning. Re: autoscaling: Our focus at the moment is getting the core services, like etcd, working properly so that we can correctly build higher order concepts like horizontal auto scaling. More to come. Thanks for the questions.
  - secure 13y ago
    Similarly, my fundamental question is how you would implement master election for applications running within a docker container using etcd. Or is master election only for etcd itself and not a capability it provides? Also, all the examples just store simple key/value pairs. What are your best practices for actual config files? Do you store e.g. key=/etc/nginx/nginx.conf value=<contents of that file>? Does that imply all machines will have the same config? Or do you have the machine name in the key space? Or is etcd not actually for this kind of configs? If not, for which kind then?
    - philips 13y ago
      > how you would implement master election for applications running within a docker container using etcd. You build master election on top of the test and set functionality of etcd which gives you a cluster wide locking mechanism. The plan is to provide a simplified master election REST api as a module, just like the dashboard in this release is a module. But, nothing is stopping a regular etcd client from building master election on top of the current API. Something I would like to add to etcdctl is a config file generator which reads from the keys of etcd and fills in fields. You could add the entire file but it is more reasonable to simply add the metadata that is dynamic into etcd and ship a template around. Another example of this is this chef integration: https://github.com/coderanger/etcd-chef https://github.com/coderanger/etcd-chef For some applications, like load balancers, you may want to have the process itself interacting with an etcd cluster. e.g. this HTTP load balancer prototype: https://github.com/calavera/active-proxy https://github.com/calavera/active-proxy
      - ash 13y ago
        > You build master election on top of the test and set functionality of etcd which gives you a cluster wide locking mechanism... But, nothing is stopping a regular etcd client from building master election on top of the current API. Could you give some pointers on how to implement master election on top of test-and-set? What is the algorithm? I've tried searching for it, but I've likely missed correct keywords.
        ash 13y ago
        I've got reply here: https://github.com/coreos/etcd/issues/225 https://github.com/coreos/etcd/issues/225
    - SEJeff 13y ago
      Might I suggest you read up on other key value stores such as redis[1] (Although I don't think etcd supports the various data types redis does). Configuration data != configuration file. You could quite easily write something to generate a nginx config from etcd keys (or maybe do what you suggested if there isn't a size limit), but that's not really the normal idea of a K/V store. Ontop of that, etcd gives you a bulletproof storage system thanks to the raft distributed census system (conceptually similar to zookeeper's zab consensus algorithm[3]). [1] http://redis.io http://redis.io [2] http://highscalability.com/blog/2013/8/7/raft-in-search-of-an-understandable-consensus-algorithm.html http://highscalability.com/blog/2013/8/7/raft-in-search-of-a... [3] http://labs.yahoo.com/files/ladis08.pdf http://labs.yahoo.com/files/ladis08.pdf
deckiedan 13y ago
Hm. Looks kinda cool - the TTL option for keys could be a very useful tool. One question/problem: I can't find any kind of permissions/separation/ACLs for it. Yes, you can do authentication, but w/o even basic POSIX style owner/group/world type permissions, this essentially becomes single-user, with all apps/programs/etc trusted and able to break everyone else. Or am I missing something?
- philips 13y ago
  We are adding ACLs with POSIX style permissions. You can see the doc on the 0.2 branch: https://github.com/coreos/etcd/blob/0.2/Documentation/etcd-file-system.md#acl https://github.com/coreos/etcd/blob/0.2/Documentation/etcd-f...

62 points by polvi 13y ago

Etcd and CoreOS look really great, but I'm still missing a fundamental question. If I've got a set of CoreOS instances running on EC2 with auto scaling on, how does a new instance add itself to the cluster when it starts up? How does it know which other servers to associate itself to, and how do these server know to trust it?

philips 13y ago
> How does it know which other servers to associate itself to Right now we have a service, called coordination (c10n), which helps boot strap an example cluster on ec2. The basic idea is that in order to get things going you need to know the IP of at least one other machine and c10n holds on to this information. The root for this bootstrap is a unique token you put in the AWS user data. You either need an external service like c10n with a shared token or a stable hostname that you setup yourself that can distribute the initial seed of information. An external service makes for an easier experience. > and how do these server know to trust it? As we get further along we will be adding a API/dashboard to help with boot strapping on etcd and approving/denying new machines as followers. That is part of why we added client certs to etcd from the very beginning. Re: autoscaling: Our focus at the moment is getting the core services, like etcd, working properly so that we can correctly build higher order concepts like horizontal auto scaling. More to come. Thanks for the questions.
- secure 13y ago
  Similarly, my fundamental question is how you would implement master election for applications running within a docker container using etcd. Or is master election only for etcd itself and not a capability it provides? Also, all the examples just store simple key/value pairs. What are your best practices for actual config files? Do you store e.g. key=/etc/nginx/nginx.conf value=<contents of that file>? Does that imply all machines will have the same config? Or do you have the machine name in the key space? Or is etcd not actually for this kind of configs? If not, for which kind then?
  - philips 13y ago
    > how you would implement master election for applications running within a docker container using etcd. You build master election on top of the test and set functionality of etcd which gives you a cluster wide locking mechanism. The plan is to provide a simplified master election REST api as a module, just like the dashboard in this release is a module. But, nothing is stopping a regular etcd client from building master election on top of the current API. Something I would like to add to etcdctl is a config file generator which reads from the keys of etcd and fills in fields. You could add the entire file but it is more reasonable to simply add the metadata that is dynamic into etcd and ship a template around. Another example of this is this chef integration: https://github.com/coderanger/etcd-chef https://github.com/coderanger/etcd-chef For some applications, like load balancers, you may want to have the process itself interacting with an etcd cluster. e.g. this HTTP load balancer prototype: https://github.com/calavera/active-proxy https://github.com/calavera/active-proxy
    - ash 13y ago
      > You build master election on top of the test and set functionality of etcd which gives you a cluster wide locking mechanism... But, nothing is stopping a regular etcd client from building master election on top of the current API. Could you give some pointers on how to implement master election on top of test-and-set? What is the algorithm? I've tried searching for it, but I've likely missed correct keywords.
      - ash 13y ago
        I've got reply here: https://github.com/coreos/etcd/issues/225 https://github.com/coreos/etcd/issues/225
  - SEJeff 13y ago
    Might I suggest you read up on other key value stores such as redis[1] (Although I don't think etcd supports the various data types redis does). Configuration data != configuration file. You could quite easily write something to generate a nginx config from etcd keys (or maybe do what you suggested if there isn't a size limit), but that's not really the normal idea of a K/V store. Ontop of that, etcd gives you a bulletproof storage system thanks to the raft distributed census system (conceptually similar to zookeeper's zab consensus algorithm[3]). [1] http://redis.io http://redis.io [2] http://highscalability.com/blog/2013/8/7/raft-in-search-of-an-understandable-consensus-algorithm.html http://highscalability.com/blog/2013/8/7/raft-in-search-of-a... [3] http://labs.yahoo.com/files/ladis08.pdf http://labs.yahoo.com/files/ladis08.pdf

deckiedan 13y ago

Hm. Looks kinda cool - the TTL option for keys could be a very useful tool. One question/problem: I can't find any kind of permissions/separation/ACLs for it. Yes, you can do authentication, but w/o even basic POSIX style owner/group/world type permissions, this essentially becomes single-user, with all apps/programs/etc trusted and able to break everyone else. Or am I missing something?

philips 13y ago
We are adding ACLs with POSIX style permissions. You can see the doc on the 0.2 branch: https://github.com/coreos/etcd/blob/0.2/Documentation/etcd-file-system.md#acl https://github.com/coreos/etcd/blob/0.2/Documentation/etcd-f...