6 ms·

Etcd Clustering in AWS

95 points by polvi 11y ago

jefe78 11y ago
... thanks Monsanto? In all seriousness, this is really interesting. They solved some of the problems associated with persisting a cluster and we're likely going to use that. Feels weird thanking them for anything though. Edit: Is anyone using CoreOS in a physical DC? We're using AWS with ~1.5k VMs but have another 5-6k hosts in physical DCs. Trying to move us towards containers but struggling.
- mooreds 11y ago
  I just sent an email to some clients that I've been trying to get to blog about technical issues (for recruiting and retention purposes)--if Monsanto can do it, most anyone can.
  - justizin 11y ago
    They're certainly known for not always knowing the difference between 'can' and 'should'. ;)
- mkiwala 11y ago
  I'm using coeros to operate a deis cluster. We deploy to our own data center using openstack.
KnownSubset 11y ago
From my experience etcd is pretty rock solid, until you start using it across availability zones. Then if you add in SSL into the mix, the reliability drops even further if you are using the default configuration. At that point you need to start tweaking the heartbeat and timeout parameters for a the cluster to stay stable.
- ideal0227 11y ago
  We have fixed the SSL issue in 2.1. We are also considering back-port it to 2.0 release if possible.
justizin 11y ago
"If this happened our cluster would become unavailable and may have trouble re-clustering." This was basically the repeated experience I had which caused me to abandon etcd for the time being. If it can barely ever heal, what the fuck good is it? And I found that it could barely ever heal. A 3-node CoreOS cluster I ran _always_ crashed when it attempted a coordinated update, and rarely could be repaired with the help of #CoreOS over hours. Because CoreOS pushes out updates with versions of etcd incompatible with recent versions, the etcd cluster could never survive the upgrade. Add this to the fact that the CEO of CoreOS told me in person that he expected them to be the _only_ Operating System on the internet, and I'm generally not along for the ride with CoreOS any longer. Consul, Mesos, and Docker are looking good. Anyone interested in this space should check out: https://github.com/CiscoCloud/microservices-infrastructure
- jefe78 11y ago
  Have you tried using Mesos? We're doing a POC but ran into some issues that we're going to wait out. Also, I've spoken to Mesos and they stated that they had no intentions to make deployments easier/more stable, in favour of pushing their commercial offering.
  - lobster_johnson 11y ago
    > Also, I've spoken to Mesos Mesos is an Apache project, did you mean Mesosphere?
    - justizin 11y ago
      Yeah, we're clearly talking about Mesosphere.
    - jefe78 11y ago
      The context makes that a silly question.
      - lobster_johnson 11y ago
yeukhon 11y ago
I think they fixed etcd cluster problem in 2.0 release (previously this is 0.5 branch). For example, we use CF (old version), and we hit https://github.com/coreos/etcd/issues/863 https://github.com/coreos/etcd/issues/863.
gct 11y ago
I can't get over how bad a name etcd is. Everytime I see it I think it's some sort of daemon for /etc files.
- phildougherty 11y ago
  I think what they were going for was: /etc is where your config files go on your server etcd is where your config goes for the distributed system
- MikeTLive 11y ago
  etcd IS a mashup of /etc (configuration files) and distributed. https://youtu.be/z6tjawXZ71E?t=30s https://youtu.be/z6tjawXZ71E?t=30s
codewithcheese 11y ago
Running docker clusters on AWS seems a little foolish to me unless your trying to save money. Instead of manage containers why not just manage instances?
- vidarh 11y ago
  If you're trying to save money, you wouldn't be using AWS in the first place. Here's one reason to run Docker on AWS: Putting everything in Docker containers makes it far easier to migrate off AWS if/when you want to.
  - brianwawok 11y ago
    Well unless you use sqs and sns and s3 and redshift... Then you are pretty locked in anyway
- pnathan 11y ago
  boot time is one reason. another reason is integration with the aws ecosystem of services - it's not just ec2 these days.
narsil 11y ago
We solve the bootstrapping problem with an internal ELB instead. Autoscaling Groups can be configured to have instances join multiple ELBs. We have one be the regular ELB to access the instances with, and the other is an internal ELB that only allows connections from instances in the cluster to other instances in the cluster on the etcd port (controlled via security groups). When an instance comes up, it adds itself to the cluster via the internal ELB's hostname. The hostname is set in Route 53. The biggest issues we've been having with etcd continue to be simultaneous reboots and/or joins to the cluster. It would also be great if the membership timeout feature that used to exist in 0.4 made its way back in. Right now, each member has to be explicitly removed rather than eventually timing out if it hasn't joined back in. Looking forward to hear any other approaches folks have taken.

95 points by polvi 11y ago

... thanks Monsanto? In all seriousness, this is really interesting. They solved some of the problems associated with persisting a cluster and we're likely going to use that. Feels weird thanking them for anything though. Edit: Is anyone using CoreOS in a physical DC? We're using AWS with ~1.5k VMs but have another 5-6k hosts in physical DCs. Trying to move us towards containers but struggling.

mooreds 11y ago
I just sent an email to some clients that I've been trying to get to blog about technical issues (for recruiting and retention purposes)--if Monsanto can do it, most anyone can.
- justizin 11y ago
  They're certainly known for not always knowing the difference between 'can' and 'should'. ;)
mkiwala 11y ago
I'm using coeros to operate a deis cluster. We deploy to our own data center using openstack.

KnownSubset 11y ago

From my experience etcd is pretty rock solid, until you start using it across availability zones. Then if you add in SSL into the mix, the reliability drops even further if you are using the default configuration. At that point you need to start tweaking the heartbeat and timeout parameters for a the cluster to stay stable.

ideal0227 11y ago
We have fixed the SSL issue in 2.1. We are also considering back-port it to 2.0 release if possible.

justizin 11y ago

"If this happened our cluster would become unavailable and may have trouble re-clustering." This was basically the repeated experience I had which caused me to abandon etcd for the time being. If it can barely ever heal, what the fuck good is it? And I found that it could barely ever heal. A 3-node CoreOS cluster I ran _always_ crashed when it attempted a coordinated update, and rarely could be repaired with the help of #CoreOS over hours. Because CoreOS pushes out updates with versions of etcd incompatible with recent versions, the etcd cluster could never survive the upgrade. Add this to the fact that the CEO of CoreOS told me in person that he expected them to be the _only_ Operating System on the internet, and I'm generally not along for the ride with CoreOS any longer. Consul, Mesos, and Docker are looking good. Anyone interested in this space should check out: https://github.com/CiscoCloud/microservices-infrastructure

jefe78 11y ago
Have you tried using Mesos? We're doing a POC but ran into some issues that we're going to wait out. Also, I've spoken to Mesos and they stated that they had no intentions to make deployments easier/more stable, in favour of pushing their commercial offering.
- lobster_johnson 11y ago
  > Also, I've spoken to Mesos Mesos is an Apache project, did you mean Mesosphere?
  - justizin 11y ago
    Yeah, we're clearly talking about Mesosphere.
  - jefe78 11y ago
    The context makes that a silly question.
    - lobster_johnson 11y ago

yeukhon 11y ago

I think they fixed etcd cluster problem in 2.0 release (previously this is 0.5 branch). For example, we use CF (old version), and we hit https://github.com/coreos/etcd/issues/863 https://github.com/coreos/etcd/issues/863.

gct 11y ago

I can't get over how bad a name etcd is. Everytime I see it I think it's some sort of daemon for /etc files.

phildougherty 11y ago
I think what they were going for was: /etc is where your config files go on your server etcd is where your config goes for the distributed system
MikeTLive 11y ago
etcd IS a mashup of /etc (configuration files) and distributed. https://youtu.be/z6tjawXZ71E?t=30s https://youtu.be/z6tjawXZ71E?t=30s

codewithcheese 11y ago

Running docker clusters on AWS seems a little foolish to me unless your trying to save money. Instead of manage containers why not just manage instances?

vidarh 11y ago
If you're trying to save money, you wouldn't be using AWS in the first place. Here's one reason to run Docker on AWS: Putting everything in Docker containers makes it far easier to migrate off AWS if/when you want to.
- brianwawok 11y ago
  Well unless you use sqs and sns and s3 and redshift... Then you are pretty locked in anyway
pnathan 11y ago
boot time is one reason. another reason is integration with the aws ecosystem of services - it's not just ec2 these days.

narsil 11y ago

We solve the bootstrapping problem with an internal ELB instead. Autoscaling Groups can be configured to have instances join multiple ELBs. We have one be the regular ELB to access the instances with, and the other is an internal ELB that only allows connections from instances in the cluster to other instances in the cluster on the etcd port (controlled via security groups). When an instance comes up, it adds itself to the cluster via the internal ELB's hostname. The hostname is set in Route 53. The biggest issues we've been having with etcd continue to be simultaneous reboots and/or joins to the cluster. It would also be great if the membership timeout feature that used to exist in 0.4 made its way back in. Right now, each member has to be explicitly removed rather than eventually timing out if it hasn't joined back in. Looking forward to hear any other approaches folks have taken.