4 ms·

HN, I'm a co-founder at Appuri. Happy to answer questions! PS: We LOVE most AWS services like Amazon Redshift. Just not ECS ;)

by maslam 10y ago

ntumlin 10y ago
Off topic from the article but just wanted to let you know that I love the design of your blog.
- maslam 10y ago
  Thank you!
dstroot 10y ago
Did you deploy K8S on AWS? If so can you add any details about how? Or are you using K8S elsewhere? I love AWS but planning on spinning up on GCE this weekend to play with K8S.
- tmacie 10y ago
  We deployed K8S on AWS (I'm a dev at Appuri). Like Bilal mentioned we run pretty much everything on AWS, so it was an easy decision.
- lobster_johnson 10y ago
  The kube-up.sh script can start a complete Kubernetes cluster on AWS in one fell swoop. It's pretty smooth.
- velkyk 10y ago
  I am ops at Appuri. We deployed k8s on AWS since we are using other services like Redshift and RDS inside the VPC, also happy with how EC2 works plus of course we have Reserved instances so we didn't look into GCP yet. We're running kube on CentOS 7, we bootstrap nodes using cloud-init (user-data) to setup k8s, which we then use to run everything else. I would love to give you more details, I might write blog post about our kube setup decisions later. Kelsey wrote nice manual for setting up k8s - https://github.com/kelseyhightower/kubernetes-the-hard-way https://github.com/kelseyhightower/kubernetes-the-hard-way which is definitely on my to-read list this weekend :)
  - boulos 10y ago
    How I wish we had Postgres support in Cloud SQL already... Depending on how many RIs you have, maybe our custom shapes, much lower prices with per-minute billing and sustained usage discounts, and BigQuery vs RedShift you'd still come out ahead. You could always try your hand at reselling your RIs on the "market". Would you rather run pgsql by hand or K8s? (And yes, I don't want to make you choose, that's just the current state of the world). Disclosure: I work on Compute Engine, so of course we want your business.
    - velkyk 10y ago
      heh, I would rather run psql as a service which is what we have with RDS :). We are aware of BigQuery advantages, Redshift resizing is tedious job, but we currently don't think that the cost of migrating to GCP would be so beneficial that it is worth the effort. This can of course change in future, but for now we are reasonably happy with our setup on AWS and its reliability.
      - boulos 10y ago
        I knew it. I'll chalk you up as yet another customer that "might" move if only we had managed Postgres.
      - vgt 10y ago
        Perhaps you can take a look at some folks who've done this with great success: https://cloud.google.com/customers/sharethis/ https://cloud.google.com/customers/sharethis/ https://youtu.be/6Nv18xmJirs?t=8m7s https://youtu.be/6Nv18xmJirs?t=8m7s
robbles 10y ago
Did you run ECS on a custom AMI, or use the stock one? We've been running with vanilla EB + ECS for months and haven't seen this at all. From an outside perspective, it sounds like the primary issue you referenced here (the agent disconnecting) could have been due to a mismatch in configuration between the agent and docker, or maybe just a permissions issue. IIRC, the ECS agent tries to clean up containers every few hours so perhaps not being configured correctly caused it to get stuck?
- tmacie 10y ago
  We used the stock AMI. I would not be surprised if we had a configuration issue, but we spent a lot of time trying to debug it and were never able to find the root cause of the issue.
  - velkyk 10y ago
    yep, we were using Amazon ECS-Optimized Amazon Linux AMI, docker and agent should've matched there. It was strange behavior, either all agents were connected or most of them were disconnected at the same time...
    - maslam 10y ago
      Yes. This is pretty much what we saw.
    - robbles 10y ago
      How did you first discover the bug? Did your deploys just start failing in an obvious way, or did your instances disappear (from ECS's perspective)? I'm wondering if Elastic Beanstalk is hiding this from us in some way. Or maybe just has a workaround built in?

HN, I'm a co-founder at Appuri. Happy to answer questions! PS: We LOVE most AWS services like Amazon Redshift. Just not ECS ;)

by maslam 10y ago

HN, I'm a co-founder at Appuri. Happy to answer questions! PS: We LOVE most AWS services like Amazon Redshift. Just not ECS ;)

ntumlin 10y ago

Off topic from the article but just wanted to let you know that I love the design of your blog.

maslam 10y ago
Thank you!

dstroot 10y ago

Did you deploy K8S on AWS? If so can you add any details about how? Or are you using K8S elsewhere? I love AWS but planning on spinning up on GCE this weekend to play with K8S.

tmacie 10y ago
We deployed K8S on AWS (I'm a dev at Appuri). Like Bilal mentioned we run pretty much everything on AWS, so it was an easy decision.
lobster_johnson 10y ago
The kube-up.sh script can start a complete Kubernetes cluster on AWS in one fell swoop. It's pretty smooth.
velkyk 10y ago
I am ops at Appuri. We deployed k8s on AWS since we are using other services like Redshift and RDS inside the VPC, also happy with how EC2 works plus of course we have Reserved instances so we didn't look into GCP yet. We're running kube on CentOS 7, we bootstrap nodes using cloud-init (user-data) to setup k8s, which we then use to run everything else. I would love to give you more details, I might write blog post about our kube setup decisions later. Kelsey wrote nice manual for setting up k8s - https://github.com/kelseyhightower/kubernetes-the-hard-way https://github.com/kelseyhightower/kubernetes-the-hard-way which is definitely on my to-read list this weekend :)
- boulos 10y ago
  How I wish we had Postgres support in Cloud SQL already... Depending on how many RIs you have, maybe our custom shapes, much lower prices with per-minute billing and sustained usage discounts, and BigQuery vs RedShift you'd still come out ahead. You could always try your hand at reselling your RIs on the "market". Would you rather run pgsql by hand or K8s? (And yes, I don't want to make you choose, that's just the current state of the world). Disclosure: I work on Compute Engine, so of course we want your business.
  - velkyk 10y ago
    heh, I would rather run psql as a service which is what we have with RDS :). We are aware of BigQuery advantages, Redshift resizing is tedious job, but we currently don't think that the cost of migrating to GCP would be so beneficial that it is worth the effort. This can of course change in future, but for now we are reasonably happy with our setup on AWS and its reliability.
    - boulos 10y ago
      I knew it. I'll chalk you up as yet another customer that "might" move if only we had managed Postgres.
    - vgt 10y ago
      Perhaps you can take a look at some folks who've done this with great success: https://cloud.google.com/customers/sharethis/ https://cloud.google.com/customers/sharethis/ https://youtu.be/6Nv18xmJirs?t=8m7s https://youtu.be/6Nv18xmJirs?t=8m7s

robbles 10y ago

Did you run ECS on a custom AMI, or use the stock one? We've been running with vanilla EB + ECS for months and haven't seen this at all. From an outside perspective, it sounds like the primary issue you referenced here (the agent disconnecting) could have been due to a mismatch in configuration between the agent and docker, or maybe just a permissions issue. IIRC, the ECS agent tries to clean up containers every few hours so perhaps not being configured correctly caused it to get stuck?

tmacie 10y ago
We used the stock AMI. I would not be surprised if we had a configuration issue, but we spent a lot of time trying to debug it and were never able to find the root cause of the issue.
- velkyk 10y ago
  yep, we were using Amazon ECS-Optimized Amazon Linux AMI, docker and agent should've matched there. It was strange behavior, either all agents were connected or most of them were disconnected at the same time...
  - maslam 10y ago
    Yes. This is pretty much what we saw.
  - robbles 10y ago
    How did you first discover the bug? Did your deploys just start failing in an obvious way, or did your instances disappear (from ECS's perspective)? I'm wondering if Elastic Beanstalk is hiding this from us in some way. Or maybe just has a workaround built in?