10 ms·

I am a big believer in containerization technology from a practical standpoint. It has allowed me to create repositories that act as services. Database, search,

I am a big believer in containerization technology from a practical standpoint. It has allowed me to create repositories that act as services. Database, search, api, admin, etc.. they are all their own service. I do not have to configure any servers this way; instead, I declare what the system ought to be and Docker makes it happen. I don't even have to configure init scripts because a proper Dockerfile will contain a start mechanism usually in the form of any other executable: `docker run your/api --port 80 --host host.company.com`. The only thing that matters then between services is their bindings, which gives you the ability to use any programming language for any service. Deployment with ECS has been going well so far for me. My flow: 1.) Push code to GitHub 2.) GitHub tells Docker, which builds a private image 3.) Docker tells my build server 4.) Build server tells ECS to update the given service 5.) ECS pulls from DockerHub, stops service, starts service The only thing missing is that DockerHub doesn't tell my build server what tag it just built! It builds tags like dev / staging for the given branches, but doesn't relay that info over its webhook. There's a ticket open about this already and I'm sure they'll get to it soon. Nevertheless, I'm able to administer any system -- things like Elasticsearch, db, api -- from code on a branch. This is powerful to me because I have to administer environments for everything. Rather than do all this work with Puppet, Chef, or even Ansible, I can just declare what the systems ought to be and cluster them within code branches. With ECS coming into picture, you're encouraged to forget you even have physical boxes in the first place. If you think of the power at your finger tips that results from this development workflow, I believe it's a no brainer for everyone to jump on board and get this as good as it can be. It's going to be a huge boon to the software community and enable more services sharing.

emiller829 11y ago
I'd love to hear how you're handling the ECS part of this. From the experimentation I've done, it's not just a matter of telling ECS to update the service unless you're running 2x the number of container instances you need -- it won't pull resources out from the existing service revision in order to spin up the new ones. Certainly can be worked around, but it's a little annoying.
- joslin01 11y ago
  There's two ways to go about this: 1.) Create a new task and update the service with the task. From the documentation (http://docs.aws.amazon.com/AmazonECS/latest/developerguide/update-service.html http://docs.aws.amazon.com/AmazonECS/latest/developerguide/u...): > If you have updated the Docker image of your application, you can create a new task definition with that image and deploy it to your service, one task at a time. The service scheduler creates a task with the new task definition (provided there is an available container instance to place it on), and after it reaches the RUNNING state, a task that is using the old task definition is drained and stopped. This process continues until all of the desired tasks in your service are using the new task definition. 2.) (way I'm doing it) Update the desired count to 0 and utilize wait_until (http://docs.aws.amazon.com/sdkforruby/api/Aws/ECS/Client.html#wait_until-instance_method http://docs.aws.amazon.com/sdkforruby/api/Aws/ECS/Client.htm...) services are stable. After this, update desired count to 1. You're going to get some downtime with #2. I'm not sure about #1 because I don't understand how something like an API service could be run without error'ing out that the port it's trying to bind to and forward is not available. So I'm not sure how it would ever reach the running state.. but I haven't experimented going about it that way. I can handle the small amount of downtime.
  - emiller829 11y ago
    Thanks! Yeah, those were the ony options I was aware of, but the "provided there is an available container instance to place it on" issue is a problem due to port availability, and I was looking for zero-downtime deploy options.
fosk 11y ago
On a side note, for microservices that expose a RESTful API I would also take a look at Kong (https://github.com/Mashape/kong https://github.com/Mashape/kong) for delegating common functionality like security or rate-limiting.
- simonebrunozzi 11y ago
  Made (and open-sourced) by the smart guys at Mashape.com. Thanks for doing this.
JTenerife 11y ago
I don't understand how you configure your host-specific servicc dependencies, e.g. connection strings. Ansible (Puppet, Chef) does variable interpolation / templating for this purpose.
- joslin01 11y ago
  I have configuration files for each environment, so `dev.config`, `staging.config`, etc...
  - michaelt 11y ago
    Do you deploy the same docker image to each environment, and somehow retrieve the configuration files on startup? Or do you rebuild your images as you promote from dev to staging to prod?
    - joslin01 11y ago
      Yes same docker image, but with different environment variables passed in for dev / staging / prod.
- superuser2 11y ago
  This is a hard problem, but outside the scope of Docker. There are numerous cluster orchestration/management systems springing up around the ecosystem. (We don't necessarily have on ready for prime-time yet, but a lot of very good work is taking place.)
jfoster 11y ago
What happens if there is a security flaw somewhere in your software stack? (kernel, web server, etc.) Do you end up having to update multiple containers with the same patch? Context: I've looked at Docker, but not used any containerization yet.
- geggam 11y ago
  Security ? How do you find out what libraries / versions are in a container ? ... you are pulling the curtain dont look back there :)
  - NeutronBoy 11y ago
    > Security ? How do you find out what libraries / versions are in a container ? ... The same way you do in any application, container or not - you look up your documentation ;)
  - jfoster 11y ago
    Is that really the state of things? It is what I concluded after looking into this a bit for Docker, but it seems incredible to me that so many companies are jumping into this idea of containerization without any good & available solutions for this problem. One potential solution that came to mind was that if there was a standard way of deploying an application into containers, and Google/Amazon/Microsoft provided auto-updating containers, the maintenance of a secure container would be in the hands of companies who (hopefully) have the resources necessary to keep the entire stack up-to-date.
    - digi_owl 11y ago
      They seem to be betting the farm on the containerization will contain (heh) whatever security issues that come up. This in the sense, i guess, that if they have a security flaw in their php that gives disk access, all the attacker will see is the content of the php container as the database will be on the next container over. Then again the containerization seems to have come alongside devops, where the mantra seems to be "update early, update often, to hell with stable branches".
jbhatab 11y ago
I am a more front end heavy web developer. I know my way around apis and what not, but what you are describing sounds like backend bliss. I don't even know where to start but is there someway you can share more about the magic of your process? Like an example of a github repo of search and how you plug that into an app? Either way, thanks.
- joshcanhelp 11y ago
  Would love to see this as well. I know my way around the CLI but not enough to even grok what Docker does. Any good overviews/getting starteds/tutorials out there?
  - joslin01 11y ago
    This is a somewhat novel workflow from what I can gather, so no. If you worked backwards from my flow though, you could figure it out no problem. Maybe I'll do a blog post in the future about it. I just don't have the time these days.
- pea 11y ago
  Hey, we're actually hacking on solving this exact problem at StackHut.com, which is a platform for containerised services that you can call immediately through HTTP. I think your sentiment is super-common: building microservices / container-enabled services and getting all the good stuff that comes with them is still too much of a learning curve for most devs. We're just about to push live the ability to add new services: would love to get your thoughts on what you'd like to see: leo@stackhut.com
nogox 11y ago
CaaS is definitely the future of cloud. However I don't think ECS (IaaS+Container) is the way to go. In this approach, you still need to take care of the cluster capacity planning, scaling, failover, etc. And at most time, there always are some capacity sitting idle in your ECS cluster. The evolution of CaaS would be sth. like hyper.sh. Due to its hypervisor nature, the whole cluster+scheduling thing can be made transparent to developers, who simply "compose" the app spec and submit to run.
xorcist 11y ago
Rather than using Puppet etc.? Basically you've returned to before configuration management and that is somehow good? How do you change network configuration, load balancers or other external configuration in lock step with your application? Scripting by hand? And deployments are a just small part of configuration management. How do you find out which of your applications use libz 1.2.3 (to measure CVE applicability for example)? By looking in each and every one of them? How do you find out which application run a cron job? And which application connect to backend x without going through the load balancer? Which application has a client certificate about to expire? How do you guarantee two applications have the same shared secret, and change it in lock step? With configuration management all this is just a look up away. And best of all, most of this information is authoritative. When I come somewhere new, the use of these tools instead of a directory of miscellaneous scripts is like night and day. It literally turns a two day job into a ten minute one when the environments are complex. Docker is great for a lot of things. It enables to do daring things to your environment even if it's so complex you don't fully understand it all because you can just shuffle containers around. But I would never use it instead of configuration management.
- jilted 11y ago
  Re: '...Rather than do all this work with Puppet, Chef, or even Ansible, I can just declare what the systems ought to be and cluster them within code branches...' It sounds like the poster is doing something similar to a 'Gold System Image' though with Container technologies. Configuration Management is great, but I'm with the view that you should start with a combination of gold images/system builds and then stack on Cfg Mgmt on top of that. Theory is you will have a known base and identical builds etc, otherwise you would get subtle drift in your configurations, especially for things which are not explicitly under control of Config Mgmt (shared library versions, packages). Of course, this is not always feasible but if I were to start from scratch, I would probably try to do it this way.
  - tjbiddle 11y ago
    It should be a combination, really. You want to have your "Gold system image" available and this is most certainly what you should be deploying from; however your configuration management - whether it's Chef, Puppet, whatever - should be able to take a base operating system and get it setup completely from scratch. This then solves the problem of ensure that you can completely reproduce your application from scratch; but also removes the possibly horrendous slow scale up time by using a "Gold image" that has already done this. My current process is: Jenkins runs puppet/chef, verifies that the application is healthy after the run and everything is happy, calls AWS and images the machine, then it iterates over all instances in my load balancer 33% at a time and replaces them with the new image resulting in zero-downtime. Of course another solution is to pull out those instances and apply the update, and then put them back in. And I'm sure someone else will have their $0.02 on their own process, which actually I'd love to hear :-)
pea 11y ago
Re services sharing, we are actually hacking on a database of useful containerized services which are hosted on Github and can be run anywhere through HTTP or JsonRPC. Product is super early, but would love your feedback on what you'd like to see: www.stackhut.com . Just about to push ability to submit your own. I think the key thing is that users shouldn't have to have a knowledge of Sysops or Devops to be able to take advantage of the benefits of containers. Most developer we've spoken to don't have time to learn a completely new paradigm, they just want the benefits. ECS is pretty exciting, but isn't the same level of abstraction accomplished with the GCE/Fleet/k8s? Even the Kubernetes or Fleet install process on EC2 lets you quickly forget about the actual boxes.

emiller829 11y ago

I'd love to hear how you're handling the ECS part of this. From the experimentation I've done, it's not just a matter of telling ECS to update the service unless you're running 2x the number of container instances you need -- it won't pull resources out from the existing service revision in order to spin up the new ones. Certainly can be worked around, but it's a little annoying.

joslin01 11y ago
There's two ways to go about this: 1.) Create a new task and update the service with the task. From the documentation (http://docs.aws.amazon.com/AmazonECS/latest/developerguide/update-service.html http://docs.aws.amazon.com/AmazonECS/latest/developerguide/u...): > If you have updated the Docker image of your application, you can create a new task definition with that image and deploy it to your service, one task at a time. The service scheduler creates a task with the new task definition (provided there is an available container instance to place it on), and after it reaches the RUNNING state, a task that is using the old task definition is drained and stopped. This process continues until all of the desired tasks in your service are using the new task definition. 2.) (way I'm doing it) Update the desired count to 0 and utilize wait_until (http://docs.aws.amazon.com/sdkforruby/api/Aws/ECS/Client.html#wait_until-instance_method http://docs.aws.amazon.com/sdkforruby/api/Aws/ECS/Client.htm...) services are stable. After this, update desired count to 1. You're going to get some downtime with #2. I'm not sure about #1 because I don't understand how something like an API service could be run without error'ing out that the port it's trying to bind to and forward is not available. So I'm not sure how it would ever reach the running state.. but I haven't experimented going about it that way. I can handle the small amount of downtime.
- emiller829 11y ago
  Thanks! Yeah, those were the ony options I was aware of, but the "provided there is an available container instance to place it on" issue is a problem due to port availability, and I was looking for zero-downtime deploy options.

fosk 11y ago

On a side note, for microservices that expose a RESTful API I would also take a look at Kong (https://github.com/Mashape/kong https://github.com/Mashape/kong) for delegating common functionality like security or rate-limiting.

simonebrunozzi 11y ago
Made (and open-sourced) by the smart guys at Mashape.com. Thanks for doing this.

JTenerife 11y ago

I don't understand how you configure your host-specific servicc dependencies, e.g. connection strings. Ansible (Puppet, Chef) does variable interpolation / templating for this purpose.

joslin01 11y ago
I have configuration files for each environment, so `dev.config`, `staging.config`, etc...
- michaelt 11y ago
  Do you deploy the same docker image to each environment, and somehow retrieve the configuration files on startup? Or do you rebuild your images as you promote from dev to staging to prod?
  - joslin01 11y ago
    Yes same docker image, but with different environment variables passed in for dev / staging / prod.
superuser2 11y ago
This is a hard problem, but outside the scope of Docker. There are numerous cluster orchestration/management systems springing up around the ecosystem. (We don't necessarily have on ready for prime-time yet, but a lot of very good work is taking place.)

jfoster 11y ago

What happens if there is a security flaw somewhere in your software stack? (kernel, web server, etc.) Do you end up having to update multiple containers with the same patch? Context: I've looked at Docker, but not used any containerization yet.

geggam 11y ago
Security ? How do you find out what libraries / versions are in a container ? ... you are pulling the curtain dont look back there :)
- NeutronBoy 11y ago
  > Security ? How do you find out what libraries / versions are in a container ? ... The same way you do in any application, container or not - you look up your documentation ;)
- jfoster 11y ago
  Is that really the state of things? It is what I concluded after looking into this a bit for Docker, but it seems incredible to me that so many companies are jumping into this idea of containerization without any good & available solutions for this problem. One potential solution that came to mind was that if there was a standard way of deploying an application into containers, and Google/Amazon/Microsoft provided auto-updating containers, the maintenance of a secure container would be in the hands of companies who (hopefully) have the resources necessary to keep the entire stack up-to-date.
  - digi_owl 11y ago
    They seem to be betting the farm on the containerization will contain (heh) whatever security issues that come up. This in the sense, i guess, that if they have a security flaw in their php that gives disk access, all the attacker will see is the content of the php container as the database will be on the next container over. Then again the containerization seems to have come alongside devops, where the mantra seems to be "update early, update often, to hell with stable branches".

jbhatab 11y ago

I am a more front end heavy web developer. I know my way around apis and what not, but what you are describing sounds like backend bliss. I don't even know where to start but is there someway you can share more about the magic of your process? Like an example of a github repo of search and how you plug that into an app? Either way, thanks.

joshcanhelp 11y ago
Would love to see this as well. I know my way around the CLI but not enough to even grok what Docker does. Any good overviews/getting starteds/tutorials out there?
- joslin01 11y ago
  This is a somewhat novel workflow from what I can gather, so no. If you worked backwards from my flow though, you could figure it out no problem. Maybe I'll do a blog post in the future about it. I just don't have the time these days.
pea 11y ago
Hey, we're actually hacking on solving this exact problem at StackHut.com, which is a platform for containerised services that you can call immediately through HTTP. I think your sentiment is super-common: building microservices / container-enabled services and getting all the good stuff that comes with them is still too much of a learning curve for most devs. We're just about to push live the ability to add new services: would love to get your thoughts on what you'd like to see: leo@stackhut.com

nogox 11y ago

CaaS is definitely the future of cloud. However I don't think ECS (IaaS+Container) is the way to go. In this approach, you still need to take care of the cluster capacity planning, scaling, failover, etc. And at most time, there always are some capacity sitting idle in your ECS cluster. The evolution of CaaS would be sth. like hyper.sh. Due to its hypervisor nature, the whole cluster+scheduling thing can be made transparent to developers, who simply "compose" the app spec and submit to run.

xorcist 11y ago

Rather than using Puppet etc.? Basically you've returned to before configuration management and that is somehow good? How do you change network configuration, load balancers or other external configuration in lock step with your application? Scripting by hand? And deployments are a just small part of configuration management. How do you find out which of your applications use libz 1.2.3 (to measure CVE applicability for example)? By looking in each and every one of them? How do you find out which application run a cron job? And which application connect to backend x without going through the load balancer? Which application has a client certificate about to expire? How do you guarantee two applications have the same shared secret, and change it in lock step? With configuration management all this is just a look up away. And best of all, most of this information is authoritative. When I come somewhere new, the use of these tools instead of a directory of miscellaneous scripts is like night and day. It literally turns a two day job into a ten minute one when the environments are complex. Docker is great for a lot of things. It enables to do daring things to your environment even if it's so complex you don't fully understand it all because you can just shuffle containers around. But I would never use it instead of configuration management.

jilted 11y ago
Re: '...Rather than do all this work with Puppet, Chef, or even Ansible, I can just declare what the systems ought to be and cluster them within code branches...' It sounds like the poster is doing something similar to a 'Gold System Image' though with Container technologies. Configuration Management is great, but I'm with the view that you should start with a combination of gold images/system builds and then stack on Cfg Mgmt on top of that. Theory is you will have a known base and identical builds etc, otherwise you would get subtle drift in your configurations, especially for things which are not explicitly under control of Config Mgmt (shared library versions, packages). Of course, this is not always feasible but if I were to start from scratch, I would probably try to do it this way.
- tjbiddle 11y ago
  It should be a combination, really. You want to have your "Gold system image" available and this is most certainly what you should be deploying from; however your configuration management - whether it's Chef, Puppet, whatever - should be able to take a base operating system and get it setup completely from scratch. This then solves the problem of ensure that you can completely reproduce your application from scratch; but also removes the possibly horrendous slow scale up time by using a "Gold image" that has already done this. My current process is: Jenkins runs puppet/chef, verifies that the application is healthy after the run and everything is happy, calls AWS and images the machine, then it iterates over all instances in my load balancer 33% at a time and replaces them with the new image resulting in zero-downtime. Of course another solution is to pull out those instances and apply the update, and then put them back in. And I'm sure someone else will have their $0.02 on their own process, which actually I'd love to hear :-)

pea 11y ago

Re services sharing, we are actually hacking on a database of useful containerized services which are hosted on Github and can be run anywhere through HTTP or JsonRPC. Product is super early, but would love your feedback on what you'd like to see: www.stackhut.com . Just about to push ability to submit your own. I think the key thing is that users shouldn't have to have a knowledge of Sysops or Devops to be able to take advantage of the benefits of containers. Most developer we've spoken to don't have time to learn a completely new paradigm, they just want the benefits. ECS is pretty exciting, but isn't the same level of abstraction accomplished with the GCE/Fleet/k8s? Even the Kubernetes or Fleet install process on EC2 lets you quickly forget about the actual boxes.