6 ms·

How We Moved from Heroku to Containers with No Docker Experience

67 points by sergiotapia 11y ago

jacques_chester 11y ago
For those who want a less all-or-nothing migration path from Heroku, might I suggest looking into Cloud Foundry? Several of the buildpacks (most notably Ruby) are derived from Heroku's upstream repos. If you decide later that you want to build your own containers, Cloud Foundry has you covered there too, with a pathway for Docker images to be placed, launched and monitored on an equal basis with buildpack droplets. I'm biased, of course. I work for Pivotal, which donates the majority of engineering time to Cloud Foundry.
samcheng 11y ago
What about people who don't want Heroku (or Heroku-style) dynos as application servers and workers, because you pay excessively for RAM and CPU and wrestle with configuration constraints, but want to keep the rest of the PaaS ecosystem? I'm comfortable maintaining a fleet of stateless application servers, but don't want to maintain my own database servers, or queue servers, or SSL endpoint, or logging aggregator, etc. etc. Is this a feasible setup?
- anewhnaccount 11y ago
  Isn't this exactly what AWS offers?
  - jacques_chester 11y ago
    Yes and no. Amazon is mostly IaaS in its outlook -- the closest they come to a "true" PaaS would be Lambda. What you get from Amazon is a box full of building blocks, which you have to assemble yourself. For many people that is compelling. For others it is daunting. PaaSes are there for the second group -- an architecture-in-a-box.
    - jmduke 11y ago
      I'd argue that Elastic Beanstalk, too, is Amazon's explicit attempt to package and present AWS as a PaaS.
    - nostrademons 11y ago
      Elastic Beanstalk is the Amazon PaaS offering. Lambda is much more like a BaaS provider - it abstracts away the concept of a server entirely, and instead just gives you an API to execute code on some cloud service.
      - jacques_chester 11y ago
        You're right. I overlooked Beanstalk.
- jacques_chester 11y ago
  > Is this a feasible setup? Yes, there are open source PaaSes you can install privately or on an IaaS provider. I like Cloud Foundry, some people like OpenShift. There are others emerging as well.
  - TheLogothete 11y ago
    It's not a matter of if you could, but of if you should. In OP's case, clearly not.
    - samcheng 11y ago
      Right. In another time, we had racks of 1U servers, a NAS cluster with fast backplane, redundant switching fabric, a tape library, and fiber to the meetup room. This time around, if we were worrying about anything like that, it would be a distraction from the product!
- mkulke 11y ago
  to a degree, i guess. On AWS there are a lot of services, which would qualify as PaaS offerings. Even on Rackspace, you get Databases (and HA Groups), Load-Balancers (w/ SSL termination), Queues (albeit the obscure, OpenStack one) and ElasticSearch for Logging (the latter is pretty hefty priced IMO, though). So we're able to run quite a lot of stateless 12factor-like services on Rackspace (using kubernetes), RabbitMQ being the exception we have to manage ourselves (which sucks).
- cmelbye 11y ago
  It's certainly feasible. Your containers run on a public cloud, just as Heroku does, which enables access to things like logging-as-a-service. But you'll soon find that such services are just as horrifically marked up as Heroku, because they too are middlemen who need to make money. (I'm referring to services like Loggly or Logentries, to continue with the logging example.) We've been really enjoying AWS's hosted ElasticSearch, PostgreSQL, Elastic Load Balancer, etc. Most of the benefits of a hosted service at the cost of an EC2 instance.
  - samcheng 11y ago
    Definitely agree that the other services are marked-up too! The issue is one of leveraging developer time at a dynamic startup. At some scale, it becomes cost-effective to manage your own application cluster. At some (usually larger?) scale, it becomes a good idea to administrate your own database cluster. We're definitely not big enough to consider managing our own RabbitMQ or log aggregator or other ancillary services from a developer-time opportunity cost/benefit perspective, though!
  - hrez 11y ago
    FWIW Cloudwatch has logs service.
- jtmarmon 11y ago
  Database servers - AWS RDS Queue servers - AWS SQS SSL termination - AWS ELB Logging aggregator - depends on how you're deploying. We're deploying our containers onto elastic container service and adding one of these: https://github.com/gliderlabs/logspout https://github.com/gliderlabs/logspout to each instance. Elastic beanstalk will do the log aggregation for you
- nzoschke 11y ago
  Take a look at Convox, convox.com. It's an open source "PaaS" that runs in your own AWS account. It's a thin configuration layer for AWS. So you get easy access to ECS for running your app servers at resource cost, then simple tooling for adding an RDS database, adding a Heroku Postgres URL to you app env, or adding to Papertrail. Full disclosure: I work on Convox full time.
- ownagefool 11y ago
  Google or Amazon will both give you PaaS systems where you can run pretty much anything. I have more experience with kubernetes than AWS Container Engine. Openshift and Cloudfoundry should give you alternative PaaS type systems which in arguably are less versatile but more complete.
- kordless 11y ago
  Read up on MSaaS or CaaSi: http://venturebeat.com/2015/05/09/goodbye-saas-hello-containers-as-a-service/ http://venturebeat.com/2015/05/09/goodbye-saas-hello-contain.... The promise of this stuff is that you run the servers and control what gets deployed, but the vendors manage the software for you, like they do with SaaS. Is that what you are asking for?
- ctide 11y ago
  At a previous company, we initially set everything up at Heroku. As things got cost prohibitive, we moved them one by one to AWS. Since Heroku is setup on AWS, it was easy enough to connect to our Heroku postgres instance from AWS web servers and it doesn't add any extra latency vs. running direct on Heroku. Over time, we slowly migrated everything off to our own infrastructure on AWS, but we were able to leverage Heroku to avoid all of that development effort until it was cost effective to do so. You'll need to deal with your own SSL endpoint if you run your web servers on AWS, but it's pretty trivial to do so with ELB.
  - holografix 11y ago
    Out of curiosity: does Heroku give you a better deal as you grow? I know for a fact Salesforce will discount their licenses more and more and you buy bigger quantities and sign up for longer contracts. It'd be surprising if Heroku doesn't do this at all...!
    - joshmn 11y ago
      From my experience, no.
    - toyg 11y ago
      They have the opposite model: give it away to small players to hook them up, then rake it in from heavy users.
  - samcheng 11y ago
    That's exactly the roadmap I had in mind. Glad to hear someone had success with this route!
wrs 11y ago
If you're only containerizing a Ruby application and nginx, I don't really see how containers are much better than old school Capistrano-style rsync deployment. (How often does your nginx version change, after all?) At the cost of an awful lot of additional tooling and somewhat unstable infrastructure. For our simple Ruby workers, we just use Puppet to set up app server VMs with Ruby and nginx. Deployment consists of updating the Ruby code directory. It's pretty much cloud provider independent, as long as your provider can make a Linux VM, and it works fine, just like it did in 2010. :)
- MaxGabriel 11y ago
  There's also the version of Ruby to consider. Switching to a tool like rbenv can require changing existing scripts (and iirc it felt like it was always more for development), and Ubuntu has removed tools like ruby-switch in Ubuntu 14.
- tra3 11y ago
  I find this attitude interesting. By the same reasoning why are .debs better than building from source, and the ultimate - horse drawn buggies vs automobiles. I've seen containerized applications described as "statically linked apps", it's one blob with dependencies included. I'm moving in that direction (vs cap-style deployments) because it guarantees that all machines are identical (from QA to staging, to production). Spinning up additional machines becomes trivial.
  - wrs 11y ago
    Containers add their own considerable complexity to the stack. If the dependencies are complicated enough, or you need fast scaling/reconfiguration on the fly, then containers can certainly be a net positive. But if you're adding Docker, Compose, DCHQ, and whatever else to your stack just so you can put Nginx and some Ruby code in a container, I'm questioning whether that ends up being a net benefit over Puppet/Chef + rsync.
    - tra3 11y ago
      I hear you. I like the idea of "deploying" a pre-built image. Capistrano "just-in-time" deployments (particularly deployment time "bundle") seem a little non-deterministic. Docker is certainly non-trivial though, but I think (I hope), it's a net win. Time will show. In my mind, it certainly parallels the binary package (debian) vs build-on-demand package (arch? OS X brew) approach. Both seem to work, but I certainly prefer the debs.
siliconc0w 11y ago
This is a submarine for DCHQ which looks to be some proprietary docker orchestration service that doesn't list prices and doesn't have an open source version. Pro tip - customers don't want to be forced to talk to your sales team before they can try your product.
- chris_st 11y ago
  It does look like an advertisement. I guess it's changed since you last looked -- DCHQ does have a pricing link on their front page now.
dantiberian 11y ago
I'd be really curious to know how many servers they're running. You can get a lot of iron for $3200/month. Also, if I read that correctly it looked like they maintained two sets of infrastructure and one was always spare so they could do green/blue deployment. That sounds like a great opportunity to use on demand instances so you only pay for what you use.
blantonl 11y ago
The cloud environment is becoming a "wild wild west" just like the app stores were a few years ago. It all has to do with how much money is being made in those markets. Maybe this is the result of AWS finally breaking out their revenue numbers. There is so much marketing and plugging of alternatives to [INSERT CLOUD PROVIDER/LAYER HERE] that there really isn't any substance to all this other than reading like an advertisement.

67 points by sergiotapia 11y ago

For those who want a less all-or-nothing migration path from Heroku, might I suggest looking into Cloud Foundry? Several of the buildpacks (most notably Ruby) are derived from Heroku's upstream repos. If you decide later that you want to build your own containers, Cloud Foundry has you covered there too, with a pathway for Docker images to be placed, launched and monitored on an equal basis with buildpack droplets. I'm biased, of course. I work for Pivotal, which donates the majority of engineering time to Cloud Foundry.

samcheng 11y ago

What about people who don't want Heroku (or Heroku-style) dynos as application servers and workers, because you pay excessively for RAM and CPU and wrestle with configuration constraints, but want to keep the rest of the PaaS ecosystem? I'm comfortable maintaining a fleet of stateless application servers, but don't want to maintain my own database servers, or queue servers, or SSL endpoint, or logging aggregator, etc. etc. Is this a feasible setup?

anewhnaccount 11y ago
Isn't this exactly what AWS offers?
- jacques_chester 11y ago
  Yes and no. Amazon is mostly IaaS in its outlook -- the closest they come to a "true" PaaS would be Lambda. What you get from Amazon is a box full of building blocks, which you have to assemble yourself. For many people that is compelling. For others it is daunting. PaaSes are there for the second group -- an architecture-in-a-box.
  - jmduke 11y ago
    I'd argue that Elastic Beanstalk, too, is Amazon's explicit attempt to package and present AWS as a PaaS.
  - nostrademons 11y ago
    Elastic Beanstalk is the Amazon PaaS offering. Lambda is much more like a BaaS provider - it abstracts away the concept of a server entirely, and instead just gives you an API to execute code on some cloud service.
    - jacques_chester 11y ago
      You're right. I overlooked Beanstalk.
jacques_chester 11y ago
> Is this a feasible setup? Yes, there are open source PaaSes you can install privately or on an IaaS provider. I like Cloud Foundry, some people like OpenShift. There are others emerging as well.
- TheLogothete 11y ago
  It's not a matter of if you could, but of if you should. In OP's case, clearly not.
  - samcheng 11y ago
    Right. In another time, we had racks of 1U servers, a NAS cluster with fast backplane, redundant switching fabric, a tape library, and fiber to the meetup room. This time around, if we were worrying about anything like that, it would be a distraction from the product!
mkulke 11y ago
to a degree, i guess. On AWS there are a lot of services, which would qualify as PaaS offerings. Even on Rackspace, you get Databases (and HA Groups), Load-Balancers (w/ SSL termination), Queues (albeit the obscure, OpenStack one) and ElasticSearch for Logging (the latter is pretty hefty priced IMO, though). So we're able to run quite a lot of stateless 12factor-like services on Rackspace (using kubernetes), RabbitMQ being the exception we have to manage ourselves (which sucks).
cmelbye 11y ago
It's certainly feasible. Your containers run on a public cloud, just as Heroku does, which enables access to things like logging-as-a-service. But you'll soon find that such services are just as horrifically marked up as Heroku, because they too are middlemen who need to make money. (I'm referring to services like Loggly or Logentries, to continue with the logging example.) We've been really enjoying AWS's hosted ElasticSearch, PostgreSQL, Elastic Load Balancer, etc. Most of the benefits of a hosted service at the cost of an EC2 instance.
- samcheng 11y ago
  Definitely agree that the other services are marked-up too! The issue is one of leveraging developer time at a dynamic startup. At some scale, it becomes cost-effective to manage your own application cluster. At some (usually larger?) scale, it becomes a good idea to administrate your own database cluster. We're definitely not big enough to consider managing our own RabbitMQ or log aggregator or other ancillary services from a developer-time opportunity cost/benefit perspective, though!
- hrez 11y ago
  FWIW Cloudwatch has logs service.
jtmarmon 11y ago
Database servers - AWS RDS Queue servers - AWS SQS SSL termination - AWS ELB Logging aggregator - depends on how you're deploying. We're deploying our containers onto elastic container service and adding one of these: https://github.com/gliderlabs/logspout https://github.com/gliderlabs/logspout to each instance. Elastic beanstalk will do the log aggregation for you
nzoschke 11y ago
Take a look at Convox, convox.com. It's an open source "PaaS" that runs in your own AWS account. It's a thin configuration layer for AWS. So you get easy access to ECS for running your app servers at resource cost, then simple tooling for adding an RDS database, adding a Heroku Postgres URL to you app env, or adding to Papertrail. Full disclosure: I work on Convox full time.
ownagefool 11y ago
Google or Amazon will both give you PaaS systems where you can run pretty much anything. I have more experience with kubernetes than AWS Container Engine. Openshift and Cloudfoundry should give you alternative PaaS type systems which in arguably are less versatile but more complete.
kordless 11y ago
Read up on MSaaS or CaaSi: http://venturebeat.com/2015/05/09/goodbye-saas-hello-containers-as-a-service/ http://venturebeat.com/2015/05/09/goodbye-saas-hello-contain.... The promise of this stuff is that you run the servers and control what gets deployed, but the vendors manage the software for you, like they do with SaaS. Is that what you are asking for?
ctide 11y ago
At a previous company, we initially set everything up at Heroku. As things got cost prohibitive, we moved them one by one to AWS. Since Heroku is setup on AWS, it was easy enough to connect to our Heroku postgres instance from AWS web servers and it doesn't add any extra latency vs. running direct on Heroku. Over time, we slowly migrated everything off to our own infrastructure on AWS, but we were able to leverage Heroku to avoid all of that development effort until it was cost effective to do so. You'll need to deal with your own SSL endpoint if you run your web servers on AWS, but it's pretty trivial to do so with ELB.
- holografix 11y ago
  Out of curiosity: does Heroku give you a better deal as you grow? I know for a fact Salesforce will discount their licenses more and more and you buy bigger quantities and sign up for longer contracts. It'd be surprising if Heroku doesn't do this at all...!
  - joshmn 11y ago
    From my experience, no.
  - toyg 11y ago
    They have the opposite model: give it away to small players to hook them up, then rake it in from heavy users.
- samcheng 11y ago
  That's exactly the roadmap I had in mind. Glad to hear someone had success with this route!

wrs 11y ago

If you're only containerizing a Ruby application and nginx, I don't really see how containers are much better than old school Capistrano-style rsync deployment. (How often does your nginx version change, after all?) At the cost of an awful lot of additional tooling and somewhat unstable infrastructure. For our simple Ruby workers, we just use Puppet to set up app server VMs with Ruby and nginx. Deployment consists of updating the Ruby code directory. It's pretty much cloud provider independent, as long as your provider can make a Linux VM, and it works fine, just like it did in 2010. :)

MaxGabriel 11y ago
There's also the version of Ruby to consider. Switching to a tool like rbenv can require changing existing scripts (and iirc it felt like it was always more for development), and Ubuntu has removed tools like ruby-switch in Ubuntu 14.
tra3 11y ago
I find this attitude interesting. By the same reasoning why are .debs better than building from source, and the ultimate - horse drawn buggies vs automobiles. I've seen containerized applications described as "statically linked apps", it's one blob with dependencies included. I'm moving in that direction (vs cap-style deployments) because it guarantees that all machines are identical (from QA to staging, to production). Spinning up additional machines becomes trivial.
- wrs 11y ago
  Containers add their own considerable complexity to the stack. If the dependencies are complicated enough, or you need fast scaling/reconfiguration on the fly, then containers can certainly be a net positive. But if you're adding Docker, Compose, DCHQ, and whatever else to your stack just so you can put Nginx and some Ruby code in a container, I'm questioning whether that ends up being a net benefit over Puppet/Chef + rsync.
  - tra3 11y ago
    I hear you. I like the idea of "deploying" a pre-built image. Capistrano "just-in-time" deployments (particularly deployment time "bundle") seem a little non-deterministic. Docker is certainly non-trivial though, but I think (I hope), it's a net win. Time will show. In my mind, it certainly parallels the binary package (debian) vs build-on-demand package (arch? OS X brew) approach. Both seem to work, but I certainly prefer the debs.

siliconc0w 11y ago

This is a submarine for DCHQ which looks to be some proprietary docker orchestration service that doesn't list prices and doesn't have an open source version. Pro tip - customers don't want to be forced to talk to your sales team before they can try your product.

chris_st 11y ago
It does look like an advertisement. I guess it's changed since you last looked -- DCHQ does have a pricing link on their front page now.

dantiberian 11y ago

I'd be really curious to know how many servers they're running. You can get a lot of iron for $3200/month. Also, if I read that correctly it looked like they maintained two sets of infrastructure and one was always spare so they could do green/blue deployment. That sounds like a great opportunity to use on demand instances so you only pay for what you use.

blantonl 11y ago

The cloud environment is becoming a "wild wild west" just like the app stores were a few years ago. It all has to do with how much money is being made in those markets. Maybe this is the result of AWS finally breaking out their revenue numbers. There is so much marketing and plugging of alternatives to [INSERT CLOUD PROVIDER/LAYER HERE] that there really isn't any substance to all this other than reading like an advertisement.