8 ms·

I left Heroku for OpsWorks and shaved 40% off response times

73 points by swrobel 13y ago

andrewvc 13y ago
Opsworks is nice, and it's a good packaged solution. A large number of smaller apps would benefit from a switch. It's also more fairly priced. That being said, opsworks is still based on chef, which is still cumbersome. For most people Ansible is far easier to deal with if they don't fit into the opsworks box. The overhead of starting from scratch with ansible can be made up for with the faster overall development time.
- swrobel 13y ago
  Good to know! I've been meaning to look into Ansible & SaltStack, and you gave me the kick in the butt I needed. EDIT: ohai Andrew! Thanks for the comment.
- wc- 13y ago
  Big thumbs up for ansible, I have had great results with it it in a number of different production environments, I think the docs are great, and the devs in the irc channel on freenode are very helpful. If you are about to embark on the journey of provisioning for the first time, or if you are looking for something a little more straightforward than chef/puppet (in my opinion at least...), I highly recommend checking out Ansible!
  - phowat 13y ago
    Another thumbs up. I'm currently in the process of migrating away from CFEngine to Ansible and it's been a breeze.
- sync 13y ago
  Seems like Opsworks does a lot more than just the provisioning, no? "AWS OpsWorks can scale your application using automatic load-based or time-based scaling and maintain the health of your application by detecting failed instances and replacing them. You have full control of deployments and automation of each component." How would you suggest using Ansible to cover that side of things?
  - wc- 13y ago
    Autoscaling groups with ansible-pull + cloud init to configure instances as they are spun up by AWS is what I've had the most success with. So your autoscaling group spins up a new instance using a prebaked AMI, and on first boot cloud-init will run ansible-pull locally to grab the latest configs and start up services as needed. Alternatively you can just use ansible to bake a new AMI as needed and skip the ansible-pull + cloud-init part altogether.
  - imperialWicket 13y ago
    This just introduces additional tools/products to the stack, but you can use Ansible with the Netflix OSS stack, such that Ansible provisions your AMIs and Asgard manages auto-scaling and deployment (with very fine-grained control).
- darklajid 13y ago
  Ansible: I was in the "Don't go there" camp a while ago - they documentation was horrible, the whole website a mess and lots of things weren't discoverable at all (people using ansible told me that they used google site search to find stuff in the manual). I returned a couple of days ago to look at the current state of things and I have to admit that I was impressed. Technically I really liked ansible for quite some while. Now it even comes with decent documentation. I'd back the suggestion to check it out.
  - ghayes 13y ago
    I was really excited by ansible, but I quickly found out that using `register: result` and `when: result|success` throughout so many playbooks meant I was back to the mess that is understanding the flow of a chef script that you didn't personally write. Maybe that's not a problem since I rarely found an open source ansible playbook for anything I needed and had to always write my own.
    - zenojevski 13y ago
      Hi, I've published some reusable roles for ansible. They are still quite basic, but I'd love for you to have a look at them. https://github.com/search?q=%40zenoamaro+ansible https://github.com/search?q=%40zenoamaro+ansible
      - ghayes 13y ago
        Always great to see people putting up open-source projects like this. I'll definitely give them a look, and at the least, get a grip on how other people found solutions with Ansible. Thanks man.
  - mercurial 13y ago
    The doc is indeed pretty good, and available offline (ansible-doc). I quite like that it uses a very declarative syntax, and it's relatively easy to get idempotent operations. And the fact that you don't need to install anything on the machine you administer is definitely a strong point. One downside is that I can see how tracking down role dependencies would be tricky if you had many different roles. I also haven't tried at scale.
- tgeek 13y ago
  "That being said, opsworks is still based on chef, which is still cumbersome." re:chef I don't think this is necessarily a fair assessment and true for all. There are a ton of community cookbooks out there, and a very large community now online to help people with Chef. I've been using Chef for years and like anything else once you get over the learning period, it becomes second nature. Any tool that you use will have some learning curve to it, and so the trade offs that make it cumbersome for some, make it easy for others. Also, you could technically Chef in Ansible in OpsWorks(if OpsWorks met other needs for you), but thats just crazy talk haha
iagooar 13y ago
While I can understand the advantages the author of the article mentions, I must say that one should think about making the move to OpsWorks, because you can't compare their prices without considering other costs / tradeoffs. A friend of mine and I have been working on a startup for the last 8 months. It has been tough as we have been working full time for a company and hacking like crazy once we got home, or on weekends. We chose to use Heroku because of the dead simple workflow that would allow us to spend time developing features instead of maintaining the infrastructure. This may sound like a cliché, but boy was it a good decision. The platform is rock solid, deployments are a bless and the add-on system is just one of the best time savers you can get. We know OpsWorks (or earlier Scalarium) from our full-time job, and we know how cool it is. But we also know that it requires a lot of time and additional effort to keep things running. This may pay off if you run a company which can afford it, but you should think about it twice if you have limited resources (and other, more important things to do!). Even now, after we have launched, we still see no reason why we should move from Heroku to OpsWorks (or another platform). We have made some optimizations and clever (to us) decisions to avoid running into scalability problems. Maybe we will think about migrating once we get a ton more clients and even more traffic. It is cheaper to just add some dynos than spending time on server administration. In the end, it's all about making the right decision, so don't hurry and if you think Heroku is too expensive for your business, maybe you are not making enough revenue from your customers. On the other hand, if you have the money and time, than go for it.
- swrobel 13y ago
  Agree with most of what you say, but adding dynos increases concurrency but doesn't do anything to reduce response times. Sometimes you just need a faster box.
  - thinkbohemian 13y ago
    Or a bigger (more RAM) box, I posted this to the Ruby Rouges mailing list on tail latencies: https://gist.github.com/schneems/8849116 https://gist.github.com/schneems/8849116 Concurrency is only half the battle with tail latencies. Of course dedicated CPU helps as well :)
incision 13y ago
One limitation with OpsWorks is that as far as I know, there's no option to use spot instances whereas it's not a problem with EC2 launch configurations for AutoScaling Groups. Also, ASGs and OpsWorks both have options for time and load-based scaling with different interfaces. It's easy enough to understand how this happened as OpsWorks came about via an acquisition, but consistency would certainly be nice. Presenting overlapping yet incompatible features under the blanket of AWS is confusing and potentially pretty frustrating.
Ryel 13y ago
How's Science-inc working out for you?
- swrobel 13y ago
  That's a whole other conversation ;)
  - Ryel 13y ago
    Hopefully good, lol. I've been watching them recently and thinking about reaching out with a couple prototypes.
juandazapata 13y ago
At WeHostels we moved from Heroku to Rackspace last year and we shaved ~30% off response times.
- thinkbohemian 13y ago
  Curious from a user experience perspective would you consider going back to try out the PX dyno service? Do you use chef or puppet or any other software to manage your Rackspace instances?
ericd 13y ago
Based on his mention of SSD-backed instances (c3.large) being only $100/mo, I looked at EC2 pricing again to see if it had become more competitive. Turns out it only has 2x16 gigs of storage and 1 physical core. Looking at the c3.8xlarge instances, they're specced pretty much exactly like the servers I just built for $9k/each, except they have half the ram (60 vs. 128) and half the storage (2x320 vs 4x320). And they cost the same to rent for 6 months as it does to buy one, including coloing. So, if you need more than a fraction of a physical server, EC2 still loses pretty hard on the pricing. It seems as though it has become competitive with Softlayer, though, at least. EDIT: Ah, but if you pay $11.6k upfront to reserve it for three years, it reduces the monthly bill from $1728 to ~$370, which is a much better deal than Softlayer offers. That's actually a pretty reasonable premium compared to racking them yourself, though you lose flexibility and the hardware is worse than what you'd get yourself. Well played, Amazon.
- Sssnake 13y ago
  >Looking at the c3.8xlarge instances, they're specced pretty much exactly like the servers I just built for $9k/each, except they have half the ram (60 vs. 128) and half the storage (2x320 vs 4x320) Your servers are also likely 2-5 times as fast in the CPU department too.
- ericd 13y ago
  >Your servers are also likely 2-5 times as fast in the CPU department too. FYI, you've been hellbanned, Sssnake. They use slightly better processors for the C3s than I use (2x8 core 2.8 vs. 2x8 core 2.6). I'd assume that if you get the largest instance, you probably have the whole machine to yourself and don't have issues with bad neighbors, so it's probably actually slightly faster than mine.
  - vinceguidry 13y ago
    Possible he's been hellbanned, but his latest post is the only dead one.
    - ericd 13y ago
joeblau 13y ago
Nice writeup. I recently switched from Nodejitsu over to Heroku for my open source project gitignore.io and I noticed there was a slight slowdown in response time. The app score index in New Relic went from around 92 to 100 though so I guess Heroku's servers on AWS were a bit more reliable. I also added in a CDN to increase the speed of serving the static assets. All in all, I can say that I'm happy with the transition, but I'm also running one Dyno which I'm far from maxing out. I feel like for a larger product, Heroku would be more problems down the line than it's worth.
- swrobel 13y ago
  Exactly how I felt, great at small scale, but increasingly painful to use as you reach scale. By the way, great tool, thanks for building it!
banachtarski 13y ago
What do Opswork, Chef, and the Heroku platform have to do with response time? Did the author investigate why the response time were that slow to begin with?
- swrobel 13y ago
  I wouldn't say they were "slow to begin with" but improved speed is always nice. I think the major contributing factor was access to newer, more powerful instances. Heroku Dynos run on very old EC2 instances.
DrJ 13y ago
FYI OpsWorks runs their Chef on Ruby 1.8, make sure your cookbooks passes on 1.8, and it doesn't depend on +1.9 :) This is nothing on AWS OpsWorks, as they greatly simplified interface into managing clusters (for deployments, organization). I found about this recently as I updated a community cookbook(python), as the Chef community is moving forward (with Chef 11+) that 1.8 has reach EOL and 1.9 is stable (2015) and 2.0 current, and 2.1 is new-hot-ness.

73 points by swrobel 13y ago

Opsworks is nice, and it's a good packaged solution. A large number of smaller apps would benefit from a switch. It's also more fairly priced. That being said, opsworks is still based on chef, which is still cumbersome. For most people Ansible is far easier to deal with if they don't fit into the opsworks box. The overhead of starting from scratch with ansible can be made up for with the faster overall development time.

swrobel 13y ago
Good to know! I've been meaning to look into Ansible & SaltStack, and you gave me the kick in the butt I needed. EDIT: ohai Andrew! Thanks for the comment.
wc- 13y ago
Big thumbs up for ansible, I have had great results with it it in a number of different production environments, I think the docs are great, and the devs in the irc channel on freenode are very helpful. If you are about to embark on the journey of provisioning for the first time, or if you are looking for something a little more straightforward than chef/puppet (in my opinion at least...), I highly recommend checking out Ansible!
- phowat 13y ago
  Another thumbs up. I'm currently in the process of migrating away from CFEngine to Ansible and it's been a breeze.
sync 13y ago
Seems like Opsworks does a lot more than just the provisioning, no? "AWS OpsWorks can scale your application using automatic load-based or time-based scaling and maintain the health of your application by detecting failed instances and replacing them. You have full control of deployments and automation of each component." How would you suggest using Ansible to cover that side of things?
- wc- 13y ago
  Autoscaling groups with ansible-pull + cloud init to configure instances as they are spun up by AWS is what I've had the most success with. So your autoscaling group spins up a new instance using a prebaked AMI, and on first boot cloud-init will run ansible-pull locally to grab the latest configs and start up services as needed. Alternatively you can just use ansible to bake a new AMI as needed and skip the ansible-pull + cloud-init part altogether.
- imperialWicket 13y ago
  This just introduces additional tools/products to the stack, but you can use Ansible with the Netflix OSS stack, such that Ansible provisions your AMIs and Asgard manages auto-scaling and deployment (with very fine-grained control).
darklajid 13y ago
Ansible: I was in the "Don't go there" camp a while ago - they documentation was horrible, the whole website a mess and lots of things weren't discoverable at all (people using ansible told me that they used google site search to find stuff in the manual). I returned a couple of days ago to look at the current state of things and I have to admit that I was impressed. Technically I really liked ansible for quite some while. Now it even comes with decent documentation. I'd back the suggestion to check it out.
- ghayes 13y ago
  I was really excited by ansible, but I quickly found out that using `register: result` and `when: result|success` throughout so many playbooks meant I was back to the mess that is understanding the flow of a chef script that you didn't personally write. Maybe that's not a problem since I rarely found an open source ansible playbook for anything I needed and had to always write my own.
  - zenojevski 13y ago
    Hi, I've published some reusable roles for ansible. They are still quite basic, but I'd love for you to have a look at them. https://github.com/search?q=%40zenoamaro+ansible https://github.com/search?q=%40zenoamaro+ansible
    - ghayes 13y ago
      Always great to see people putting up open-source projects like this. I'll definitely give them a look, and at the least, get a grip on how other people found solutions with Ansible. Thanks man.
- mercurial 13y ago
  The doc is indeed pretty good, and available offline (ansible-doc). I quite like that it uses a very declarative syntax, and it's relatively easy to get idempotent operations. And the fact that you don't need to install anything on the machine you administer is definitely a strong point. One downside is that I can see how tracking down role dependencies would be tricky if you had many different roles. I also haven't tried at scale.
tgeek 13y ago
"That being said, opsworks is still based on chef, which is still cumbersome." re:chef I don't think this is necessarily a fair assessment and true for all. There are a ton of community cookbooks out there, and a very large community now online to help people with Chef. I've been using Chef for years and like anything else once you get over the learning period, it becomes second nature. Any tool that you use will have some learning curve to it, and so the trade offs that make it cumbersome for some, make it easy for others. Also, you could technically Chef in Ansible in OpsWorks(if OpsWorks met other needs for you), but thats just crazy talk haha

iagooar 13y ago

While I can understand the advantages the author of the article mentions, I must say that one should think about making the move to OpsWorks, because you can't compare their prices without considering other costs / tradeoffs. A friend of mine and I have been working on a startup for the last 8 months. It has been tough as we have been working full time for a company and hacking like crazy once we got home, or on weekends. We chose to use Heroku because of the dead simple workflow that would allow us to spend time developing features instead of maintaining the infrastructure. This may sound like a cliché, but boy was it a good decision. The platform is rock solid, deployments are a bless and the add-on system is just one of the best time savers you can get. We know OpsWorks (or earlier Scalarium) from our full-time job, and we know how cool it is. But we also know that it requires a lot of time and additional effort to keep things running. This may pay off if you run a company which can afford it, but you should think about it twice if you have limited resources (and other, more important things to do!). Even now, after we have launched, we still see no reason why we should move from Heroku to OpsWorks (or another platform). We have made some optimizations and clever (to us) decisions to avoid running into scalability problems. Maybe we will think about migrating once we get a ton more clients and even more traffic. It is cheaper to just add some dynos than spending time on server administration. In the end, it's all about making the right decision, so don't hurry and if you think Heroku is too expensive for your business, maybe you are not making enough revenue from your customers. On the other hand, if you have the money and time, than go for it.

swrobel 13y ago
Agree with most of what you say, but adding dynos increases concurrency but doesn't do anything to reduce response times. Sometimes you just need a faster box.
- thinkbohemian 13y ago
  Or a bigger (more RAM) box, I posted this to the Ruby Rouges mailing list on tail latencies: https://gist.github.com/schneems/8849116 https://gist.github.com/schneems/8849116 Concurrency is only half the battle with tail latencies. Of course dedicated CPU helps as well :)

incision 13y ago

One limitation with OpsWorks is that as far as I know, there's no option to use spot instances whereas it's not a problem with EC2 launch configurations for AutoScaling Groups. Also, ASGs and OpsWorks both have options for time and load-based scaling with different interfaces. It's easy enough to understand how this happened as OpsWorks came about via an acquisition, but consistency would certainly be nice. Presenting overlapping yet incompatible features under the blanket of AWS is confusing and potentially pretty frustrating.

Ryel 13y ago

How's Science-inc working out for you?

swrobel 13y ago
That's a whole other conversation ;)
- Ryel 13y ago
  Hopefully good, lol. I've been watching them recently and thinking about reaching out with a couple prototypes.

juandazapata 13y ago

At WeHostels we moved from Heroku to Rackspace last year and we shaved ~30% off response times.

thinkbohemian 13y ago
Curious from a user experience perspective would you consider going back to try out the PX dyno service? Do you use chef or puppet or any other software to manage your Rackspace instances?

ericd 13y ago

Based on his mention of SSD-backed instances (c3.large) being only $100/mo, I looked at EC2 pricing again to see if it had become more competitive. Turns out it only has 2x16 gigs of storage and 1 physical core. Looking at the c3.8xlarge instances, they're specced pretty much exactly like the servers I just built for $9k/each, except they have half the ram (60 vs. 128) and half the storage (2x320 vs 4x320). And they cost the same to rent for 6 months as it does to buy one, including coloing. So, if you need more than a fraction of a physical server, EC2 still loses pretty hard on the pricing. It seems as though it has become competitive with Softlayer, though, at least. EDIT: Ah, but if you pay $11.6k upfront to reserve it for three years, it reduces the monthly bill from $1728 to ~$370, which is a much better deal than Softlayer offers. That's actually a pretty reasonable premium compared to racking them yourself, though you lose flexibility and the hardware is worse than what you'd get yourself. Well played, Amazon.

Sssnake 13y ago
>Looking at the c3.8xlarge instances, they're specced pretty much exactly like the servers I just built for $9k/each, except they have half the ram (60 vs. 128) and half the storage (2x320 vs 4x320) Your servers are also likely 2-5 times as fast in the CPU department too.
ericd 13y ago
>Your servers are also likely 2-5 times as fast in the CPU department too. FYI, you've been hellbanned, Sssnake. They use slightly better processors for the C3s than I use (2x8 core 2.8 vs. 2x8 core 2.6). I'd assume that if you get the largest instance, you probably have the whole machine to yourself and don't have issues with bad neighbors, so it's probably actually slightly faster than mine.
- vinceguidry 13y ago
  Possible he's been hellbanned, but his latest post is the only dead one.
  - ericd 13y ago

joeblau 13y ago

Nice writeup. I recently switched from Nodejitsu over to Heroku for my open source project gitignore.io and I noticed there was a slight slowdown in response time. The app score index in New Relic went from around 92 to 100 though so I guess Heroku's servers on AWS were a bit more reliable. I also added in a CDN to increase the speed of serving the static assets. All in all, I can say that I'm happy with the transition, but I'm also running one Dyno which I'm far from maxing out. I feel like for a larger product, Heroku would be more problems down the line than it's worth.

swrobel 13y ago
Exactly how I felt, great at small scale, but increasingly painful to use as you reach scale. By the way, great tool, thanks for building it!

banachtarski 13y ago

What do Opswork, Chef, and the Heroku platform have to do with response time? Did the author investigate why the response time were that slow to begin with?

swrobel 13y ago
I wouldn't say they were "slow to begin with" but improved speed is always nice. I think the major contributing factor was access to newer, more powerful instances. Heroku Dynos run on very old EC2 instances.

DrJ 13y ago

FYI OpsWorks runs their Chef on Ruby 1.8, make sure your cookbooks passes on 1.8, and it doesn't depend on +1.9 :) This is nothing on AWS OpsWorks, as they greatly simplified interface into managing clusters (for deployments, organization). I found about this recently as I updated a community cookbook(python), as the Chef community is moving forward (with Chef 11+) that 1.8 has reach EOL and 1.9 is stable (2015) and 2.0 current, and 2.1 is new-hot-ness.