6 ms·

The Switch from Heroku to Hardware

100 points by zeeg 14y ago

timr 14y ago
"On the limited hardware I run for getsentry.com, that is, two servers that actually service requests (one database, one app), we’ve serviced around 25 million requests since August 1st, doing anywhere from 500k to 2 million in a single day. That isn’t that much traffic, but what’s important is it services those requests very quickly, and is using very little of the resources that are dedicated to it. In the end, this means that Sentry’s revenue will grow much more quickly than it’s monthly bill will." There's nothing in this justification that doesn't also apply to Heroku. The cost structures just aren't significantly different at a two machine scale. However, as people keep pointing out, the roll-your-own-cloud approach requires that you build and maintain a bunch of infrastructure that Heroku has already built for you, or you forego redundancy and fault tolerance that Heroku has already built for you. The best lines of code are the ones you don't have to write.
- zeeg 14y ago
  I didn't note in this post (but in others), but before I switched, my Heroku bill was almost $700 (and I couldnt get it to perform well), the current bill is far less even with growth.
  - timr 14y ago
    Yeah, I read your original post. I've worked pretty extensively with Heroku and with large, custom, in-house infrastructure, and I don't share your experience. There's an I/O penalty for working on AWS, but it's on the order of tens of percent, not hundreds. I suspect that your original problems were related to working set size relative to cache (since Ronin => Fugu bumps cache by over 2GB, and you said that Fugu was working well). Heroku's largest database has a 68GB cache at an (admittedly expensive) $6,400 a month. But even so, $6,400 is a small expense for a growing web application. A mediocre developer costs more than that. Trading off server cost for developer cost is an asymptotically bad bet.
    - deleted 14y ago
      [deleted]
    - zeeg 14y ago
      Actually I think I/O was my primary bottleneck. Once I addressed that I started hitting CPU/memory constraints on Dynos. The database definitely wasnt unreasonable at $400, but for a bootstrapped project (especially something thats a side project for me), that was a big consideration. I probably would have toughed it out with Heroku if I could have gotten things to perform better. At one point I was running 20 dynos trying to get enough CPU for worker tasks to actually keep up, and I unfortunately couldn't solve the bottleneck to where the cost was reasonable. The application isn't typical (what Sentry does pushes some boundaries of SQL storage for starters), but it was costing too much of my time to struggle with optimizing something that really shouldn't have needed that much effort. I definitely like the redundancy provided, and the ability to add application servers with zero-thought is a huge plus, I just couldn't justify the cost of the service in addition to my frustrations/time of trying to scale it on Dynos.
      - timr 14y ago
        "Actually I think I/O was my primary bottleneck. Once I addressed that I started hitting CPU/memory constraints on Dynos....At one point I was running 20 dynos trying to get enough CPU for worker tasks to actually keep up" Something doesn't make sense. If you "addressed" your I/O problem, your CPUs were therefore all busy doing something much, much slower than a disk read/write, in software (which would have to be both obvious and unbelievably horrible). If that's true, something pathological was going on in your code. I'm going to assume that you would have noticed it -- swapping, for example. So let's go back to I/O: if your database was slow, you might observe something superficially similar to what you've described: throwing lots of extra CPUs at the problem would result in lots of blocked request threads, and appear that your dynos were all pegged. The exact symptoms would depend on your database connection code, and your monitoring tools. But in no case would throwing more dynos at a slow database make sense, so I'm going to assume that you didn't do that on purpose (right?) Given the above, I still can't meet you at the conclusion that abandoning Heroku was the magic bullet for your problems. There's not enough information, and it doesn't add up. My money is on one or more of the following: DB cache misses (i.e. not enough cache); a heavy DB write load; frequent, small writes to an indexed table; or pathological memory usage on your web nodes. And if it turns out that the cause is due to I/O, you've only bought yourself a temporary respite by moving off Heroku. Eventually, you'll get big enough that the problem will re-emerge, even though your homebuilt servers are 10% faster (or whatever). EDIT: Aha! Your comment in another thread actually explains your problem: you were swapping your web nodes by using more than 500MB RAM (http://news.ycombinator.com/item?id=4458657 http://news.ycombinator.com/item?id=4458657).
- deleted 14y ago
  [deleted]
count 14y ago
"DNS being slow (fuck it, use IPs)" So, yeah, ops isn't hard at all, if you don't fucking take the time to do it right.
- zeeg 14y ago
  It takes just as much time for DNS to propagate as it does for me to deploy a config change. It literally doesn't matter at this stage.
  - deleted 14y ago
    [deleted]
  - koko775 14y ago
    Run DNSMasq locally (as in, same datacenter as the computers that will be using it) and tell it to cache. It's dead-simple to set up. Then point your computers to resolve using it. You can even add to /etc/hosts and the computers using it as their DNS will resolve it. Depending on how much control you have, DNSMasq will also function as a DHCP server and TFTP server from which you can netboot other servers and do such nifty thing as automatic reinstalls. Useful if you have a separate, internal network and want to set internal IPs, too.
    - gizzlon 14y ago
      DNSMasq is nice, it's so easy to make up your own, local, dns names. Do you run several to avoid a single point of failure, or do you just fall back to the "real" dns? Even with a local DNS server, there has to be some overhead though.. OTOH, avoid premature optimization etc..
- donavanm 14y ago
  "DNS being slow (fuck it, use IPs)" GAAAHH!! <table flip>. The 1980s called, they said youre an f'n idiot. Seriously, this is not "operations". This is "phoning it in" and/or "not understanding the problem". Ps: to the op, not parent.
  - 14y ago
papercruncher 14y ago
Not directly relevant to the blog post, but Sentry is an amazing piece of software. We're running it on an m1.small AWS instance along with a bunch of other stuff and it is rock solid.
jacques_chester 14y ago
So basically nothing has gone wrong yet, therefore it's a slam dunk case for Doing It Yourself. I'm sorry, but no. You don't pay an ops guy or Heroku buckets of $$ for when things are going well, just as you don't pay $$ for software that only handles the happy case. You pay $$ for someone who has fixed shit that went horribly wrong and has the scars to prove it. "That deep purple welt on my lower ego ... is where I only had a backup script and never tested the backups. This interesting zigzag is where I learnt about things that can wrong with heartbeat protocols ..." Edit: though see below for a more nuanced discussion of reasons from OP.
- zeeg 14y ago
  You're saying before Heroku (or the cloud) everyone paid an operations guy to do anything? Nope. I don't need to suffer the same acts of terror operations people have gone through to be able to avoid, prevent, or recover from them. I'm paying myself to be the operations guy, as well as the engineer.
  - jacques_chester 14y ago
    It's about value delivered and opportunity cost paid. You're betting foregone engineering time against Heroku's value-as-delivered. I don't think that the hours you will inevitably spend fixing the ops infrastructure will turn out to be very profitable. For a large business, moving off Heroku to inhouse operations is probably justifiable, because they can capture sufficient value from a smart ops team to offset the potentially very high monthly bill Heroku would levy for their business. But for a smaller firm, Heroku so abundantly oversupplies value compared to the bill that you will simply never be able to capture that value from internal work at anything like the same price. It's like saying "I should stop buying food from the supermarket, it would be cheaper to grow my own vegetables!" In a naive dollar-cost analysis this is true. In an actual consideration of whether raising a vege patch, bearing the risks of starvation, doing it poorly because you're new at it, spending the hundreds of hours of labour it will require -- means that letting a professional farmer do it is much smarter. Gains from trade are gains. I have a pretty strong opinion on this given how much time I've already sunk on similar work: http://chester.id.au/2012/06/27/a-not-sobrief-aside-on-reigning-in-chaos/ http://chester.id.au/2012/06/27/a-not-sobrief-aside-on-reign... And for my secondary startup I am thinking I will just let Heroku handle it.
dangrossman 14y ago
A bit off topic but this is a good thread to ask in: Those of you running startups that don't colocate your own hardware, and don't run in a cloud, where do you rent servers from these days? Most of my stuff is at Softlayer, but their RAM pricing is killer ($25/mo/GB).
- zeeg 14y ago
  SL has become very expensive in the last couple of years (we host primarily there at Disqus). For Sentry I'm using Incero, but will likely be switching to Hivelocity (or something similar), as I currently don't have internal network and that's a big, annoying deal for me. Also, if you're loooking for deals or more information on hosts: webhostingtalk.com
  - benologist 14y ago
    Funny that we'd run into each other here again - Hivelocity is super cheap, I still have a massive, single server there and have used them for ~3 years now and actually all those servers I replaced were with them! They usually have really good specials, incredible value. But their live support is useless, the best thing they can do is log tickets on your behalf for the real support. Most of the time that doesn't matter. SoftLayer have much better support, geographically diverse locations and a wicked suite of offerings. They're also giving startups $1000/month credit for 12 months, I got almost instantly accepted... I was still asking questions and then the rep was all "I made you an account here's the details". https://www.softlayer.com/partners/catalyst https://www.softlayer.com/partners/catalyst
    - zeeg 14y ago
      I wasnt aware about that credit, that's actually really nice! I used to have a really good deal with SL about 2-3 years ago, and when I was looking around for Sentry they were my first stop. Unfortunately the prices had gone up a lot while I was gone, and for a reasonable DB server it was looking to be pretty pricey.
    - darkarmani 14y ago
JDavo 14y ago
My company is self hosting sentry for about 30 Python & Java in-house services and it was the easiest deployment of anything that's taken me more than apt-get install to deploy.
kuanghe 14y ago
test,hahahaha
kuanghe 14y ago
test,thanks
kingrolo 14y ago
I'm okay with the concept of trading cost/time for convenience, and why that might work for some folks, but even before we get to that argument, my experience with Heroku is that it just isn't reliable enough for client sites. Since reviewing it (over the last few months) we've had a couple of instances of real downtime (ie, greater than 30 mins), and a few spots of smaller amounts (a few mins each). We don't get that from our Linode+Puppet sites (taught myself Puppet as we went along, I'm a dev rather than an ops guy really).
- taligent 14y ago
  Well I guess you are lucky then because Linode has had massive downtimes in their London and Dallas data centres in the last few days and when I was a customer their Fremont data centre went down all the time. http://status.linode.com http://status.linode.com Heroku at least uses EC2 which will be far more reliable over time.
  - citricsquid 14y ago
    Not sure if the London one was really "massive", I have 12 Linodes in London all on different machines and I had one become unavailable during the downtime a few days ago and it was resolved very quickly.
  - slurgfest 14y ago
    Heroku goes down every time US-East goes down, so it isn't true that its reliability is the same as EC2's in general
davestheraves 14y ago
Sounds to me that the main issue was running out of RAM on workers. Would this not be soved by moving to another cloud provider (such as AWS) where you are not limited to the tiny RAM provided on Heroku? Is this not similar in strategy to choosing hardware and spinning up your own stack? EDIT: I getting at the rather sweeping statement against all cloud providers based on a specific Heroku problem
- mark_l_watson 14y ago
  I had the same thought: why not run the workers on a few large memory EC2s and use Heroku for everything else. Run on east coast region for free bandwidth between Heroku and EC2s.
- Xylakant 14y ago
  The moment you do this, you forego one of the biggest advantages of using heroku. As long as you're on heroku only, you don't need to take care of securing the underlying stack - that is firewall rules, OS-updates, general maintenance. The moment you spin up a single AWS instance besides it, it's your problem. Depending on your use-case it could be a better choice to just go all the way to dedicated hardware: The primary advantage AWS has over dedicated hardware is flexibility. You can spin up instances depending on your current need. If your load behavior is a flat, predictable curve you might just not need that - and then real hardware is cheaper in most cases.
josephlord 14y ago
This stuff is really useful even if you don't want to move off Heroku. Just knowing that you can is really reassuring in case they stop offering what you need. Simple how to guides are even better. It was the fact that Heroku offer an environment to run fairly standard Rails + Postgres that made me pick them over the more unique and harder to move from Google platform. Even though I was starting from scratch. It's always good to have an exit route.

100 points by zeeg 14y ago

timr 14y ago

"On the limited hardware I run for getsentry.com, that is, two servers that actually service requests (one database, one app), we’ve serviced around 25 million requests since August 1st, doing anywhere from 500k to 2 million in a single day. That isn’t that much traffic, but what’s important is it services those requests very quickly, and is using very little of the resources that are dedicated to it. In the end, this means that Sentry’s revenue will grow much more quickly than it’s monthly bill will." There's nothing in this justification that doesn't also apply to Heroku. The cost structures just aren't significantly different at a two machine scale. However, as people keep pointing out, the roll-your-own-cloud approach requires that you build and maintain a bunch of infrastructure that Heroku has already built for you, or you forego redundancy and fault tolerance that Heroku has already built for you. The best lines of code are the ones you don't have to write.

zeeg 14y ago
I didn't note in this post (but in others), but before I switched, my Heroku bill was almost $700 (and I couldnt get it to perform well), the current bill is far less even with growth.
- timr 14y ago
  Yeah, I read your original post. I've worked pretty extensively with Heroku and with large, custom, in-house infrastructure, and I don't share your experience. There's an I/O penalty for working on AWS, but it's on the order of tens of percent, not hundreds. I suspect that your original problems were related to working set size relative to cache (since Ronin => Fugu bumps cache by over 2GB, and you said that Fugu was working well). Heroku's largest database has a 68GB cache at an (admittedly expensive) $6,400 a month. But even so, $6,400 is a small expense for a growing web application. A mediocre developer costs more than that. Trading off server cost for developer cost is an asymptotically bad bet.
  - deleted 14y ago
    [deleted]
  - zeeg 14y ago
    Actually I think I/O was my primary bottleneck. Once I addressed that I started hitting CPU/memory constraints on Dynos. The database definitely wasnt unreasonable at $400, but for a bootstrapped project (especially something thats a side project for me), that was a big consideration. I probably would have toughed it out with Heroku if I could have gotten things to perform better. At one point I was running 20 dynos trying to get enough CPU for worker tasks to actually keep up, and I unfortunately couldn't solve the bottleneck to where the cost was reasonable. The application isn't typical (what Sentry does pushes some boundaries of SQL storage for starters), but it was costing too much of my time to struggle with optimizing something that really shouldn't have needed that much effort. I definitely like the redundancy provided, and the ability to add application servers with zero-thought is a huge plus, I just couldn't justify the cost of the service in addition to my frustrations/time of trying to scale it on Dynos.
    - timr 14y ago
      "Actually I think I/O was my primary bottleneck. Once I addressed that I started hitting CPU/memory constraints on Dynos....At one point I was running 20 dynos trying to get enough CPU for worker tasks to actually keep up" Something doesn't make sense. If you "addressed" your I/O problem, your CPUs were therefore all busy doing something much, much slower than a disk read/write, in software (which would have to be both obvious and unbelievably horrible). If that's true, something pathological was going on in your code. I'm going to assume that you would have noticed it -- swapping, for example. So let's go back to I/O: if your database was slow, you might observe something superficially similar to what you've described: throwing lots of extra CPUs at the problem would result in lots of blocked request threads, and appear that your dynos were all pegged. The exact symptoms would depend on your database connection code, and your monitoring tools. But in no case would throwing more dynos at a slow database make sense, so I'm going to assume that you didn't do that on purpose (right?) Given the above, I still can't meet you at the conclusion that abandoning Heroku was the magic bullet for your problems. There's not enough information, and it doesn't add up. My money is on one or more of the following: DB cache misses (i.e. not enough cache); a heavy DB write load; frequent, small writes to an indexed table; or pathological memory usage on your web nodes. And if it turns out that the cause is due to I/O, you've only bought yourself a temporary respite by moving off Heroku. Eventually, you'll get big enough that the problem will re-emerge, even though your homebuilt servers are 10% faster (or whatever). EDIT: Aha! Your comment in another thread actually explains your problem: you were swapping your web nodes by using more than 500MB RAM (http://news.ycombinator.com/item?id=4458657 http://news.ycombinator.com/item?id=4458657).
deleted 14y ago
[deleted]

count 14y ago

"DNS being slow (fuck it, use IPs)" So, yeah, ops isn't hard at all, if you don't fucking take the time to do it right.

zeeg 14y ago
It takes just as much time for DNS to propagate as it does for me to deploy a config change. It literally doesn't matter at this stage.
- deleted 14y ago
  [deleted]
- koko775 14y ago
  Run DNSMasq locally (as in, same datacenter as the computers that will be using it) and tell it to cache. It's dead-simple to set up. Then point your computers to resolve using it. You can even add to /etc/hosts and the computers using it as their DNS will resolve it. Depending on how much control you have, DNSMasq will also function as a DHCP server and TFTP server from which you can netboot other servers and do such nifty thing as automatic reinstalls. Useful if you have a separate, internal network and want to set internal IPs, too.
  - gizzlon 14y ago
    DNSMasq is nice, it's so easy to make up your own, local, dns names. Do you run several to avoid a single point of failure, or do you just fall back to the "real" dns? Even with a local DNS server, there has to be some overhead though.. OTOH, avoid premature optimization etc..
donavanm 14y ago
"DNS being slow (fuck it, use IPs)" GAAAHH!! <table flip>. The 1980s called, they said youre an f'n idiot. Seriously, this is not "operations". This is "phoning it in" and/or "not understanding the problem". Ps: to the op, not parent.
- 14y ago

papercruncher 14y ago

Not directly relevant to the blog post, but Sentry is an amazing piece of software. We're running it on an m1.small AWS instance along with a bunch of other stuff and it is rock solid.

jacques_chester 14y ago

So basically nothing has gone wrong yet, therefore it's a slam dunk case for Doing It Yourself. I'm sorry, but no. You don't pay an ops guy or Heroku buckets of $$ for when things are going well, just as you don't pay $$ for software that only handles the happy case. You pay $$ for someone who has fixed shit that went horribly wrong and has the scars to prove it. "That deep purple welt on my lower ego ... is where I only had a backup script and never tested the backups. This interesting zigzag is where I learnt about things that can wrong with heartbeat protocols ..." Edit: though see below for a more nuanced discussion of reasons from OP.

zeeg 14y ago
You're saying before Heroku (or the cloud) everyone paid an operations guy to do anything? Nope. I don't need to suffer the same acts of terror operations people have gone through to be able to avoid, prevent, or recover from them. I'm paying myself to be the operations guy, as well as the engineer.
- jacques_chester 14y ago
  It's about value delivered and opportunity cost paid. You're betting foregone engineering time against Heroku's value-as-delivered. I don't think that the hours you will inevitably spend fixing the ops infrastructure will turn out to be very profitable. For a large business, moving off Heroku to inhouse operations is probably justifiable, because they can capture sufficient value from a smart ops team to offset the potentially very high monthly bill Heroku would levy for their business. But for a smaller firm, Heroku so abundantly oversupplies value compared to the bill that you will simply never be able to capture that value from internal work at anything like the same price. It's like saying "I should stop buying food from the supermarket, it would be cheaper to grow my own vegetables!" In a naive dollar-cost analysis this is true. In an actual consideration of whether raising a vege patch, bearing the risks of starvation, doing it poorly because you're new at it, spending the hundreds of hours of labour it will require -- means that letting a professional farmer do it is much smarter. Gains from trade are gains. I have a pretty strong opinion on this given how much time I've already sunk on similar work: http://chester.id.au/2012/06/27/a-not-sobrief-aside-on-reigning-in-chaos/ http://chester.id.au/2012/06/27/a-not-sobrief-aside-on-reign... And for my secondary startup I am thinking I will just let Heroku handle it.

dangrossman 14y ago

A bit off topic but this is a good thread to ask in: Those of you running startups that don't colocate your own hardware, and don't run in a cloud, where do you rent servers from these days? Most of my stuff is at Softlayer, but their RAM pricing is killer ($25/mo/GB).

zeeg 14y ago
SL has become very expensive in the last couple of years (we host primarily there at Disqus). For Sentry I'm using Incero, but will likely be switching to Hivelocity (or something similar), as I currently don't have internal network and that's a big, annoying deal for me. Also, if you're loooking for deals or more information on hosts: webhostingtalk.com
- benologist 14y ago
  Funny that we'd run into each other here again - Hivelocity is super cheap, I still have a massive, single server there and have used them for ~3 years now and actually all those servers I replaced were with them! They usually have really good specials, incredible value. But their live support is useless, the best thing they can do is log tickets on your behalf for the real support. Most of the time that doesn't matter. SoftLayer have much better support, geographically diverse locations and a wicked suite of offerings. They're also giving startups $1000/month credit for 12 months, I got almost instantly accepted... I was still asking questions and then the rep was all "I made you an account here's the details". https://www.softlayer.com/partners/catalyst https://www.softlayer.com/partners/catalyst
  - zeeg 14y ago
    I wasnt aware about that credit, that's actually really nice! I used to have a really good deal with SL about 2-3 years ago, and when I was looking around for Sentry they were my first stop. Unfortunately the prices had gone up a lot while I was gone, and for a reasonable DB server it was looking to be pretty pricey.
  - darkarmani 14y ago

JDavo 14y ago

My company is self hosting sentry for about 30 Python & Java in-house services and it was the easiest deployment of anything that's taken me more than apt-get install to deploy.

kuanghe 14y ago

test,hahahaha

kuanghe 14y ago

test,thanks

kingrolo 14y ago

I'm okay with the concept of trading cost/time for convenience, and why that might work for some folks, but even before we get to that argument, my experience with Heroku is that it just isn't reliable enough for client sites. Since reviewing it (over the last few months) we've had a couple of instances of real downtime (ie, greater than 30 mins), and a few spots of smaller amounts (a few mins each). We don't get that from our Linode+Puppet sites (taught myself Puppet as we went along, I'm a dev rather than an ops guy really).

taligent 14y ago
Well I guess you are lucky then because Linode has had massive downtimes in their London and Dallas data centres in the last few days and when I was a customer their Fremont data centre went down all the time. http://status.linode.com http://status.linode.com Heroku at least uses EC2 which will be far more reliable over time.
- citricsquid 14y ago
  Not sure if the London one was really "massive", I have 12 Linodes in London all on different machines and I had one become unavailable during the downtime a few days ago and it was resolved very quickly.
- slurgfest 14y ago
  Heroku goes down every time US-East goes down, so it isn't true that its reliability is the same as EC2's in general

davestheraves 14y ago

Sounds to me that the main issue was running out of RAM on workers. Would this not be soved by moving to another cloud provider (such as AWS) where you are not limited to the tiny RAM provided on Heroku? Is this not similar in strategy to choosing hardware and spinning up your own stack? EDIT: I getting at the rather sweeping statement against all cloud providers based on a specific Heroku problem

mark_l_watson 14y ago
I had the same thought: why not run the workers on a few large memory EC2s and use Heroku for everything else. Run on east coast region for free bandwidth between Heroku and EC2s.
Xylakant 14y ago
The moment you do this, you forego one of the biggest advantages of using heroku. As long as you're on heroku only, you don't need to take care of securing the underlying stack - that is firewall rules, OS-updates, general maintenance. The moment you spin up a single AWS instance besides it, it's your problem. Depending on your use-case it could be a better choice to just go all the way to dedicated hardware: The primary advantage AWS has over dedicated hardware is flexibility. You can spin up instances depending on your current need. If your load behavior is a flat, predictable curve you might just not need that - and then real hardware is cheaper in most cases.

josephlord 14y ago

This stuff is really useful even if you don't want to move off Heroku. Just knowing that you can is really reassuring in case they stop offering what you need. Simple how to guides are even better. It was the fact that Heroku offer an environment to run fairly standard Rails + Postgres that made me pick them over the more unique and harder to move from Google platform. Even though I was starting from scratch. It's always good to have an exit route.