6 ms·

I love this post, and hope it keeps coming up. Some day, developers will learn that even though they could do it as well as Heroku, given enough time and effort

by pbiggar 14y ago

I love this post, and hope it keeps coming up. Some day, developers will learn that even though they could do it as well as Heroku, given enough time and effort, it's just a lot cheaper not to! This comes up all the time with my company. We make hosted Continous Integration (https://circleci.com https://circleci.com), and often hear "can't I just set up Jenkins?". And the answer is the same, "you could, but ...". Run it on EC2, where your tests fail because the IO is bad? What about when three people push at once and you want to get results ASAP? Are you going to manually compile Postgres 9.1? And again when you add a second box to you cluster? I could go on.

aaronblohowiak 14y ago
... except heroku doesn't do multi-region, which is where things start getting tricky. They make the easy immediate, but haven't really tackled the hard (yet.)
- MattRogish 14y ago
  We use heroku, and every now and again that terrifies us. I'm not sure which will come first: Heroku going multi-region, or us having an outage due to another AWS problem and switching providers. :fingers crossed:
  - batista 14y ago
    >or us having an outage due to another AWS problem and switching providers. Why, because the other providers (or a roll-your-own solution) will not have outages? Or because the customers cannot tolerate even the more or less half a day of outage per year AWS has? Even critical businesses like banking have work stoppages far more often than that for tons of reasons, in both the physical world and the digital/networked one.
heretohelp 14y ago
The persistence scaling story for Heroku seems pretty questionable to me. Once you've maxed out what they offer for MySQL and Postgres, what exactly are you supposed to do? Start using EC2 RDS? Heroku seems to be like a more useful Google App Engine, a good place to host a blog or experimental project if you're not into dev-ops. If you have a knack (at all) for dev-ops, you're not saving yourself anything. The downtime is pretty bleh too. The moment you start doing multi-provider to offset this, you'll end up doing all the dev-ops work you would've had to have done before. Except now, you have to do it all at once in a time of what is probably high stress. If you do the dev-ops/automation yourself from the start, you can start small/simple and grow that as you go, deploying your services to arbitrary hosting providers (EC2, Linode, dedicated boxes, whatever). This is why whenever anybody asks me my opinion of Heroku, I respond, "it's a great place to host that blog engine you wrote in Haskell/Clojure/{hipster_language_of_choice}".
- deleted 14y ago
  [deleted]
- j-kidd 14y ago
  > The persistence scaling story for Heroku seems pretty questionable to me. Agree. Those people who tell horror stories about Heroku/EC2, usually give solid numbers, i.e. this was how much I spent, this is how much I save by moving away, and our response time is now X% faster. On the other hand, we have article like this that shows pretty graph for a web app serving 16.2 requests per minute, and make bold claims that everything will scale.
  - heretohelp 14y ago
    >16.2 requests per minute Was it written in TCL? ;)
  - benologist 14y ago
    Here's my pretty graph for serving ~500,000 requests per minute on Heroku: https://api.playtomic.com/load.html https://api.playtomic.com/load.html Savings aren't exclusively because of Heroku, I also switched the underlying architecture during that migration from C# / ASP.NET to NodeJS which is exceptionally well suited to what I'm doing there. Previously: Dedicated servers, ~$1600 a month 8x dedicated servers at ~$200 each, each running 3 to 5 instances of the API depending on how many IP addresses they were provisioned with. Uploading was done via a simple hand-rolled script that'd just FTP everything to each server. Now: Heroku, current usage $400 - $500 a month With Heroku I don't have to worry about concurrent connections (typically 200 - 400 thousand people at once), I don't have to maintain all those servers and I don't have to fuck around with all the stupid things that can go wrong when you're operating at scale. It was a lot of work to get to this point and I made a lot of mistakes like having a heavy redis pub/sub outside of the EC2 network that cost me $350 in excess bandwidth providing inter-dyno communication, and I saturated database connections lots of times because in the old days those dedicated servers each had a local mongodb which could keep up with ordinary connection pooling, but it was totally worth it.
    - 14y ago
deleted 14y ago
[deleted]
Deinumite 14y ago
I tried out Circle CI... I have some feedback if you want. First the UI could use some work, but it was really awesome seeing it pull down a Flask app and knowing to activate venv and install pip. Also, not having an option to delete your account is a bit scary, seeing as I'm part of an organization on github and not comfortable with you having access to that code.
- pbiggar 14y ago
  Cool, thanks for trying! The UI could definitely use work, and we're working on it. There is an option to delete your account. One option is to contact us (I know, that's not great, but we delete same day). The other option is to rip out Circle's authentication via GitHub, if you're worried about us doing it properly.
jordanthoms 14y ago
Hey, BTW the join using github button at the bottom of the page doesn't work (links to undefined), the one at the top does. Probably want to fix that :)
- pbiggar 14y ago
  Cool, thanks!
calinet6 14y ago
Good to know Circle is running on Heroku. At least now we'll know why our CI isn't up when there's another lightning storm in North Carolina. Heroku does make a lot of sense: it's an abstraction. You're using a system designed to do a lot of the heavy lifting so you don't have to deal with those details, and for many startups and small operations, it's a great value proposition. But you have to admit that it is only a value proposition; and fundamentally a tradeoff. The truth is, especially from the perspective of a seasoned operations manager, all of the things above are just not that hard, nor are any of the complaints outlined in the OP article. The more layers you put between you and your infrastructure, the harder it will be to control your availability. When EBS's start failing and you have no idea why and Amazon only shows a green dot on their status page with a cryptic message like "minor availability issues experienced in certain availability zones; investigating." Now that is hard. Cloning a server and keeping a mirrored backup in another data center if you really need the availability? On hardware you have full control over? Not that hard. http://whoownsmyavailability.com/ http://whoownsmyavailability.com/

by pbiggar 14y ago

aaronblohowiak 14y ago

... except heroku doesn't do multi-region, which is where things start getting tricky. They make the easy immediate, but haven't really tackled the hard (yet.)

MattRogish 14y ago
We use heroku, and every now and again that terrifies us. I'm not sure which will come first: Heroku going multi-region, or us having an outage due to another AWS problem and switching providers. :fingers crossed:
- batista 14y ago
  >or us having an outage due to another AWS problem and switching providers. Why, because the other providers (or a roll-your-own solution) will not have outages? Or because the customers cannot tolerate even the more or less half a day of outage per year AWS has? Even critical businesses like banking have work stoppages far more often than that for tons of reasons, in both the physical world and the digital/networked one.

heretohelp 14y ago

The persistence scaling story for Heroku seems pretty questionable to me. Once you've maxed out what they offer for MySQL and Postgres, what exactly are you supposed to do? Start using EC2 RDS? Heroku seems to be like a more useful Google App Engine, a good place to host a blog or experimental project if you're not into dev-ops. If you have a knack (at all) for dev-ops, you're not saving yourself anything. The downtime is pretty bleh too. The moment you start doing multi-provider to offset this, you'll end up doing all the dev-ops work you would've had to have done before. Except now, you have to do it all at once in a time of what is probably high stress. If you do the dev-ops/automation yourself from the start, you can start small/simple and grow that as you go, deploying your services to arbitrary hosting providers (EC2, Linode, dedicated boxes, whatever). This is why whenever anybody asks me my opinion of Heroku, I respond, "it's a great place to host that blog engine you wrote in Haskell/Clojure/{hipster_language_of_choice}".

deleted 14y ago
[deleted]
j-kidd 14y ago
> The persistence scaling story for Heroku seems pretty questionable to me. Agree. Those people who tell horror stories about Heroku/EC2, usually give solid numbers, i.e. this was how much I spent, this is how much I save by moving away, and our response time is now X% faster. On the other hand, we have article like this that shows pretty graph for a web app serving 16.2 requests per minute, and make bold claims that everything will scale.
- heretohelp 14y ago
  >16.2 requests per minute Was it written in TCL? ;)
- benologist 14y ago
  Here's my pretty graph for serving ~500,000 requests per minute on Heroku: https://api.playtomic.com/load.html https://api.playtomic.com/load.html Savings aren't exclusively because of Heroku, I also switched the underlying architecture during that migration from C# / ASP.NET to NodeJS which is exceptionally well suited to what I'm doing there. Previously: Dedicated servers, ~$1600 a month 8x dedicated servers at ~$200 each, each running 3 to 5 instances of the API depending on how many IP addresses they were provisioned with. Uploading was done via a simple hand-rolled script that'd just FTP everything to each server. Now: Heroku, current usage $400 - $500 a month With Heroku I don't have to worry about concurrent connections (typically 200 - 400 thousand people at once), I don't have to maintain all those servers and I don't have to fuck around with all the stupid things that can go wrong when you're operating at scale. It was a lot of work to get to this point and I made a lot of mistakes like having a heavy redis pub/sub outside of the EC2 network that cost me $350 in excess bandwidth providing inter-dyno communication, and I saturated database connections lots of times because in the old days those dedicated servers each had a local mongodb which could keep up with ordinary connection pooling, but it was totally worth it.
  - 14y ago

deleted 14y ago

[deleted]

Deinumite 14y ago

I tried out Circle CI... I have some feedback if you want. First the UI could use some work, but it was really awesome seeing it pull down a Flask app and knowing to activate venv and install pip. Also, not having an option to delete your account is a bit scary, seeing as I'm part of an organization on github and not comfortable with you having access to that code.

pbiggar 14y ago
Cool, thanks for trying! The UI could definitely use work, and we're working on it. There is an option to delete your account. One option is to contact us (I know, that's not great, but we delete same day). The other option is to rip out Circle's authentication via GitHub, if you're worried about us doing it properly.

jordanthoms 14y ago

Hey, BTW the join using github button at the bottom of the page doesn't work (links to undefined), the one at the top does. Probably want to fix that :)

pbiggar 14y ago
Cool, thanks!

calinet6 14y ago

Good to know Circle is running on Heroku. At least now we'll know why our CI isn't up when there's another lightning storm in North Carolina. Heroku does make a lot of sense: it's an abstraction. You're using a system designed to do a lot of the heavy lifting so you don't have to deal with those details, and for many startups and small operations, it's a great value proposition. But you have to admit that it is only a value proposition; and fundamentally a tradeoff. The truth is, especially from the perspective of a seasoned operations manager, all of the things above are just not that hard, nor are any of the complaints outlined in the OP article. The more layers you put between you and your infrastructure, the harder it will be to control your availability. When EBS's start failing and you have no idea why and Amazon only shows a green dot on their status page with a cryptic message like "minor availability issues experienced in certain availability zones; investigating." Now that is hard. Cloning a server and keeping a mirrored backup in another data center if you really need the availability? On hardware you have full control over? Not that hard. http://whoownsmyavailability.com/ http://whoownsmyavailability.com/