5 ms·

My startup is heavily invested in and tied to Heroku. We don't have the resources right now to get off of the platform quickly, but plan to as soon as possible.

by aneth4 14y ago

My startup is heavily invested in and tied to Heroku. We don't have the resources right now to get off of the platform quickly, but plan to as soon as possible. Like Rapleaf, we have spent an enormous amount of effort optimizing and searching for the causes of latency, and overpay for dynos to reduce but not eliminate the chance of latency issues. Heroku has always told us it's our fault. While I have no interest in recovering money through a class action lawsuit - which is just grossly unproductive - my confidence in heroku has been shattered and I am embarrassed to have chosen and advocated for them for so long. I look forward to getting off their platform for good, and there is no way I could recommend it to others for similar applications. Heroku's most recent statement does nothing to resolve the issue for most of their customers, and does not reduce what is clearly a gross overpricing and misrepresentation of what they provide. It's quite clear that Heroku is not just a bad, but a horrible choice for rails applications that are not carefully designed for concurrency and don't go against Heroku recommendations and use a concurrent application server.

deleted 14y ago
[deleted]
- JangoSteve 14y ago
  EDIT: I don't necessarily support a class action lawsuit against Heroku, but in the interest of conversation... Scalability is not a magic knob you can twist as far as you want… Isn't that the problem? That's Heroku's selling point is that scaling is basically a magic knob. Here are some of the things on their homepage right now: deploy and scale powerful apps Focus 100% on your code, and never think about servers, instances, or VMs again. Run any web or background process with any web framework or worker type. Scale distributed processes effortlessly with a single command. Easily scale to millions of users. Your comment seems to be defending Heroku, by saying Heroku is lying.
  - deleted 14y ago
    [deleted]
    - JangoSteve 14y ago
      If you build your app in the correct way That's interesting terminology to use, "the correct way". Heroku's own devcenter article on deploying Rails apps to Heroku [1] still recommends the thin webserver, which isn't concurrent, and therefore causes all the issues being discussed for any rails app. In other words, if you're using thin, it doesn't matter how you architected your app, it's not going to run in concurrent processes on one dyno. So I'll assume by "the correct way", you're referring to both building your app for concurrency and using unicorn as your webserver. Doing only one of those things will not work, both are necessary. But their article only mentions unicorn only as an alternative, and up until recently their documentation actually said "We recommend thin" [2] over unicorn and that unicorn was possible but wasn't officially supported (EDIT: actually the most recent version of the article in wayback machine doesn't even mention unicorn on the page). I wouldn't call using thin "naive", given that it's what Heroku's docs have recommended for a long time. I don't think it's fair to call customers of Heroku naive for not knowing that when Heroku says "easily scale" and "use thin", they are talking about two mutually exclusive possibilites. [1] https://devcenter.heroku.com/articles/rails3 https://devcenter.heroku.com/articles/rails3 [2] http://web.archive.org/web/20120801214821/https://devcenter.heroku.com/articles/rails3 http://web.archive.org/web/20120801214821/https://devcenter.... EDIT: Added quotes with references.
- aneth4 14y ago
  "Scalable way" as defined by the platform Heroku was built and marketed for is single threaded. I won't argue that the lack of concurrency support in rails and ruby is not also a stupid horrible unacceptable choice - and I never understood why the rails community defended it. Regardless, Heroku was originally built for and continues to market itself as a platform that can scale Rails by using intelligent routing, but it's quite clear that they can not. Maybe you understand the above, though it seems you are just trolling rails for being single-threaded. This is not the place.
- gojomo 14y ago
  As bad as the latency-creating bottlenecks with random-routing and single-threaded apps can get, it is always the case that "twisting the magic knob" of adding more dynos would help. It just wouldn't help as much as people expected it to. Further, the amount of paid-for capacity left idle (while other dynos were backed-up) offends many sensibilities about how a system "ought to" work. But at least at the margin, the "magic knob" still worked a little. I think that's why the problem festered: there was always a default recommendation -- "more dynos!" -- that helped, which also just happened to fit the mental model Heroku promotes (while also boosting its revenue). In one of the older support logs RapGenius posted, it was evident that while RapGenius was suspicious about so many dynos being necessary, they let the ticket close once that expensive-but-temporarily-suitable 'fix' was in place. They (and Heroku) turned the magic knob and then let their attention move on to other things.
tedchs 14y ago
> We don't have the resources right now to get off of the platform quickly, I hear people say this a lot but I believe it comes more from the emotion of fear than a rational consideration of the possibilities. For example, Amazon Elastic Beanstalk now supports Rails deployments that are just as easy as Heroku, and you retain 100% control over the underlying EC2/ELB/etc resources while having a nice management layer to help you out.
- aneth4 14y ago
  Honestly I have not looked into it so far and hope you're right. We also use a number of addons, though I don't think any would be too difficult to replicate. Regardless, it's a significant amount of work for a startup with just a few engineers, perhaps a week of two of distraction. I expect a bunch of "how to migrate off of heroku" blog posts will be out there shortly.
  - tedchs 14y ago
    Thanks very much for the blog post idea, I think I'll do something with that! FYI you can keep all your Heroku addons inside your Heroku account while utilizing them from a server outside Heroku. For example, if you use the "Redis To Go" Heroku addon, you can fire up your own EC2 instance (possibly through AWS Elastic Beanstalk) and connect to your Redis To Go instance using the same connection info. The main difference will be that you won't have access to Heroku's environment variables that you're using now, but you can use a config file instead.
jpdoctor 14y ago
> While I have no interest in recovering money through a class action lawsuit I note that there are zero words devoted to what fraction of the money will leave the coffers of the lawyers and actually make it to the customers.
- wilfra 14y ago
  That's regulated by law.
lsc 14y ago
So, uh, what was the thought process when choosing a PAAS provider? I mean, step back for a moment, and ignore the specifics of Heroku; they are a PAAS provider with no completely compatible competitors, is that not so? Whenever I choose service providers (or choose to outsource services) my first thought is "what happens if I have to switch away from them?" Even if you think my first rule of business is too cynical, everyone has problems, and if you are locked into a provider while they are having problems, well, you have a problem. I mean, PAAS seems like a great idea for people who want to write apps but not be sysadmins. But personally? I don't understand why anyone would sign up with a PAAS provider that was unique. I mean, if you have to re-write your app to change providers, you are locked in, in a terrifying way.
- aneth4 14y ago
  That's certainly a consideration, but every PAAS is "unique" by definition. Something like Google Apps has more lock-in than others. Some like Heroku are attractive because they have features not available elsewhere that would take a lot of resources to reproduce. We chose heroku and Rails for the reasons you suggest, and even tried to avoid major tie-ins to the platform. That doesn't mean it's trivial to move to another platform.
  - lsc 14y ago
    > That's certainly a consideration, but every PAAS is "unique" by definition. I don't see how this is true. There is nothing about PaaS that says there can't be compatible competitors. Hell, there have been PHP shared hosting platforms that were very nearly identical between providers for a decade before anyone started saying PaaS, and really, the "Innovation" that made shared hosting PaaS lies in per-usage billing.
    - deleted 14y ago
      [deleted]
  - okalex 14y ago
    That's not entirely true. Cloud Foundry (the software) is an open-source platform that allows for compatibility across providers. You can see a list of compatible providers here, in addition to cloudfoundry.com: http://www.cloudfoundry.com/partners http://www.cloudfoundry.com/partners Full disclosure: I work for AppFog, one of the providers listed.
talklittle 14y ago
> Rapleaf You mean Rap Genius?
grey-area 14y ago
Like Rapleaf, we have spent an enormous amount of effort optimizing and searching for the causes of latency, and overpay for dynos to reduce but not eliminate the chance of latency issues. Heroku has always told us it's our fault. You may well find that running your own ops means you run into exactly the same sort of issues - most companies see multiple major problems with their infrastructure as they scale. Heroku certainly haven't delivered for everyone, and it seems their promises of painless scaling are just not being met when it comes to larger customers, but this stuff is hard, and moving off heroku won't get rid of these problems for you, but it will make them your problems and perhaps give you more of a chance to fix them. It would be really interesting (for heroku customers and for everyone else) to see a write up of your transition when you do move - it may not be as painful to do so as you anticipate. Kudos on not recommending a lawsuit, which I can't see helping anyone but lawyers at this point.

by aneth4 14y ago

deleted 14y ago

[deleted]

JangoSteve 14y ago
EDIT: I don't necessarily support a class action lawsuit against Heroku, but in the interest of conversation... Scalability is not a magic knob you can twist as far as you want… Isn't that the problem? That's Heroku's selling point is that scaling is basically a magic knob. Here are some of the things on their homepage right now: deploy and scale powerful apps Focus 100% on your code, and never think about servers, instances, or VMs again. Run any web or background process with any web framework or worker type. Scale distributed processes effortlessly with a single command. Easily scale to millions of users. Your comment seems to be defending Heroku, by saying Heroku is lying.
- deleted 14y ago
  [deleted]
  - JangoSteve 14y ago
    If you build your app in the correct way That's interesting terminology to use, "the correct way". Heroku's own devcenter article on deploying Rails apps to Heroku [1] still recommends the thin webserver, which isn't concurrent, and therefore causes all the issues being discussed for any rails app. In other words, if you're using thin, it doesn't matter how you architected your app, it's not going to run in concurrent processes on one dyno. So I'll assume by "the correct way", you're referring to both building your app for concurrency and using unicorn as your webserver. Doing only one of those things will not work, both are necessary. But their article only mentions unicorn only as an alternative, and up until recently their documentation actually said "We recommend thin" [2] over unicorn and that unicorn was possible but wasn't officially supported (EDIT: actually the most recent version of the article in wayback machine doesn't even mention unicorn on the page). I wouldn't call using thin "naive", given that it's what Heroku's docs have recommended for a long time. I don't think it's fair to call customers of Heroku naive for not knowing that when Heroku says "easily scale" and "use thin", they are talking about two mutually exclusive possibilites. [1] https://devcenter.heroku.com/articles/rails3 https://devcenter.heroku.com/articles/rails3 [2] http://web.archive.org/web/20120801214821/https://devcenter.heroku.com/articles/rails3 http://web.archive.org/web/20120801214821/https://devcenter.... EDIT: Added quotes with references.
aneth4 14y ago
"Scalable way" as defined by the platform Heroku was built and marketed for is single threaded. I won't argue that the lack of concurrency support in rails and ruby is not also a stupid horrible unacceptable choice - and I never understood why the rails community defended it. Regardless, Heroku was originally built for and continues to market itself as a platform that can scale Rails by using intelligent routing, but it's quite clear that they can not. Maybe you understand the above, though it seems you are just trolling rails for being single-threaded. This is not the place.
gojomo 14y ago
As bad as the latency-creating bottlenecks with random-routing and single-threaded apps can get, it is always the case that "twisting the magic knob" of adding more dynos would help. It just wouldn't help as much as people expected it to. Further, the amount of paid-for capacity left idle (while other dynos were backed-up) offends many sensibilities about how a system "ought to" work. But at least at the margin, the "magic knob" still worked a little. I think that's why the problem festered: there was always a default recommendation -- "more dynos!" -- that helped, which also just happened to fit the mental model Heroku promotes (while also boosting its revenue). In one of the older support logs RapGenius posted, it was evident that while RapGenius was suspicious about so many dynos being necessary, they let the ticket close once that expensive-but-temporarily-suitable 'fix' was in place. They (and Heroku) turned the magic knob and then let their attention move on to other things.

tedchs 14y ago

> We don't have the resources right now to get off of the platform quickly, I hear people say this a lot but I believe it comes more from the emotion of fear than a rational consideration of the possibilities. For example, Amazon Elastic Beanstalk now supports Rails deployments that are just as easy as Heroku, and you retain 100% control over the underlying EC2/ELB/etc resources while having a nice management layer to help you out.

aneth4 14y ago
Honestly I have not looked into it so far and hope you're right. We also use a number of addons, though I don't think any would be too difficult to replicate. Regardless, it's a significant amount of work for a startup with just a few engineers, perhaps a week of two of distraction. I expect a bunch of "how to migrate off of heroku" blog posts will be out there shortly.
- tedchs 14y ago
  Thanks very much for the blog post idea, I think I'll do something with that! FYI you can keep all your Heroku addons inside your Heroku account while utilizing them from a server outside Heroku. For example, if you use the "Redis To Go" Heroku addon, you can fire up your own EC2 instance (possibly through AWS Elastic Beanstalk) and connect to your Redis To Go instance using the same connection info. The main difference will be that you won't have access to Heroku's environment variables that you're using now, but you can use a config file instead.

jpdoctor 14y ago

> While I have no interest in recovering money through a class action lawsuit I note that there are zero words devoted to what fraction of the money will leave the coffers of the lawyers and actually make it to the customers.

wilfra 14y ago
That's regulated by law.

lsc 14y ago

So, uh, what was the thought process when choosing a PAAS provider? I mean, step back for a moment, and ignore the specifics of Heroku; they are a PAAS provider with no completely compatible competitors, is that not so? Whenever I choose service providers (or choose to outsource services) my first thought is "what happens if I have to switch away from them?" Even if you think my first rule of business is too cynical, everyone has problems, and if you are locked into a provider while they are having problems, well, you have a problem. I mean, PAAS seems like a great idea for people who want to write apps but not be sysadmins. But personally? I don't understand why anyone would sign up with a PAAS provider that was unique. I mean, if you have to re-write your app to change providers, you are locked in, in a terrifying way.

aneth4 14y ago
That's certainly a consideration, but every PAAS is "unique" by definition. Something like Google Apps has more lock-in than others. Some like Heroku are attractive because they have features not available elsewhere that would take a lot of resources to reproduce. We chose heroku and Rails for the reasons you suggest, and even tried to avoid major tie-ins to the platform. That doesn't mean it's trivial to move to another platform.
- lsc 14y ago
  > That's certainly a consideration, but every PAAS is "unique" by definition. I don't see how this is true. There is nothing about PaaS that says there can't be compatible competitors. Hell, there have been PHP shared hosting platforms that were very nearly identical between providers for a decade before anyone started saying PaaS, and really, the "Innovation" that made shared hosting PaaS lies in per-usage billing.
  - deleted 14y ago
    [deleted]
- okalex 14y ago
  That's not entirely true. Cloud Foundry (the software) is an open-source platform that allows for compatibility across providers. You can see a list of compatible providers here, in addition to cloudfoundry.com: http://www.cloudfoundry.com/partners http://www.cloudfoundry.com/partners Full disclosure: I work for AppFog, one of the providers listed.

talklittle 14y ago

> Rapleaf You mean Rap Genius?

grey-area 14y ago

Like Rapleaf, we have spent an enormous amount of effort optimizing and searching for the causes of latency, and overpay for dynos to reduce but not eliminate the chance of latency issues. Heroku has always told us it's our fault. You may well find that running your own ops means you run into exactly the same sort of issues - most companies see multiple major problems with their infrastructure as they scale. Heroku certainly haven't delivered for everyone, and it seems their promises of painless scaling are just not being met when it comes to larger customers, but this stuff is hard, and moving off heroku won't get rid of these problems for you, but it will make them your problems and perhaps give you more of a chance to fix them. It would be really interesting (for heroku customers and for everyone else) to see a write up of your transition when you do move - it may not be as painful to do so as you anticipate. Kudos on not recommending a lawsuit, which I can't see helping anyone but lawyers at this point.