24 ms·

Heroku Blog: Routing Performance Update

309 points by sinak 14y ago

msoad 14y ago
I am not a Rails developer nor a sys admin. I didn't really understand what was the problem the reported via Rap Genius. My question is does it affect Node.js apps?
- latchkey 14y ago
  The smoking gun is H12 errors and they occur quite a bit on our NodeJS app. The key is to get the instances to serve requests as fast as possible, which is a good thing, except at some point you can only get responses out at fast as you can and there is quite a few mysterious unexplained things that cause responses to happen slowly. The routing layer is fundamentally designed incorrectly. H12 errors should never happen in a random fashion and they do. A request can sometimes take 10ms and other times, timeout after 30s. Even if it is just returning 'ok' and not doing anything else.
- boundlessdreamz 14y ago
  It will. If X is the max number of requests node can handle per dyno, the router can send >X requests to a single dyno even if you have other dynos which are processing <X requests. Essentially the router sends requests by random. This results in request pileups. In the original rap genius post, there is an animated image which shows this in action
jlgaddis 14y ago
> "... but until this week, we failed to see a common thread among these reports." So Rap Genius, a customer, was able to figure out the issues (from the outside looking in) but Heroku, "on the inside" wasn't able to figure them out? Or they're playing the "we didn't know, we're going to fix it right away" angle? EDIT: also, s/failed to/did not/ makes more sense. "failed" implies they tried.
- wging 14y ago
  If you want a press release to be ignored, release it at EOD on a Friday.
  - dav 14y ago
    Bingo
  - rdl 14y ago
    Until your customer blogs about it on Monday morning :)
    - jlgaddis 14y ago
      Yeah, it's not like this one was gonna slip by; that somehow, over the weekend, everyone would forget about it. Although, if I were Heroku, that's certainly what I would have been praying for. =)
  - Tloewald 14y ago
    ...after a meteor explodes over Russia.
- jonny_eh 14y ago
  To me it read as "we didn't know, we're not going to fix it right away (if ever), we'll just document it better"
eaurouge 14y ago
Our documentation recommends the use of Thin, which is a single-threaded, evented web server. In theory, an evented server like Thin can process multiple concurrent requests, but doing this successfully depends on the code you write and the libraries you use. Rails, in fact, does not yet reliably support concurrent request handling. This leaves Rails developers unable to leverage the additional concurrency capabilities offered by the Cedar stack, unless they move to a concurrent web server like Puma or Unicorn. So, do/will they now recommend Puma/Unicorn over Thin?
seldo 14y ago
Not to be too harsh, but I'm not sure whether "we had no idea it was so bad" is better or worse than "we knew it was bad, but didn't tell anybody" for a platform company. The tone of the post is appropriately apologetic, but this does make you wonder what other problems they're missing.
- trafficlight 14y ago
  As a long time NetAdmin for a small WISP I understand where they are coming from. Even with monitoring and health checks, there are a lot of unknowns on the network. Sometimes I don't know there's a problem until a customer calls to complain. And even when they do complain, it can be tough to identify a root cause even with all the data we collect. We try to do our best to be proactive, but sometimes you don't even know to look for something until it's a problem. Heroku has quite a few more moving parts than our modest network with a large percentage of that in the form of code written by thousands of other people. As long as Heroku is learning lessons from their failures, which I believe is the case, they are doing a good job.
  - fatbird 14y ago
    In this whole sad, sorry tale, the problem is that the Heroku support engineers clearly knew, and communicated to the customer, that the unexpected/unreported lag was caused by the random queuing. They knew what the problem was for three years; it was only when a big customer went very public that they adopted the right tone and action plan. I imagine there's great discomfort inside Heroku these days because somewhere between the line engineers who knew about this issue, and the CEO, the fact that customers were complaining was swallowed up.
    - deleted 14y ago
      [deleted]
  - fauigerzigerk 14y ago
    That's all very understandable up to a point. But what I find more than a little disconcerting is that they had paying customers complaining about this issue for years but were "unable" to find the root cause or were unwilling to explain it. After one customer's comlaint was picked up by a wider public it took them barely a day or two to find out and explain what was going on. In other words, as long as it suited them financially, they couldn't be bothered analyze, document and fix the issue properly. Now they come out apologizing because the reputational damage is greater the money they make from letting customers use their stupid architecture. This is a classic really. Time for a refund.
auggierose 14y ago
Nothing really changed. If you cannot see how Heroku fucked you as a customer, then they actually did not fuck you; Otherwise, just do it yourself.
jkat 14y ago
Depending on what side of Hanlon's razor you fall, the only conclusion I get from this is that they are either incompetent or dishonest. I have a very hard time believing that this issue remained unknown to them for years. As for the post, it's pretty much just documentation. I didn't see any apology. And the only promise of a better tomorrow is a vague "Working to better support concurrent-request Rails apps on Cedar".
- bunkat 14y ago
  I also didn't see any mention of refunds for all of the extra dynos that were needed due to the degrading performance of their service - or all the extra support hours where they told everyone 'not our problem!'.
  - redguava 14y ago
    I wish I could vote this up many more times. It's exactly what I want to find out about.
    - res0nat0r 14y ago
      What SLA did they explicitly fail to deliver on such that they should offer a rebate?
- jlgaddis 14y ago
  > ... they are either incompetent or dishonest ... Exactly. If they didn't know, they should have. If they did know, well, ...
- trustfundbaby 14y ago
  I didn't see any apology. And the only promise of a better tomorrow is a vague "Working to better support concurrent-request Rails apps on Cedar". ------- They apologized in another earlier post. https://blog.heroku.com/archives/2013/2/15/bamboo_routing_performance/ https://blog.heroku.com/archives/2013/2/15/bamboo_routing_pe... "We failed to explain how our product works. We failed to help our customers scale. We failed our community at large. I want to personally apologize, and commit to resolving this issue"
deleted 14y ago
[deleted]
mjackson 14y ago
tl;dr Don't run your single-threaded Rails app (read: the vast majority of Rails apps) on Heroku. Neither the Cedar or Bamboo stacks are optimized for them at this point. It's incredible to me how the performance of their stack has degraded so far for what is probably the most common use case for their platform.
- wpietri 14y ago
  I think that's partly an insensitivity driven by Rails itself. Ruby and Rails have many good features, but speed is not one of them. Spending time working with Rails definitely wore down my previously high standards for page speed. Given that their metrics hid the problem, I could see most shops saying, "Well, the stats are fast, so the occasional slow page we see must be a fluke."
csense 14y ago
It sounds like they have one routing cluster for all of Heroku. If this is the case, and large routing clusters are the problem's root cause, they should just shard the cluster. I.e., give their bigger customers like RapGenius (who said they pay Heroku $20k/month and whose HN post spurred this debate) their own dedicated routing cluster with 2-5 nodes. Once a single customer exceeds that size, they're probably paying Heroku enough that Heroku can afford to devote an engineer to working specifically with them to implement large-scale architecture choices like running the app's DB on a cluster, etc. Pile several smaller customers on a shared routing cluster to cut costs and keep the cluster utilization high, but once the cluster gets to be a certain size (reaches a fixed number of backend dynos or metrics get bad enough), start putting customers on a new cluster. It should be fairly trivial to use DNS or router rules to dynamically move existing customers from one routing cluster to another.
- fatbird 14y ago
  The problem, as documented by the customer who went public with this issue, is that their request distribution scheme went from intelligent (i.e., load-based) to random, and a random distribution of requests is almost guaranteed to cause significant queuing for some non-trivial number of requests unless one has an absurd amount of extra capacity in place already, with ruinous financial aspects.
  - sofal 14y ago
    As the artictle says, it was the gradual scaling of the router cluster that eventually led to the request routing being effectively random on the Bamboo stack.
  - csense 14y ago
    The problem, as documented by the article, is that requests have always been served randomly, but the maximum number of requests that can be queued for any single backend node is equal to the number of frontend nodes N in the routing cluster. When N is equal to one, it's exactly what the previous discussion has labeled "intelligent routing;" when N is small, it's similar enough to intelligent routing that few will notice the difference. As N becomes large, you need a proportional (hence also large) number of requests to trigger the load balancing feature. At some point, the load balancing no longer kicks in, because for any real-world application the total workload is finite. But the performance complaints start much earlier; the "load balancing" on e.g. a 50-node balancing cluster might still kick in and stop routing requests to a hung server, but the 49 users waiting behind the hung request still suffer the latency and complain. I've adopted this picture of the situation because it agrees with both the reported behavior discussed by RapGenius and others in the previous thread, and this article's discussion of Heroku's architecture.
Shank 14y ago
From the perspective of someone who might be looking at Heroku as a host in the future, this is a bit scary. Their response appears to be mostly apologetic in that they're sorry that it happened - but does nothing to address the issue. It's more of a "we screwed up, oh well" than anything else. They would have warranted a better response if they said they were actively looking into how to improve the routing system, but by the looks of things they're going to sit by and hope developers switch practices so they don't have to solve their problem.
- d1plo1d 14y ago
  Well they did say that they were working on fixing the speed of concurrent requests for rails on their platform. While that is vague it would point to them actively working on a solution. Regardless, this is a problem of web applications at scale. Personally, I've never had to scale an app above 2 dynamos. So I will continue to use their service since it works as advertised for the domain of small startups that are not yet at scale. The work of porting from Heroku to AWS is work that I would have to put in anyways so I see no reason to waist that time any earlier in a project then I have to. Sure moving the db will be a pain but it's something I'm willing to live with. The bit where they mention their working on concurrency: Working to better support concurrent-request Rails apps on Cedar
jeswin 14y ago
I'm not able to follow parts of the post. Our routing cluster remained small for most of Bamboo’s history, which masked this inefficiency. If you went from 1 router to 2, 50% of routers can't optimally route a request. If you went from 2 to 3, you would have 66% which can't route. 3 to 4, 75%. Once you get to say 10 routers, you are already at 90% sub-optimal routing. So are they saying, the had only 1 or 2 routers earlier?
- gojomo 14y ago
  I think that is what they're saying: for a long time Bamboo only had very few routers. One simple HTTP router can move a lot of requests! (Maybe other topologies also hid the problem for a while, such as apps having traffic that still tended to come through one router even though coming from any was possible.)
- cpleppert 14y ago
  This is exactly what I was thinking as I was reading this blog post. I did a double take when I read that because it makes absolutely no sense at all. It seems to imply that their fabled web serving stack never actually worked as advertised but no one cared or complained publicly enough to make Heroku explain the issue. I just can't believe that Heroku had like 2-5 routers in their entire stack. I'm wondering if maybe they have some kind of simple sharding for routers based on the app name so all hash values go to a small set of routers. I can understand if that was the case how large numbers of small applications would seem to developers and heroku customer support to be using a per application queue. Of course, this would break down the instant there were very large applications served on heroku that would overwhelm the small number of routers assigned to each shard bucket.
beambot 14y ago
This is still not ideal. Even if you're running unicorn, you're still susceptible to queueing spikes due to random load balancing. The concurrency just gives you a small buffer and/or some smoothing on 95th percentile responses. Right? At least there's a commitment to update the reporting tools... getting bad data in New Relic was (IMHO) the worst -- even worse than out-of-date docs.
- bgentry 14y ago
  Actually, no. Using unicorn with only 2 workers makes a tremendous difference, not just incremental. RapGenius' own statistical model demonstrates this. Picture each individual dyno in that case as its own "intelligent router". Since it's not distributed and this requires no network coordination, the job of knowing which workers are available becomes trivial. If you're inclined to read up on queuing theory, you'll see that having at least 2 processes per worker makes the problem much simpler.
  - beambot 14y ago
    Fantastic, that's good to hear. I'm able to run ~5 workers on Heroku with gunicorn (Django), so I imagine that means I'm outta the woods for a while at least. I would love to read up a bit on queuing theory. Any good pointers?
    - shabble 14y ago
      There's a basic interactive + theory overview at http://homepages.inf.ed.ac.uk/jeh/Simjava/queueing/mm1_q/mm1_q.html http://homepages.inf.ed.ac.uk/jeh/Simjava/queueing/mm1_q/mm1... (NB: via in-browser java applets, which most things have disabled by default just now)
chubot 14y ago
Well, this certainly calls into question their competence. They're a PaaS company that doesn't understand or measure their load balancing performance. If you are a PaaS company, and you only have 5 metrics you can record, then 99% percentile latency across all apps should be one of them. On another note: why is Rails single-threaded??? That seems unbelievable. So if you have a 2 second database query, your Rails process does nothing else for that 2 seconds? I mean people complain about the GIL in Python, which actually has reasons behind it, but this is just crazy.
- trustfundbaby 14y ago
  On another note: why is Rails single-threaded??? That seems unbelievable. So if you have a 2 second database query, your Rails process does nothing else for that 2 seconds? --------------------- Thats not what happens, Rails does not block on IO, it will switch to another thread/process another request while it waits for IO to complete in the first request. http://yehudakatz.com/2010/08/14/threads-in-ruby-enough-already/ http://yehudakatz.com/2010/08/14/threads-in-ruby-enough-alre...
  - chubot 14y ago
    I think you only read the beginning of that article, not the end. What the article seems to imply is that you have to use a web server that spins off Rails in multiple processes. Ruby does concurrent I/O (obviously) but Rails itself does not appear to do it by default, causing the problems discussed here. "Rails, in fact, does not yet reliably support concurrent request handling. This leaves Rails developers unable to leverage the additional concurrency capabilities offered by the Cedar stack, unless they move to a concurrent web server like Puma or Unicorn." "Most obviously, early versions of Rails were not threadsafe. As a result, all Rails users were operating with a mutex around the entire request, forcing Rails to behave like the first “Imagined” diagram above. Annoyingly, Mongrel, the most common Ruby web server for a few years, hardcoded this mutex into its Rails handler. As a result, if you spun up Rails in “threadsafe” mode a year ago using Mongrel, you would have gotten exactly zero concurrency."
- charliesome 14y ago
tomlemon 14y ago
http://rapgenius.com/Jesper-joergensen-routing-performance-update-lyrics http://rapgenius.com/Jesper-joergensen-routing-performance-u... Just put it up on Rap Genius – create an account to help explain this post to the Heroku users it affects!
michaelfairley 14y ago
There's another large issue with queueing requests on the dyno: when the app restarts, all the requests currently queued up on that dyno get dropped and the client receives a 503.
- j-kidd 14y ago
  Sad to know. `httpd -k restart` was available since the beginning of time.
gfodor 14y ago
Welp, I was waiting for their official response to decide if I should deploy my app with Heroku or roll up my sleeves and rig up AWS servers (which I've done before but was looking forward to not having to deal with it.) Based upon this post, it sounds like there are really no concrete steps that they have planned to fix the underlying issue. So, AWS it is. I am still considering having Heroku manage my PostgreSQL instance. This would be a large burden lifted leaving me to just manage the app servers, etc. Is there any reason to be concerned about their PostgreSQL hosting? Any horror stories?
- gojomo 14y ago
  Or, use their Cedar stack and multi-worker dynos, where the problem is much less acute (and is only going to affect you once you need many dynos). Figure that in a month or two they'll have learned and deployed more than you would on your own.
  - unclebucknasty 14y ago
    Good points, but I think the other (maybe bigger) issue here is one of perception. Yes, they may nail this particular problem, but will customers trust their handling of the next issue? Heroku's offering is based on instilling confidence in customers that it will just plain do what is advertised without the pain. The story here is that customers experienced additional pain not only because they used Heroku's service, but also because of Heroku's failure to address real customer concerns for so long.
- homosaur 14y ago
  If you're considering Heroku, don't automatically dismiss it because of all this. For one, it's unlikely your site/app will ever be as big as RapGenius. That's not a shot, just reality. They ran into problems as an edge case. Is Heroku and their architecture at fault? Hell yeah it is, but I have faith that they will fix it. Why? Because I really think that when push comes to shove, Heroku was actually trying to do the right thing with the changes they made and perhaps didn't consider or understand some of the ramifications of the changes they made to the Rails community. They may have fallen in love a little too much with the new node.js hotness and the like. Their CORE audience is startups/new business where Rails is very popular and they understand what this has done to their reputation. If they don't address this in a serious way it will damage their business severely. I don't personally use Heroku but I have used it in the past and would not hesitate to use it on an appropriate project.
dylanvee 14y ago
For what it's worth, Google App Engine uses a so-called "intelligent" global request queue/scheduler. In most circumstances it's quite effective.
- vetrom 14y ago
  Source please?
  - dylanvee 14y ago
    "Pending request latency arises when all of your application's available instances are too busy to serve new requests. When this happens, incoming requests go to a pending request queue." https://developers.google.com/appengine/docs/adminconsole/performancesettings#scheduler https://developers.google.com/appengine/docs/adminconsole/pe... Pending latency is exposed as a metric on the App Engine Dashboard.
- latchkey 14y ago
  It is effective, but not without problems. http://code.google.com/p/googleappengine/issues/detail?id=7865 http://code.google.com/p/googleappengine/issues/detail?id=78... http://code.google.com/p/googleappengine/issues/detail?id=5755 http://code.google.com/p/googleappengine/issues/detail?id=57... http://code.google.com/p/googleappengine/issues/detail?id=7836 http://code.google.com/p/googleappengine/issues/detail?id=78... Needless to say, it is VASTLY better than Heroku's router. You'll never see an H12 error with GAE.
  - dylanvee 14y ago
    Indeed, while having a global request queue is good, the scheduler's one-size-fits-all behavior can sometimes be suboptimal. Thankfully there are several ways in which it can be tuned: https://developers.google.com/appengine/docs/adminconsole/performancesettings#scheduler https://developers.google.com/appengine/docs/adminconsole/pe...
gojomo 14y ago
Hmm, no tangible solutions yet, but I expect that will be next. From the discussion I've seen they have roughly two minimal options: (1) Shard/tier the Bamboo routing nodes, so that a single router tends to handle any particular app, and thus the original behavior is restored. Consistent hashing on the app name could do the trick, or DNS tricks on the app names mapping to different routing subshards. (2) Enable dynos to refuse requests, perhaps by refusing a connect or returning an error or redirect that tells a router to try the next dyno. (There are some indications a 'try another' logic already exists in their routers, so it might even be possible for customers to do this without Heroku's help. I have a question in with Heroku support about which request-shedding techniques might work without generating end-user visible errors.) Both could potentially benefit from some new per-dyno load-monitoring features... which would also allow other more-sophisticated (but more costly and fragile at scale) balancing or load-shedding strategies. I can see the commentariat lynch mob is out, but definitive recommendations and fixes take time. As they've admitted and apologized for the problem, I'd guess they'll have a more comprehensive response before their end-of-the-month user conference.
- btilly 14y ago
  I would suggest a slight variation on the first. Have initial routers pick a random appropriate dyno then pick the router for that dyno. The second router picks a random appropriate dyno from the list it is in charge of. If you make sure that the dynos for a given app are clumped behind few routers in the second layer, then you effectively get the old behavior. But you get it in a much more scaleable way. The cost is, of course, that you add an extra router to everything. (I emailed this suggestion to them. I have no idea whether they will listen.)
- mixedbit 14y ago
  > (1) Shard/tier the Bamboo routing nodes... This would work, but would require another layer before routers to do the sharding. I'm not sure if Heroku currently has such layer, or is it beyond their control (provided by AWS). But this is definitely an option. > (2) Enable dynos to refuse requests... The way I understand the architecture is that Dynos code is fully controlled by a client. They would need to introduce another component that would be responsible for rejecting requests. With such additional component they could also try to reverse the flow: each dyno is assigned to a single router and asks this router for a next request where it has finished processing the previous one. Routers queue all requests and handle them only when dynos ask them to.
Tekker 14y ago
I appreciate the honesty, but I don't see any "this is how we'll fix it", rather, just "we promise to document it and make it clear to anyone who wants to measure it". Effectively, they have a fundamental architectural problem, and don't know how to work past it.
- timv 14y ago
  They also have a fundamental cultural problem, and don't appear to have recognised it. In short, they went for ~2 years with documentation that advertised features that the implementation didn't have, while receiving a string of support issues that they wouldn't acknowledge as their problem. Yet, the blog posts show no indication that they are interested in working out why they offered such terrible service to their customers and how they can fix the company culture to take this issues seriously in the future. There's a chance they'll come up with an architectural solution for their routing problem. But unless they do some serious introspection and work out why they (as a team) stuffed up so badly, then there's no chance that they'll fix that problem. And if they don't fix that, then why would anyone have confidence that this sort of issue isn't going to be commonplace? (Disclaimer: Not a Heroku customer)
oellegaard 14y ago
What a scumbag move from Heroku. 1) Releasing a press release at 7 AM in the morning on a Saturday (CET) 2) The release looks mostly like the stuff a politicians spindoctor would ask the politician to say. Don't promis/admit too much. 3) They clearly state that they want to continue with this extremely inefficient way of routing. The right thing to do would be to make smaller clusters of Load Balancers who could then do proper routing, e.g. measuring the number of requests per dyno, last processing time, etc. I'm currently working on a large project on Heroku and I'm very disappointed about this. We chose Heroku because we believed we could just `heroku scale web=X` when needed. Instead, now we know that it will be of very little use. In the next week, I will be looking into a solution where I can utilize Heroku's add-on system without running my apps in Heroku Dynos. Creating a small system to hosts LXC's on AWS EC2 seems within my capabilities (or I could use Cloud Foundry's application server component) - and I believe I can configure a load balancer better than Heroku. Let me know if anyone else is interrested - we could make an open source project for this :-)
- jacques_chester 14y ago
  > 1) Releasing a press release at 7 AM in the morning on a Saturday (CET) I think they aimed to put out a response ASAP.
  - matt2000 14y ago
    I agree, the other way to look at this is that they're working through the weekend to figure it out.
    - oellegaard 14y ago
      That would be great - more specifically if they would release a new one on Monday stating that they found a solution. I still don't like the concept of using a PaaS and having to discover these kind of things yourself.
edouard1234567 14y ago
Performance and reliability has never been their forte. They probably have the worst up time in the industry. Scalability... yes until the point you really need to scale. The main/only advantage I really see to heroku is the ease of deployment and how they managed to keep it simple.
latchkey 14y ago
I'm watching all this debate with great interest. The company I'm consulting for has ~150 dynos, a paid support contract, etc. We're on Cedar... NodeJS latest (which just got upgraded to 0.8.19 after being ignored since October), with Mongo, Rabbit and Redis hosted by third party addon companies. After a ton of H12 errors, they helped us find out some slow points and optimize some things that were relatively slow. On our own, we did a huge amount of work to make things as fast as possible. While the H12's have gotten better, nothing has gotten rid of them completely. It really points to something fundamentally wrong with the routing layer because at some level we just can't optimize our code any further. There is definitely quite a few times in the logging where we just can't explain how things are insanely slow and we certainly can't explain why we get H12 errors anymore. To the point where we just gave up with it. The thing that bothers me the most is that we have been complaining for a month now behind the scenes through our paid support contract about the things that are now being semi admitted in public. No PaaS is perfect and certainly hard problems are being worked on by smart people... the real issue here is the way that Heroku has pointed fingers at everyone but themselves until finally someone had the time and balls to get a posting to the top of HN.
- neebz 14y ago
  150 dynos with Node.js and you are still getting H12 errors? The issue so far reported is that if we ever get multiple requests on a single dyno then we'll have a queuing delay because Rails is essentially single-threaded. But with Node.js I think it would be a fairly large amount of requests on a single dyno I suspect before we get any queuing delays
  - jharsman 14y ago
    If your requests are CPU intensive, Node.js won't help since it doesn't support preemption. And even if you're primarily IO-limited, a single request that consumes too much CPU will cause queuing.
    - latchkey 14y ago
      At this point we've done so much optimization of our app that our requests are not CPU or IO bound (those things have been offloaded to backend processes through a Rabbit message queue) and we still get H12 errors and random slowness. At the time of this writing, in the last 10 minutes, we've had 2 H12 errors. It should be zero.
tomlemon 14y ago
Rap Genius cofounder here. Below is the full unedited text of https://help.heroku.com/tickets/37665 https://help.heroku.com/tickets/37665, a Heroku support ticket I logged about 1 year ago. Sorry it is so long, but I think you'll find it interesting: Tom@Rapgenius| about 1 year ago I know this is a bit of a vague problem, but I've been getting a bunch of Error H12 (Request Timeout)s recently, and I'm not sure what to do about it. It's not like I have some particularly slow actions; I'm getting this error for actions that under most circumstances work totally fine (i.e., return in less than 300ms). Also I don't have a deep request queue (I'm running 40 dynos which is more than enough). Maybe I'm doing some slow queries? Should I upgrade my DB? Also, I do notice that most of my app's time (according to New Relic) is being spent in Ruby (http://cl.ly/29132F272W2D0K1l2I3P http://cl.ly/29132F272W2D0K1l2I3P). Would upgrading Ruby to 1.9 noticeably help this performance? (I'm a bit nervous it'll create a ton of problems). Phil@Heroku Hello - I can look into this, but I'll need access to your New Relic account. Will you make sure 'phil@heroku.com' has access? Also, from your screenshot I notice your DB times are ~ 100 ms. We recommend keeping those times closer to 50 ms. You might be able to speed things up with a database upgrade. I'll look into New Relic once I have access and let you know what I find. Tom@Rapgenius Thanks, Phil! How do I give you access to my New Relic account? I tried clicking "account settings" and got this: http://cl.ly/0V2J3i0826400I2s3b2c http://cl.ly/0V2J3i0826400I2s3b2c Phil@Heroku Tom, I have access now. I'm not sure what was blocking me earlier. After looking at New Relic and the database server, I think a larger database will help. At the very least, it will be helpful to try the next level for a week and compare performance statistics in New Relic with the prior week. Your app is using an Ika right now, and the next step up is the Zilla database. We've made the upgrade process very simple, and it's outlined here - http://devcenter.heroku.com/articles/fast-database-changeovers?preview=1 http://devcenter.heroku.com/articles/fast-database-changeove... Your database is ~ 5.4 GB in size (via the 'heroku pg:info' command) so an upgrade shouldn't take too long. You will be able to test the process by adding a Follower and timing it via the 'heroku pg:wait' command. This should give you a good idea of how long it will take to spin up the new database. Also, should the Zilla not help much, the downgrade process to an Ika will be the same. You only pay for the resources used. The current database server appears to be a bit under-powered when it comes to Compute Units. The Zilla has more power and should provide some room to grow. As for an upgrade to Ruby 1.9.2, I'm not sure how much that would help. It would be an involved upgrade that would take time to plan and deploy. The database upgrade should be a quicker solution. Long-term you may want to consider moving to the Cedar stack and Ruby 1.9.2. Tom@Rapgenius Thanks! I'm upgrading now Tom@Rapgenius I'm still getting a ton of "Request Timeout" errors. E.g.: 2011-12-08 14:46:53.222 219 1 2011-12-08T14:46:53+00:00 d. heroku router - - Error H12 (Request timeout) -> GET rapgenius.com/Wale-ambition-lyrics dyno=web.17 queue= wait= service=30000ms status=503 bytes=0 one weird thing: there aren't any values listed for the "queue" and "wait" parameters. Could that indicate a problem? Could an exception have been thrown earlier in the request before the timeout? Or does the timeout error just indicate that the request took too long? If it's the latter I'm not sure how to troubleshoot all these errors since the associated actions are fast the vast majority of the time Tom@Rapgenius Here's another interesting example: 2011-12-08 15:59:32.293 222 1 2011-12-08T15:59:32+00:00 d. heroku router - - Error H12 (Request timeout) -> GET rapgenius.com/static/templates_for_js dyno=web.17 queue= wait= service=30000ms status=503 bytes=0 This action is extremely simple – it doesn't access the DB or any external services. Here's the template: Ballin! <% unless current_user %> <% form_for User.new, :html => { :id => '' } do |f| %> Tired of entering your email address? Create a Rap Genius account and you'll never have to worry about it (or anything else) ever again: <%= render :partial => "/users/form", :object => f %> <%= f.submit "Create Account" %> <small>(Already have an account? <%= link_to 'Sign in', login_path, :class => :facebox %>)</small> <% end %> <% end %> Besides a big request queue (which there isn't), how could this action possibly time out? Phil@Heroku Tom - sorry for not getting back to you sooner. It's possible for H12s to occur even for simple actions if there is already queueing for the app. With a busy site like your's, even a few H12s can cause a cascade of H12s for successive requests. It looks like New Relic has not reported any downtime over the past 24 hours. Can we let the site run through the weekend and see how things look Monday after 3 days of New Relic data with the new Zilla? Tom@Rapgenius > It's possible for H12s to occur even for simple actions if there is already queueing for the app. I feel what you're saying, but I don't think my app's queuing. For one thing, New Relic shows 0 time spent in the queue during the period in which I'm getting all these timeouts. For another, I'm running 40 dynos and my average request time is <400ms. So: 400 ms * 3000 requests / minute * 1 min / 60000 ms = 20 simultaneous requests (i.e., 20 dynos) so 40 dynos should definitely be more than enough. Also, shouldn't Heroku be showing me the queue / wait stats at the time of the timeout? That would help prove whether my app was queuing at the time in question It looks like New Relic has not reported any downtime over the past 24 hours. New Relic isn't great at catching intermittent problems like this; you really feel it when you're using the site continuously for an hour or whatever. Also, users make many more HTTP requests than New Relic (since every page load kicks off several AJAX requests). That said, there has been downtime in the past 24 hours (though less than in the previous 24): http://cl.ly/3D2b1Z170B0w1f1m113m http://cl.ly/3D2b1Z170B0w1f1m113m Tom@Rapgenius Here's some additional data: At 5am this morning (EST), Rap Genius went down. I woke up at 11am (it's a Saturday!), did a logs --tail and observed that basically every request was timing out. I did heroku restart, and now every request started returning a backlog too deep error Finally, I added another 10 dynos (bumping the total to 50, which is a log of dynos!), and this seems to have fixed the problem – perhaps because my app needs the additional capacity, or perhaps because merely changing the number of dynos reset something else. Either way, I'm sticking with 50 dynos for now out of fear even though I doubt my app needs that many (right?) Either way, the 5 hours of unexplained downtime (there weren't any application-level exceptions or anything) that was fixable by tweaking my dyno count further supports my theory that something's going on with my app on Heroku's end. Phil@Rapgenius Tom - I've been looking over your New Relic stats. First - the good news - the upgrade to a Zilla seems to have helped. Database times are down a bit, which can only help. I checked the actual database server and it's not showing signs of over-work like the previous Ika was. Second, I notice that downtimes reported by New Relic over the past two weeks are in the early morning hours - 3 to 6 AM PST. Do you have any scheduled tasks that run during these times? Also, request queueing is nearly zero, so 50 dynos does seem like a lot. What are your usage patterns like? The RPM graph in New Relic indicates the normal cyclical usage pattern, lower during the night, but what does Google Analytics tell you? Finally, the Heroku platform has been having issues over the past week, but none of them correspond to the downtime you had Saturday morning.
pootch 14y ago
This was a great explanation of how Heroku routing works. And a terrible explanation of why anyone would use a service with such a brain dead router. Seriously is this company run by 16 year olds? Consider your allowance revoked. Rookies.
orensol 14y ago
I think the discussion comes down to which option you ultimately prefer -- 1) be in full control of your stack, having only yourself to criticize, and having to deal with all the devops action that comes with it, or 2) let heroku deal with the underlying while you focus on your app, as their promise suggests, and suffer (also financially) from time to time due to faults like this. Everyone has their own reasoning on this decision, and that's ok. I think that the voices here saying "I won't start my project on heroku because of this" are not thinking this through. I think that for any project starting up, focusing on app rather than infrastructure is much more important than routing issues.
azarias 14y ago
It seems to me that Heroku has chosen to be dishonest: Heroku's blog response: "but until this week, we failed to see a common thread among these reports." vs. Adam's response to Tim Watson, a year ago: "You're correct, the routing mesh does not behave in quite the way described by the docs. We're working on evolving away from the global backlog concept in order to provide better support for different concurrency models, and the docs are no longer accurate. The current behavior is not ideal, but we're on our way to a new model which we'll document fully once it's done." https://groups.google.com/forum/?fromgroups=#!msg/heroku/8eOosLC5nrw/Xy2j7GapebIJ https://groups.google.com/forum/?fromgroups=#!msg/heroku/8eO...
- btilly 14y ago
  It is quite possible to know that the mesh is not exactly as documented without having realized that the difference has a severe performance impact.
  - azarias 14y ago
    I think they also addressed the performance issue in addition to saying it is not documented: "...evolving away from the global backlog concept in order to provide better support for different concurrency models"
    - btilly 14y ago
      The fact that a year later, performance for their starting use case had not been addressed says that they failed to address the performance issue. It is still not clear to me that they actually really understood the performance issue, or how big it was going to be.
jlouis 14y ago
So to recap: Ruby on Rails is using a default configuration where each process can serve one request at a time. There is no cooperative switch (as in Node.js) or (near) preemptive switch (as in Erlang, Haskell, Go, ...). The routing infrastructure at Heroku is distributed. There are several routers and one router will queue at most one message per back-end dyno in the Bamboo stack and route randomly in the Cedar stack. If two front-end routers route messages to the same Dyno, then you get a queue, which happens more often on a large router mesh. Forgetting who is right and wrong, there are a couple of points to make in my opinion. The RoR model is very weak. You need to handle more than one connection concurrently, because under high load queueing will eventually happen. If one expensive request goes into the queue, then everyone further down the queue waits. In a more modern system like Node.js you can manually break up the expensive request and thus give service to other requests in the queue while the back-end works on the expensive req. In stronger models, Haskell, Go and Erlang, this break-up is usually automatic and preemption makes sure it is not a problem. If you have a 5000ms job A and 10 50ms jobs, then after 1ms, the A job will be preempted and then the 50ms jobs will get service. Thus an expensive job doesn't clog the queue. Random queueing in these models are often a very sensible choice. Note that Heroku is doing distributed routing. Thus the statistical model Rapgenius has made is wrong. One, requests does not arrive in a Poisson process. Usually one page load gives rise to several other calls to the back-end and this makes the requests dependent on each other. Two, there is not a single queue and router but multiple such. This means: * You need to take care of state between the queues - if they are to share information. This has overhead. Often considerable overhead. * You need to take care of failures of queues dynamically. A singular queue is easy to handle, but it also imposes a single point of failure and is a performance bottleneck. * You have very little knowledge of what kind of system is handling requests. Three, nobody is discussing how to handle the overload situation. What if your dynos can take 2000 req/s but the current arrival rate is 3000, if you forget about routing for a moment. How do you choose to drop requests, because you will have to do so. If you want to solve this going forward, you probably need Dyno queue feedback. Rapgenius uses the length of the queue in their test, but this is also wrong. They should use the sojourn time spent in the queue which is an indicator for how long you wait in the queue before being given service. According to rapgenius, they have a distribution where requests usually take 46ms (median) but the maximum is above 2000ms. I can roughly have a queue length of 43 and 1 have the same sojourn time then. Given this, you can feed back to the routers about how long a process will usually stay in queue. But again, this is without assuming distribution of the routers. The problem is way way harder to solve in that case. (edit for clarity in bullet list)
antirez 14y ago
Come on startups, you should be technically skilled and able to optimize in order to spend little money. If you sum EC2 and Heroku you are going to pay like 10x what it takes to run the same machines power in a dedicated server, all this because you can't handle the operations? This is absurd IMHO. Also people that want to start a business, there is a huge opportunity here, create software that makes managing Apache, Redis, PostgreSQL, ..., in dedicated servers very easy and robust. Traget a popular and robust non-commercial distribution like Ubuntu LTE, and provide all is needed to deploy web nodes, database nodes, with backups, monitoring, and everything else trivial. Startups can give you 5% of what they are giving now to EC2 and Heroku and you still will make a lot of money. "I can only write my Ruby code but can't handle operations" is not the right attitude, grow up. (This is what, in a different context, a famous security researcher told me in a private mail 15 years ago, and it was the right advice)
- Adirael 14y ago
  If your Ruby code deploys to X you should know X, if it's part of your stack it should be part of your toolbox. "create software that makes managing Apache, Redis, PostgreSQL" Like Plesk, but better, faster and less resource hungry.
- ig1 14y ago
  The primary bottleneck for most startups isn't money, it's time. Also for most startups (assuming you're not doing something CPU/bandwidth heavy) the actual cost of hosting is going to be a relatively small part of the budget. If your burn rate is a million dollars, reducing 10k of hosting costs to 1k isn't really worth the effort.
  - fauigerzigerk 14y ago
    Money can buy time in various forms. I think PaaS is one abstraction level too much to remain flexible and efficient as you grow.
  - raverbashing 14y ago
    Kind of depends There's the "two guys in a dorm eating ramen" and there's the YC backed "startup" that already has a couple of hires So if your monthly budget is of hundreds of dollars, using a VPS instead of Heroku makes sense
programminggeek 14y ago
I feel like Heroku is punting on this one in a big way. I think this really means that for a small app, Heroku is fine, once it gets bigger, you probably want to get a real scaling strategy.
- cygwin98 14y ago
  Agree. Though that's a bit ironic --- one of Heroku's selling points was scalability, at least a big part of Rails community believed it to be.
steveklabnik 14y ago
> On the Cedar stack, the root cause is the fact that Cedar is optimized for concurrent request routing, while some frameworks, like Rails, are not concurrent in their default configurations. First of all, Ruby doesn't exactly block on I/O [1], so there's that. Secondly, while Rails 3 has config.threadsafe! turned off, Rails 4 will have it turned on. [2] 1: https://github.com/tenderlove/fibur#synopsis https://github.com/tenderlove/fibur#synopsis 2: http://tenderlovemaking.com/2012/06/18/removing-config-threadsafe.html http://tenderlovemaking.com/2012/06/18/removing-config-threa... EDIT: I just realized non-Rubyists might not get the Fibur joke. Here's the code: https://github.com/tenderlove/fibur/blob/master/lib/fibur.rb#L1 https://github.com/tenderlove/fibur/blob/master/lib/fibur.rb...
meritt 14y ago
People pay $35/mo for a Heroku Dyno that can only handle 1 request at a time? That's absurd.
badgar 14y ago
So the tl;dr from Heroku is: we can't scale single-threaded applications.
jotnar 14y ago
They should of just used nodejs for routing.
jotnar 14y ago
You know, I wrote a really fast HTTP router using Redis pub/sub and a pool of single threaded nodejs processes running on seperate EC2 boxes and it worked great. I don't know why it is so hard to scale RoR when it is single-threaded.
sneak 14y ago
This guy's writing style and tone make me want to never do business with him.
hpguy 14y ago
Don't know about Rails, but I'm building a MVP using Node.JS and hosting a free dyno. Can handle 250 reqs/s for dynamic requests (i.e. full stack, Postgres, Redis, Jade, Express etc.) and 1500+ reqs/s for static contents. I do use cluster with 3 workers though. Pretty happy with the result so far.

309 points by sinak 14y ago

msoad 14y ago

I am not a Rails developer nor a sys admin. I didn't really understand what was the problem the reported via Rap Genius. My question is does it affect Node.js apps?

latchkey 14y ago
The smoking gun is H12 errors and they occur quite a bit on our NodeJS app. The key is to get the instances to serve requests as fast as possible, which is a good thing, except at some point you can only get responses out at fast as you can and there is quite a few mysterious unexplained things that cause responses to happen slowly. The routing layer is fundamentally designed incorrectly. H12 errors should never happen in a random fashion and they do. A request can sometimes take 10ms and other times, timeout after 30s. Even if it is just returning 'ok' and not doing anything else.
boundlessdreamz 14y ago
It will. If X is the max number of requests node can handle per dyno, the router can send >X requests to a single dyno even if you have other dynos which are processing <X requests. Essentially the router sends requests by random. This results in request pileups. In the original rap genius post, there is an animated image which shows this in action

jlgaddis 14y ago

> "... but until this week, we failed to see a common thread among these reports." So Rap Genius, a customer, was able to figure out the issues (from the outside looking in) but Heroku, "on the inside" wasn't able to figure them out? Or they're playing the "we didn't know, we're going to fix it right away" angle? EDIT: also, s/failed to/did not/ makes more sense. "failed" implies they tried.

wging 14y ago
If you want a press release to be ignored, release it at EOD on a Friday.
- dav 14y ago
  Bingo
- rdl 14y ago
  Until your customer blogs about it on Monday morning :)
  - jlgaddis 14y ago
    Yeah, it's not like this one was gonna slip by; that somehow, over the weekend, everyone would forget about it. Although, if I were Heroku, that's certainly what I would have been praying for. =)
- Tloewald 14y ago
  ...after a meteor explodes over Russia.
jonny_eh 14y ago
To me it read as "we didn't know, we're not going to fix it right away (if ever), we'll just document it better"

eaurouge 14y ago

Our documentation recommends the use of Thin, which is a single-threaded, evented web server. In theory, an evented server like Thin can process multiple concurrent requests, but doing this successfully depends on the code you write and the libraries you use. Rails, in fact, does not yet reliably support concurrent request handling. This leaves Rails developers unable to leverage the additional concurrency capabilities offered by the Cedar stack, unless they move to a concurrent web server like Puma or Unicorn. So, do/will they now recommend Puma/Unicorn over Thin?

seldo 14y ago

Not to be too harsh, but I'm not sure whether "we had no idea it was so bad" is better or worse than "we knew it was bad, but didn't tell anybody" for a platform company. The tone of the post is appropriately apologetic, but this does make you wonder what other problems they're missing.

trafficlight 14y ago
As a long time NetAdmin for a small WISP I understand where they are coming from. Even with monitoring and health checks, there are a lot of unknowns on the network. Sometimes I don't know there's a problem until a customer calls to complain. And even when they do complain, it can be tough to identify a root cause even with all the data we collect. We try to do our best to be proactive, but sometimes you don't even know to look for something until it's a problem. Heroku has quite a few more moving parts than our modest network with a large percentage of that in the form of code written by thousands of other people. As long as Heroku is learning lessons from their failures, which I believe is the case, they are doing a good job.
- fatbird 14y ago
  In this whole sad, sorry tale, the problem is that the Heroku support engineers clearly knew, and communicated to the customer, that the unexpected/unreported lag was caused by the random queuing. They knew what the problem was for three years; it was only when a big customer went very public that they adopted the right tone and action plan. I imagine there's great discomfort inside Heroku these days because somewhere between the line engineers who knew about this issue, and the CEO, the fact that customers were complaining was swallowed up.
  - deleted 14y ago
    [deleted]
- fauigerzigerk 14y ago
  That's all very understandable up to a point. But what I find more than a little disconcerting is that they had paying customers complaining about this issue for years but were "unable" to find the root cause or were unwilling to explain it. After one customer's comlaint was picked up by a wider public it took them barely a day or two to find out and explain what was going on. In other words, as long as it suited them financially, they couldn't be bothered analyze, document and fix the issue properly. Now they come out apologizing because the reputational damage is greater the money they make from letting customers use their stupid architecture. This is a classic really. Time for a refund.

auggierose 14y ago

Nothing really changed. If you cannot see how Heroku fucked you as a customer, then they actually did not fuck you; Otherwise, just do it yourself.

jkat 14y ago

Depending on what side of Hanlon's razor you fall, the only conclusion I get from this is that they are either incompetent or dishonest. I have a very hard time believing that this issue remained unknown to them for years. As for the post, it's pretty much just documentation. I didn't see any apology. And the only promise of a better tomorrow is a vague "Working to better support concurrent-request Rails apps on Cedar".

bunkat 14y ago
I also didn't see any mention of refunds for all of the extra dynos that were needed due to the degrading performance of their service - or all the extra support hours where they told everyone 'not our problem!'.
- redguava 14y ago
  I wish I could vote this up many more times. It's exactly what I want to find out about.
  - res0nat0r 14y ago
    What SLA did they explicitly fail to deliver on such that they should offer a rebate?
jlgaddis 14y ago
> ... they are either incompetent or dishonest ... Exactly. If they didn't know, they should have. If they did know, well, ...
trustfundbaby 14y ago
I didn't see any apology. And the only promise of a better tomorrow is a vague "Working to better support concurrent-request Rails apps on Cedar". ------- They apologized in another earlier post. https://blog.heroku.com/archives/2013/2/15/bamboo_routing_performance/ https://blog.heroku.com/archives/2013/2/15/bamboo_routing_pe... "We failed to explain how our product works. We failed to help our customers scale. We failed our community at large. I want to personally apologize, and commit to resolving this issue"

deleted 14y ago

[deleted]

mjackson 14y ago

tl;dr Don't run your single-threaded Rails app (read: the vast majority of Rails apps) on Heroku. Neither the Cedar or Bamboo stacks are optimized for them at this point. It's incredible to me how the performance of their stack has degraded so far for what is probably the most common use case for their platform.

wpietri 14y ago
I think that's partly an insensitivity driven by Rails itself. Ruby and Rails have many good features, but speed is not one of them. Spending time working with Rails definitely wore down my previously high standards for page speed. Given that their metrics hid the problem, I could see most shops saying, "Well, the stats are fast, so the occasional slow page we see must be a fluke."

csense 14y ago

It sounds like they have one routing cluster for all of Heroku. If this is the case, and large routing clusters are the problem's root cause, they should just shard the cluster. I.e., give their bigger customers like RapGenius (who said they pay Heroku $20k/month and whose HN post spurred this debate) their own dedicated routing cluster with 2-5 nodes. Once a single customer exceeds that size, they're probably paying Heroku enough that Heroku can afford to devote an engineer to working specifically with them to implement large-scale architecture choices like running the app's DB on a cluster, etc. Pile several smaller customers on a shared routing cluster to cut costs and keep the cluster utilization high, but once the cluster gets to be a certain size (reaches a fixed number of backend dynos or metrics get bad enough), start putting customers on a new cluster. It should be fairly trivial to use DNS or router rules to dynamically move existing customers from one routing cluster to another.

fatbird 14y ago
The problem, as documented by the customer who went public with this issue, is that their request distribution scheme went from intelligent (i.e., load-based) to random, and a random distribution of requests is almost guaranteed to cause significant queuing for some non-trivial number of requests unless one has an absurd amount of extra capacity in place already, with ruinous financial aspects.
- sofal 14y ago
  As the artictle says, it was the gradual scaling of the router cluster that eventually led to the request routing being effectively random on the Bamboo stack.
- csense 14y ago
  The problem, as documented by the article, is that requests have always been served randomly, but the maximum number of requests that can be queued for any single backend node is equal to the number of frontend nodes N in the routing cluster. When N is equal to one, it's exactly what the previous discussion has labeled "intelligent routing;" when N is small, it's similar enough to intelligent routing that few will notice the difference. As N becomes large, you need a proportional (hence also large) number of requests to trigger the load balancing feature. At some point, the load balancing no longer kicks in, because for any real-world application the total workload is finite. But the performance complaints start much earlier; the "load balancing" on e.g. a 50-node balancing cluster might still kick in and stop routing requests to a hung server, but the 49 users waiting behind the hung request still suffer the latency and complain. I've adopted this picture of the situation because it agrees with both the reported behavior discussed by RapGenius and others in the previous thread, and this article's discussion of Heroku's architecture.

Shank 14y ago

From the perspective of someone who might be looking at Heroku as a host in the future, this is a bit scary. Their response appears to be mostly apologetic in that they're sorry that it happened - but does nothing to address the issue. It's more of a "we screwed up, oh well" than anything else. They would have warranted a better response if they said they were actively looking into how to improve the routing system, but by the looks of things they're going to sit by and hope developers switch practices so they don't have to solve their problem.

d1plo1d 14y ago
Well they did say that they were working on fixing the speed of concurrent requests for rails on their platform. While that is vague it would point to them actively working on a solution. Regardless, this is a problem of web applications at scale. Personally, I've never had to scale an app above 2 dynamos. So I will continue to use their service since it works as advertised for the domain of small startups that are not yet at scale. The work of porting from Heroku to AWS is work that I would have to put in anyways so I see no reason to waist that time any earlier in a project then I have to. Sure moving the db will be a pain but it's something I'm willing to live with. The bit where they mention their working on concurrency: Working to better support concurrent-request Rails apps on Cedar

jeswin 14y ago

I'm not able to follow parts of the post. Our routing cluster remained small for most of Bamboo’s history, which masked this inefficiency. If you went from 1 router to 2, 50% of routers can't optimally route a request. If you went from 2 to 3, you would have 66% which can't route. 3 to 4, 75%. Once you get to say 10 routers, you are already at 90% sub-optimal routing. So are they saying, the had only 1 or 2 routers earlier?

gojomo 14y ago
I think that is what they're saying: for a long time Bamboo only had very few routers. One simple HTTP router can move a lot of requests! (Maybe other topologies also hid the problem for a while, such as apps having traffic that still tended to come through one router even though coming from any was possible.)
cpleppert 14y ago
This is exactly what I was thinking as I was reading this blog post. I did a double take when I read that because it makes absolutely no sense at all. It seems to imply that their fabled web serving stack never actually worked as advertised but no one cared or complained publicly enough to make Heroku explain the issue. I just can't believe that Heroku had like 2-5 routers in their entire stack. I'm wondering if maybe they have some kind of simple sharding for routers based on the app name so all hash values go to a small set of routers. I can understand if that was the case how large numbers of small applications would seem to developers and heroku customer support to be using a per application queue. Of course, this would break down the instant there were very large applications served on heroku that would overwhelm the small number of routers assigned to each shard bucket.

beambot 14y ago

This is still not ideal. Even if you're running unicorn, you're still susceptible to queueing spikes due to random load balancing. The concurrency just gives you a small buffer and/or some smoothing on 95th percentile responses. Right? At least there's a commitment to update the reporting tools... getting bad data in New Relic was (IMHO) the worst -- even worse than out-of-date docs.

bgentry 14y ago
Actually, no. Using unicorn with only 2 workers makes a tremendous difference, not just incremental. RapGenius' own statistical model demonstrates this. Picture each individual dyno in that case as its own "intelligent router". Since it's not distributed and this requires no network coordination, the job of knowing which workers are available becomes trivial. If you're inclined to read up on queuing theory, you'll see that having at least 2 processes per worker makes the problem much simpler.
- beambot 14y ago
  Fantastic, that's good to hear. I'm able to run ~5 workers on Heroku with gunicorn (Django), so I imagine that means I'm outta the woods for a while at least. I would love to read up a bit on queuing theory. Any good pointers?
  - shabble 14y ago
    There's a basic interactive + theory overview at http://homepages.inf.ed.ac.uk/jeh/Simjava/queueing/mm1_q/mm1_q.html http://homepages.inf.ed.ac.uk/jeh/Simjava/queueing/mm1_q/mm1... (NB: via in-browser java applets, which most things have disabled by default just now)

chubot 14y ago

Well, this certainly calls into question their competence. They're a PaaS company that doesn't understand or measure their load balancing performance. If you are a PaaS company, and you only have 5 metrics you can record, then 99% percentile latency across all apps should be one of them. On another note: why is Rails single-threaded??? That seems unbelievable. So if you have a 2 second database query, your Rails process does nothing else for that 2 seconds? I mean people complain about the GIL in Python, which actually has reasons behind it, but this is just crazy.

trustfundbaby 14y ago
On another note: why is Rails single-threaded??? That seems unbelievable. So if you have a 2 second database query, your Rails process does nothing else for that 2 seconds? --------------------- Thats not what happens, Rails does not block on IO, it will switch to another thread/process another request while it waits for IO to complete in the first request. http://yehudakatz.com/2010/08/14/threads-in-ruby-enough-already/ http://yehudakatz.com/2010/08/14/threads-in-ruby-enough-alre...
- chubot 14y ago
  I think you only read the beginning of that article, not the end. What the article seems to imply is that you have to use a web server that spins off Rails in multiple processes. Ruby does concurrent I/O (obviously) but Rails itself does not appear to do it by default, causing the problems discussed here. "Rails, in fact, does not yet reliably support concurrent request handling. This leaves Rails developers unable to leverage the additional concurrency capabilities offered by the Cedar stack, unless they move to a concurrent web server like Puma or Unicorn." "Most obviously, early versions of Rails were not threadsafe. As a result, all Rails users were operating with a mutex around the entire request, forcing Rails to behave like the first “Imagined” diagram above. Annoyingly, Mongrel, the most common Ruby web server for a few years, hardcoded this mutex into its Rails handler. As a result, if you spun up Rails in “threadsafe” mode a year ago using Mongrel, you would have gotten exactly zero concurrency."
charliesome 14y ago

tomlemon 14y ago

http://rapgenius.com/Jesper-joergensen-routing-performance-update-lyrics http://rapgenius.com/Jesper-joergensen-routing-performance-u... Just put it up on Rap Genius – create an account to help explain this post to the Heroku users it affects!

michaelfairley 14y ago

There's another large issue with queueing requests on the dyno: when the app restarts, all the requests currently queued up on that dyno get dropped and the client receives a 503.

j-kidd 14y ago
Sad to know. `httpd -k restart` was available since the beginning of time.

gfodor 14y ago

Welp, I was waiting for their official response to decide if I should deploy my app with Heroku or roll up my sleeves and rig up AWS servers (which I've done before but was looking forward to not having to deal with it.) Based upon this post, it sounds like there are really no concrete steps that they have planned to fix the underlying issue. So, AWS it is. I am still considering having Heroku manage my PostgreSQL instance. This would be a large burden lifted leaving me to just manage the app servers, etc. Is there any reason to be concerned about their PostgreSQL hosting? Any horror stories?

gojomo 14y ago
Or, use their Cedar stack and multi-worker dynos, where the problem is much less acute (and is only going to affect you once you need many dynos). Figure that in a month or two they'll have learned and deployed more than you would on your own.
- unclebucknasty 14y ago
  Good points, but I think the other (maybe bigger) issue here is one of perception. Yes, they may nail this particular problem, but will customers trust their handling of the next issue? Heroku's offering is based on instilling confidence in customers that it will just plain do what is advertised without the pain. The story here is that customers experienced additional pain not only because they used Heroku's service, but also because of Heroku's failure to address real customer concerns for so long.
homosaur 14y ago
If you're considering Heroku, don't automatically dismiss it because of all this. For one, it's unlikely your site/app will ever be as big as RapGenius. That's not a shot, just reality. They ran into problems as an edge case. Is Heroku and their architecture at fault? Hell yeah it is, but I have faith that they will fix it. Why? Because I really think that when push comes to shove, Heroku was actually trying to do the right thing with the changes they made and perhaps didn't consider or understand some of the ramifications of the changes they made to the Rails community. They may have fallen in love a little too much with the new node.js hotness and the like. Their CORE audience is startups/new business where Rails is very popular and they understand what this has done to their reputation. If they don't address this in a serious way it will damage their business severely. I don't personally use Heroku but I have used it in the past and would not hesitate to use it on an appropriate project.

dylanvee 14y ago

For what it's worth, Google App Engine uses a so-called "intelligent" global request queue/scheduler. In most circumstances it's quite effective.

vetrom 14y ago
Source please?
- dylanvee 14y ago
  "Pending request latency arises when all of your application's available instances are too busy to serve new requests. When this happens, incoming requests go to a pending request queue." https://developers.google.com/appengine/docs/adminconsole/performancesettings#scheduler https://developers.google.com/appengine/docs/adminconsole/pe... Pending latency is exposed as a metric on the App Engine Dashboard.
latchkey 14y ago
It is effective, but not without problems. http://code.google.com/p/googleappengine/issues/detail?id=7865 http://code.google.com/p/googleappengine/issues/detail?id=78... http://code.google.com/p/googleappengine/issues/detail?id=5755 http://code.google.com/p/googleappengine/issues/detail?id=57... http://code.google.com/p/googleappengine/issues/detail?id=7836 http://code.google.com/p/googleappengine/issues/detail?id=78... Needless to say, it is VASTLY better than Heroku's router. You'll never see an H12 error with GAE.
- dylanvee 14y ago
  Indeed, while having a global request queue is good, the scheduler's one-size-fits-all behavior can sometimes be suboptimal. Thankfully there are several ways in which it can be tuned: https://developers.google.com/appengine/docs/adminconsole/performancesettings#scheduler https://developers.google.com/appengine/docs/adminconsole/pe...

gojomo 14y ago

Hmm, no tangible solutions yet, but I expect that will be next. From the discussion I've seen they have roughly two minimal options: (1) Shard/tier the Bamboo routing nodes, so that a single router tends to handle any particular app, and thus the original behavior is restored. Consistent hashing on the app name could do the trick, or DNS tricks on the app names mapping to different routing subshards. (2) Enable dynos to refuse requests, perhaps by refusing a connect or returning an error or redirect that tells a router to try the next dyno. (There are some indications a 'try another' logic already exists in their routers, so it might even be possible for customers to do this without Heroku's help. I have a question in with Heroku support about which request-shedding techniques might work without generating end-user visible errors.) Both could potentially benefit from some new per-dyno load-monitoring features... which would also allow other more-sophisticated (but more costly and fragile at scale) balancing or load-shedding strategies. I can see the commentariat lynch mob is out, but definitive recommendations and fixes take time. As they've admitted and apologized for the problem, I'd guess they'll have a more comprehensive response before their end-of-the-month user conference.

btilly 14y ago
I would suggest a slight variation on the first. Have initial routers pick a random appropriate dyno then pick the router for that dyno. The second router picks a random appropriate dyno from the list it is in charge of. If you make sure that the dynos for a given app are clumped behind few routers in the second layer, then you effectively get the old behavior. But you get it in a much more scaleable way. The cost is, of course, that you add an extra router to everything. (I emailed this suggestion to them. I have no idea whether they will listen.)
mixedbit 14y ago
> (1) Shard/tier the Bamboo routing nodes... This would work, but would require another layer before routers to do the sharding. I'm not sure if Heroku currently has such layer, or is it beyond their control (provided by AWS). But this is definitely an option. > (2) Enable dynos to refuse requests... The way I understand the architecture is that Dynos code is fully controlled by a client. They would need to introduce another component that would be responsible for rejecting requests. With such additional component they could also try to reverse the flow: each dyno is assigned to a single router and asks this router for a next request where it has finished processing the previous one. Routers queue all requests and handle them only when dynos ask them to.

Tekker 14y ago

I appreciate the honesty, but I don't see any "this is how we'll fix it", rather, just "we promise to document it and make it clear to anyone who wants to measure it". Effectively, they have a fundamental architectural problem, and don't know how to work past it.

timv 14y ago
They also have a fundamental cultural problem, and don't appear to have recognised it. In short, they went for ~2 years with documentation that advertised features that the implementation didn't have, while receiving a string of support issues that they wouldn't acknowledge as their problem. Yet, the blog posts show no indication that they are interested in working out why they offered such terrible service to their customers and how they can fix the company culture to take this issues seriously in the future. There's a chance they'll come up with an architectural solution for their routing problem. But unless they do some serious introspection and work out why they (as a team) stuffed up so badly, then there's no chance that they'll fix that problem. And if they don't fix that, then why would anyone have confidence that this sort of issue isn't going to be commonplace? (Disclaimer: Not a Heroku customer)

oellegaard 14y ago

What a scumbag move from Heroku. 1) Releasing a press release at 7 AM in the morning on a Saturday (CET) 2) The release looks mostly like the stuff a politicians spindoctor would ask the politician to say. Don't promis/admit too much. 3) They clearly state that they want to continue with this extremely inefficient way of routing. The right thing to do would be to make smaller clusters of Load Balancers who could then do proper routing, e.g. measuring the number of requests per dyno, last processing time, etc. I'm currently working on a large project on Heroku and I'm very disappointed about this. We chose Heroku because we believed we could just `heroku scale web=X` when needed. Instead, now we know that it will be of very little use. In the next week, I will be looking into a solution where I can utilize Heroku's add-on system without running my apps in Heroku Dynos. Creating a small system to hosts LXC's on AWS EC2 seems within my capabilities (or I could use Cloud Foundry's application server component) - and I believe I can configure a load balancer better than Heroku. Let me know if anyone else is interrested - we could make an open source project for this :-)

jacques_chester 14y ago
> 1) Releasing a press release at 7 AM in the morning on a Saturday (CET) I think they aimed to put out a response ASAP.
- matt2000 14y ago
  I agree, the other way to look at this is that they're working through the weekend to figure it out.
  - oellegaard 14y ago
    That would be great - more specifically if they would release a new one on Monday stating that they found a solution. I still don't like the concept of using a PaaS and having to discover these kind of things yourself.

edouard1234567 14y ago

Performance and reliability has never been their forte. They probably have the worst up time in the industry. Scalability... yes until the point you really need to scale. The main/only advantage I really see to heroku is the ease of deployment and how they managed to keep it simple.

latchkey 14y ago

I'm watching all this debate with great interest. The company I'm consulting for has ~150 dynos, a paid support contract, etc. We're on Cedar... NodeJS latest (which just got upgraded to 0.8.19 after being ignored since October), with Mongo, Rabbit and Redis hosted by third party addon companies. After a ton of H12 errors, they helped us find out some slow points and optimize some things that were relatively slow. On our own, we did a huge amount of work to make things as fast as possible. While the H12's have gotten better, nothing has gotten rid of them completely. It really points to something fundamentally wrong with the routing layer because at some level we just can't optimize our code any further. There is definitely quite a few times in the logging where we just can't explain how things are insanely slow and we certainly can't explain why we get H12 errors anymore. To the point where we just gave up with it. The thing that bothers me the most is that we have been complaining for a month now behind the scenes through our paid support contract about the things that are now being semi admitted in public. No PaaS is perfect and certainly hard problems are being worked on by smart people... the real issue here is the way that Heroku has pointed fingers at everyone but themselves until finally someone had the time and balls to get a posting to the top of HN.

neebz 14y ago
150 dynos with Node.js and you are still getting H12 errors? The issue so far reported is that if we ever get multiple requests on a single dyno then we'll have a queuing delay because Rails is essentially single-threaded. But with Node.js I think it would be a fairly large amount of requests on a single dyno I suspect before we get any queuing delays
- jharsman 14y ago
  If your requests are CPU intensive, Node.js won't help since it doesn't support preemption. And even if you're primarily IO-limited, a single request that consumes too much CPU will cause queuing.
  - latchkey 14y ago
    At this point we've done so much optimization of our app that our requests are not CPU or IO bound (those things have been offloaded to backend processes through a Rabbit message queue) and we still get H12 errors and random slowness. At the time of this writing, in the last 10 minutes, we've had 2 H12 errors. It should be zero.

tomlemon 14y ago

Rap Genius cofounder here. Below is the full unedited text of https://help.heroku.com/tickets/37665 https://help.heroku.com/tickets/37665, a Heroku support ticket I logged about 1 year ago. Sorry it is so long, but I think you'll find it interesting: Tom@Rapgenius| about 1 year ago I know this is a bit of a vague problem, but I've been getting a bunch of Error H12 (Request Timeout)s recently, and I'm not sure what to do about it. It's not like I have some particularly slow actions; I'm getting this error for actions that under most circumstances work totally fine (i.e., return in less than 300ms). Also I don't have a deep request queue (I'm running 40 dynos which is more than enough). Maybe I'm doing some slow queries? Should I upgrade my DB? Also, I do notice that most of my app's time (according to New Relic) is being spent in Ruby (http://cl.ly/29132F272W2D0K1l2I3P http://cl.ly/29132F272W2D0K1l2I3P). Would upgrading Ruby to 1.9 noticeably help this performance? (I'm a bit nervous it'll create a ton of problems). Phil@Heroku Hello - I can look into this, but I'll need access to your New Relic account. Will you make sure 'phil@heroku.com' has access? Also, from your screenshot I notice your DB times are ~ 100 ms. We recommend keeping those times closer to 50 ms. You might be able to speed things up with a database upgrade. I'll look into New Relic once I have access and let you know what I find. Tom@Rapgenius Thanks, Phil! How do I give you access to my New Relic account? I tried clicking "account settings" and got this: http://cl.ly/0V2J3i0826400I2s3b2c http://cl.ly/0V2J3i0826400I2s3b2c Phil@Heroku Tom, I have access now. I'm not sure what was blocking me earlier. After looking at New Relic and the database server, I think a larger database will help. At the very least, it will be helpful to try the next level for a week and compare performance statistics in New Relic with the prior week. Your app is using an Ika right now, and the next step up is the Zilla database. We've made the upgrade process very simple, and it's outlined here - http://devcenter.heroku.com/articles/fast-database-changeovers?preview=1 http://devcenter.heroku.com/articles/fast-database-changeove... Your database is ~ 5.4 GB in size (via the 'heroku pg:info' command) so an upgrade shouldn't take too long. You will be able to test the process by adding a Follower and timing it via the 'heroku pg:wait' command. This should give you a good idea of how long it will take to spin up the new database. Also, should the Zilla not help much, the downgrade process to an Ika will be the same. You only pay for the resources used. The current database server appears to be a bit under-powered when it comes to Compute Units. The Zilla has more power and should provide some room to grow. As for an upgrade to Ruby 1.9.2, I'm not sure how much that would help. It would be an involved upgrade that would take time to plan and deploy. The database upgrade should be a quicker solution. Long-term you may want to consider moving to the Cedar stack and Ruby 1.9.2. Tom@Rapgenius Thanks! I'm upgrading now Tom@Rapgenius I'm still getting a ton of "Request Timeout" errors. E.g.: 2011-12-08 14:46:53.222 219 1 2011-12-08T14:46:53+00:00 d. heroku router - - Error H12 (Request timeout) -> GET rapgenius.com/Wale-ambition-lyrics dyno=web.17 queue= wait= service=30000ms status=503 bytes=0 one weird thing: there aren't any values listed for the "queue" and "wait" parameters. Could that indicate a problem? Could an exception have been thrown earlier in the request before the timeout? Or does the timeout error just indicate that the request took too long? If it's the latter I'm not sure how to troubleshoot all these errors since the associated actions are fast the vast majority of the time Tom@Rapgenius Here's another interesting example: 2011-12-08 15:59:32.293 222 1 2011-12-08T15:59:32+00:00 d. heroku router - - Error H12 (Request timeout) -> GET rapgenius.com/static/templates_for_js dyno=web.17 queue= wait= service=30000ms status=503 bytes=0 This action is extremely simple – it doesn't access the DB or any external services. Here's the template: Ballin! <% unless current_user %> <% form_for User.new, :html => { :id => '' } do |f| %> Tired of entering your email address? Create a Rap Genius account and you'll never have to worry about it (or anything else) ever again: <%= render :partial => "/users/form", :object => f %> <%= f.submit "Create Account" %> <small>(Already have an account? <%= link_to 'Sign in', login_path, :class => :facebox %>)</small> <% end %> <% end %> Besides a big request queue (which there isn't), how could this action possibly time out? Phil@Heroku Tom - sorry for not getting back to you sooner. It's possible for H12s to occur even for simple actions if there is already queueing for the app. With a busy site like your's, even a few H12s can cause a cascade of H12s for successive requests. It looks like New Relic has not reported any downtime over the past 24 hours. Can we let the site run through the weekend and see how things look Monday after 3 days of New Relic data with the new Zilla? Tom@Rapgenius > It's possible for H12s to occur even for simple actions if there is already queueing for the app. I feel what you're saying, but I don't think my app's queuing. For one thing, New Relic shows 0 time spent in the queue during the period in which I'm getting all these timeouts. For another, I'm running 40 dynos and my average request time is <400ms. So: 400 ms * 3000 requests / minute * 1 min / 60000 ms = 20 simultaneous requests (i.e., 20 dynos) so 40 dynos should definitely be more than enough. Also, shouldn't Heroku be showing me the queue / wait stats at the time of the timeout? That would help prove whether my app was queuing at the time in question It looks like New Relic has not reported any downtime over the past 24 hours. New Relic isn't great at catching intermittent problems like this; you really feel it when you're using the site continuously for an hour or whatever. Also, users make many more HTTP requests than New Relic (since every page load kicks off several AJAX requests). That said, there has been downtime in the past 24 hours (though less than in the previous 24): http://cl.ly/3D2b1Z170B0w1f1m113m http://cl.ly/3D2b1Z170B0w1f1m113m Tom@Rapgenius Here's some additional data: At 5am this morning (EST), Rap Genius went down. I woke up at 11am (it's a Saturday!), did a logs --tail and observed that basically every request was timing out. I did heroku restart, and now every request started returning a backlog too deep error Finally, I added another 10 dynos (bumping the total to 50, which is a log of dynos!), and this seems to have fixed the problem – perhaps because my app needs the additional capacity, or perhaps because merely changing the number of dynos reset something else. Either way, I'm sticking with 50 dynos for now out of fear even though I doubt my app needs that many (right?) Either way, the 5 hours of unexplained downtime (there weren't any application-level exceptions or anything) that was fixable by tweaking my dyno count further supports my theory that something's going on with my app on Heroku's end. Phil@Rapgenius Tom - I've been looking over your New Relic stats. First - the good news - the upgrade to a Zilla seems to have helped. Database times are down a bit, which can only help. I checked the actual database server and it's not showing signs of over-work like the previous Ika was. Second, I notice that downtimes reported by New Relic over the past two weeks are in the early morning hours - 3 to 6 AM PST. Do you have any scheduled tasks that run during these times? Also, request queueing is nearly zero, so 50 dynos does seem like a lot. What are your usage patterns like? The RPM graph in New Relic indicates the normal cyclical usage pattern, lower during the night, but what does Google Analytics tell you? Finally, the Heroku platform has been having issues over the past week, but none of them correspond to the downtime you had Saturday morning.

pootch 14y ago

This was a great explanation of how Heroku routing works. And a terrible explanation of why anyone would use a service with such a brain dead router. Seriously is this company run by 16 year olds? Consider your allowance revoked. Rookies.

orensol 14y ago

I think the discussion comes down to which option you ultimately prefer -- 1) be in full control of your stack, having only yourself to criticize, and having to deal with all the devops action that comes with it, or 2) let heroku deal with the underlying while you focus on your app, as their promise suggests, and suffer (also financially) from time to time due to faults like this. Everyone has their own reasoning on this decision, and that's ok. I think that the voices here saying "I won't start my project on heroku because of this" are not thinking this through. I think that for any project starting up, focusing on app rather than infrastructure is much more important than routing issues.

azarias 14y ago

It seems to me that Heroku has chosen to be dishonest: Heroku's blog response: "but until this week, we failed to see a common thread among these reports." vs. Adam's response to Tim Watson, a year ago: "You're correct, the routing mesh does not behave in quite the way described by the docs. We're working on evolving away from the global backlog concept in order to provide better support for different concurrency models, and the docs are no longer accurate. The current behavior is not ideal, but we're on our way to a new model which we'll document fully once it's done." https://groups.google.com/forum/?fromgroups=#!msg/heroku/8eOosLC5nrw/Xy2j7GapebIJ https://groups.google.com/forum/?fromgroups=#!msg/heroku/8eO...

btilly 14y ago
It is quite possible to know that the mesh is not exactly as documented without having realized that the difference has a severe performance impact.
- azarias 14y ago
  I think they also addressed the performance issue in addition to saying it is not documented: "...evolving away from the global backlog concept in order to provide better support for different concurrency models"
  - btilly 14y ago
    The fact that a year later, performance for their starting use case had not been addressed says that they failed to address the performance issue. It is still not clear to me that they actually really understood the performance issue, or how big it was going to be.

jlouis 14y ago

So to recap: Ruby on Rails is using a default configuration where each process can serve one request at a time. There is no cooperative switch (as in Node.js) or (near) preemptive switch (as in Erlang, Haskell, Go, ...). The routing infrastructure at Heroku is distributed. There are several routers and one router will queue at most one message per back-end dyno in the Bamboo stack and route randomly in the Cedar stack. If two front-end routers route messages to the same Dyno, then you get a queue, which happens more often on a large router mesh. Forgetting who is right and wrong, there are a couple of points to make in my opinion. The RoR model is very weak. You need to handle more than one connection concurrently, because under high load queueing will eventually happen. If one expensive request goes into the queue, then everyone further down the queue waits. In a more modern system like Node.js you can manually break up the expensive request and thus give service to other requests in the queue while the back-end works on the expensive req. In stronger models, Haskell, Go and Erlang, this break-up is usually automatic and preemption makes sure it is not a problem. If you have a 5000ms job A and 10 50ms jobs, then after 1ms, the A job will be preempted and then the 50ms jobs will get service. Thus an expensive job doesn't clog the queue. Random queueing in these models are often a very sensible choice. Note that Heroku is doing distributed routing. Thus the statistical model Rapgenius has made is wrong. One, requests does not arrive in a Poisson process. Usually one page load gives rise to several other calls to the back-end and this makes the requests dependent on each other. Two, there is not a single queue and router but multiple such. This means: * You need to take care of state between the queues - if they are to share information. This has overhead. Often considerable overhead. * You need to take care of failures of queues dynamically. A singular queue is easy to handle, but it also imposes a single point of failure and is a performance bottleneck. * You have very little knowledge of what kind of system is handling requests. Three, nobody is discussing how to handle the overload situation. What if your dynos can take 2000 req/s but the current arrival rate is 3000, if you forget about routing for a moment. How do you choose to drop requests, because you will have to do so. If you want to solve this going forward, you probably need Dyno queue feedback. Rapgenius uses the length of the queue in their test, but this is also wrong. They should use the sojourn time spent in the queue which is an indicator for how long you wait in the queue before being given service. According to rapgenius, they have a distribution where requests usually take 46ms (median) but the maximum is above 2000ms. I can roughly have a queue length of 43 and 1 have the same sojourn time then. Given this, you can feed back to the routers about how long a process will usually stay in queue. But again, this is without assuming distribution of the routers. The problem is way way harder to solve in that case. (edit for clarity in bullet list)

antirez 14y ago

Come on startups, you should be technically skilled and able to optimize in order to spend little money. If you sum EC2 and Heroku you are going to pay like 10x what it takes to run the same machines power in a dedicated server, all this because you can't handle the operations? This is absurd IMHO. Also people that want to start a business, there is a huge opportunity here, create software that makes managing Apache, Redis, PostgreSQL, ..., in dedicated servers very easy and robust. Traget a popular and robust non-commercial distribution like Ubuntu LTE, and provide all is needed to deploy web nodes, database nodes, with backups, monitoring, and everything else trivial. Startups can give you 5% of what they are giving now to EC2 and Heroku and you still will make a lot of money. "I can only write my Ruby code but can't handle operations" is not the right attitude, grow up. (This is what, in a different context, a famous security researcher told me in a private mail 15 years ago, and it was the right advice)

Adirael 14y ago
If your Ruby code deploys to X you should know X, if it's part of your stack it should be part of your toolbox. "create software that makes managing Apache, Redis, PostgreSQL" Like Plesk, but better, faster and less resource hungry.
ig1 14y ago
The primary bottleneck for most startups isn't money, it's time. Also for most startups (assuming you're not doing something CPU/bandwidth heavy) the actual cost of hosting is going to be a relatively small part of the budget. If your burn rate is a million dollars, reducing 10k of hosting costs to 1k isn't really worth the effort.
- fauigerzigerk 14y ago
  Money can buy time in various forms. I think PaaS is one abstraction level too much to remain flexible and efficient as you grow.
- raverbashing 14y ago
  Kind of depends There's the "two guys in a dorm eating ramen" and there's the YC backed "startup" that already has a couple of hires So if your monthly budget is of hundreds of dollars, using a VPS instead of Heroku makes sense

programminggeek 14y ago

I feel like Heroku is punting on this one in a big way. I think this really means that for a small app, Heroku is fine, once it gets bigger, you probably want to get a real scaling strategy.

cygwin98 14y ago
Agree. Though that's a bit ironic --- one of Heroku's selling points was scalability, at least a big part of Rails community believed it to be.

steveklabnik 14y ago

> On the Cedar stack, the root cause is the fact that Cedar is optimized for concurrent request routing, while some frameworks, like Rails, are not concurrent in their default configurations. First of all, Ruby doesn't exactly block on I/O [1], so there's that. Secondly, while Rails 3 has config.threadsafe! turned off, Rails 4 will have it turned on. [2] 1: https://github.com/tenderlove/fibur#synopsis https://github.com/tenderlove/fibur#synopsis 2: http://tenderlovemaking.com/2012/06/18/removing-config-threadsafe.html http://tenderlovemaking.com/2012/06/18/removing-config-threa... EDIT: I just realized non-Rubyists might not get the Fibur joke. Here's the code: https://github.com/tenderlove/fibur/blob/master/lib/fibur.rb#L1 https://github.com/tenderlove/fibur/blob/master/lib/fibur.rb...

meritt 14y ago

People pay $35/mo for a Heroku Dyno that can only handle 1 request at a time? That's absurd.

badgar 14y ago

So the tl;dr from Heroku is: we can't scale single-threaded applications.

jotnar 14y ago

They should of just used nodejs for routing.

jotnar 14y ago

You know, I wrote a really fast HTTP router using Redis pub/sub and a pool of single threaded nodejs processes running on seperate EC2 boxes and it worked great. I don't know why it is so hard to scale RoR when it is single-threaded.

sneak 14y ago

This guy's writing style and tone make me want to never do business with him.

hpguy 14y ago

Don't know about Rails, but I'm building a MVP using Node.JS and hosting a free dyno. Can handle 250 reqs/s for dynamic requests (i.e. full stack, Postgres, Redis, Jade, Express etc.) and 1500+ reqs/s for static contents. I do use cluster with 3 workers though. Pretty happy with the result so far.