15 ms·

Answering your questions about Heroku routing and web performance

205 points by adamwiggins 14y ago

scottshea 14y ago
So their solution to the random routing is for the customer to switch to Unicorn/Puma on JRuby. Wow.
- adamwiggins 14y ago
  Yes, because that is the solution. Empirically. We've run many experiments over the past month to try other approaches to routing, including recreating the exact layout of the Bamboo routing layer (which would never scale to where we are today, but just as a point of reference). None have produced results that are anywhere near as good as using a concurrent backend. (I'd love to publish some of these results so that you don't have to take my word for it.) That said, we're not done. There are changes to the router behavior that could have an additive effect with apps running Unicorn/Puma/etc, and we'll continue to look into those. But concurrent backends are a solution that is ready and fully-baked today.
  - habosa 14y ago
    Please publish these results. I think a chart showing that Unicorn + Random routing is better than Thin + Intelligent routing would go a long way to ending this whole thing. That's assuming that you can make deploying a Unicorn app as easy as it was with Thin ('git push heroku')
    - scottshea 14y ago
      Having been part of their efforts to test 2X, and 4X Dynos and using Unicorn long before the RapGenius issue I can tell you the added memory and Unicorn still has issues. We still see periods where queueing is above 500ms. The additional Dyno capacity distributes the chance of queue issues out over a larger numerical set but there is still the possibility of one dyno/Unicorn worker combo getting too much to handle. We use Unicorn Worker Killer to help in that case.
    - scottshea 14y ago
      I should add that using Unicorn Worker Killer to end a process on thresholds is not a solution as much as it is another stopgap.
    - adamwiggins 14y ago
      We might. But what does this actually get us? It helps clear Heroku's name, but it doesn't help our customers at all. I'd prefer to spend our time and energy making customer's lives better. Given the choice between continuing the theoretical debate over routing algorithms vs working on real customer problems (like the H12 visibility problem mentioned elsewhere in this thread), I much prefer the latter.
      - habosa 14y ago
        I respect that mindset, I just don't think it would hurt. Maybe a middle ground would be a full-scale tutorial on how to switch from Thin on Bamboo/Cedar to Unicorn on Cedar for Rails users. It's a non-trivial process and I know I'd like some help with it. And in this same tutorial/article you could throw down the benchmarks you ran as motivation/justification.
  - boundlessdreamz 14y ago
    Unless the dyno signals to the router that it is busy, isn't this just postponing the problem? Per dyno, unicorn can handle more requests but requests will still get queued at dyno level if one of the requests is a slow one (say 2-3 seconds)
    - thinkbohemian 14y ago
      If only 1 request is slow and you have 7-8 unicorn workers, only one of them will stay busy. Unicorn knows which of it's workers are busy and does not queue jobs behind individual worker but rather behind the master who delegates the request to the first available worker.
      - adamwiggins 14y ago
        Precisely. As mentioned in the FAQ, putting the queueing logic closer to the process which is ultimately going to serve the request is a more horizontal-scale friendly way of tackling the queueing problem. It works fantastically well for backends that can support 20+ concurrent connections, e.g. Node.js, Twisted, JVM threading, etc. It works less well as you can put fewer connections in each backend, which is part of why we're working on larger dynos.
  - tjb 14y ago
    We also adopted Unicorn pretty early on but still suffered issues as dynos simply ran out of memory. In fact with some apps we have seen improvements (but still far from acceptable) in stability by un-adopting this method. The issues raised by scottshea below, as a consequence concern me, what/will be the charge for these as well? To be honest it's the fumbling around in the dark that has annoyed me. I am with you 100% on your manifesto and your points about the type of service you provide. However the time we have spent on this (starting before you came clean about the issues) and the time spent on other increasingly suspicious advice to "up dynos" or spend time "optimising your app" sours this slightly. I accept the "magic black box" comes with its compromises and required understanding at our end but it also means needing to be far more communicative and honest about it at yours. Something which you are putting right I can see. I for one think the premise of Heroku is a great one and you have succeeded for us in many of the things you have set out to achieve. This whole situation has been a real shame, I'm sure this must have a been a pretty shody time for you guys and I hope you come out the better for it. The quicker the better to be honest so you can focus on the new features we'd like to see.
    - adamwiggins 14y ago
      Thanks for your support. Indeed, communication and transparency into how the product works as far as how it affects your app are two things we'd like to get better at. Regarding your app: indeed, Unicorn is a huge improvement, but far from the end of the story. "Performance" is like "security" or "uptime" — it's not a one-time feature, something you check off a list and move on. It's something that requires constant work, and every time you fix one problem or bottleneck that just leads you to the next one. Over time, though, your vigilance pays off with a service that its users deem to be fast or secure or have good uptime. Yet there's no such thing as a finish line on these. Bringing it back to details. Kazuki from Treasure Data made this Unicorn worker killer that might help you: https://github.com/kzk/unicorn-worker-killer https://github.com/kzk/unicorn-worker-killer If you're still not happy with your app's performance, give me a shout at adam at heroku dot com and we'll see if we can help.
hglaser 14y ago
This is a great response. I'm curious why something like this wasn't posted within 24 hours of RapGenius going public? I'd bet that a more thorough, technical reply would have mitigated a lot of the PR issues.
- steveklabnik 14y ago
  When admitting fault in response to a customer that spends 5 figures/month with you, wise to have a lawyer go over it first.
  - hglaser 14y ago
    I would argue that the brand risk -- e.g. the risk of what ended up happening -- outweighs the legal risk. That's not to say I wouldn't send it to a lawyer. But I'd do it with some hustle. Something like: "This is going out in 24 hours. Please comment ASAP!"
    - res0nat0r 14y ago
      As Adam mentioned in so many words: most of the people pissed off about all of this are most likely right here on HN, and not even customers of Heroku. He mentioned most customers were not that mad about the issue overall.
- adamwiggins 14y ago
  Agreed, I wish we could have done it much sooner. It took a shocking amount of time to sort through all the entangled issues, emotion, and speculation to try to get to the heart of the matter, which ultimately was about web performance and visibility. Also, we wanted to respond to our customers first and foremost, and general community discussion second. So we spent close to a month on skype/hangout/phone with hundreds of customers understanding how and at what magnitude this affected their apps. That was hugely time-consuming, but it gave us the confidence to speak about this in a manner that would be focused on customer needs instead of purely answering community speculation and theoretical discussions.
ebbv 14y ago
It seems to me that Heroku is still failing to understand (or at least cop to) the fact that the switch from intelligent to randomized routing was a loss of a major reason people chose Heroku in the first place. A lot of Heroku's apparent value came from the intelligent routing feature. Everybody knew that it was harder to implement than random routing, that's why they were willing to pay Heroku for it. Nobody's arguing random routing isn't easier and more stable; of course it is. The problem is that by switching over to it, Heroku gave up a major selling point of their platform. Are they really blind enough not to know this? I have a hard time believing that. It seems to me the real way to make people happy is to discount the "base" products which come with random routing and make intelligent routing available as a premium feature. Of course, people who thought they were getting intelligent routing should be credited.
- tptacek 14y ago
  Is anybody operating at Heroku's scale offering centralized request routing queues? At what price?
  - ebbv 14y ago
    Not that I know of but that's why I'm saying it would be a premium product. Likely pricing would have to scale with the number of dynos running behind the router. But that's the service people thought they were getting and what they wanted. If Heroku prices out the intelligent routing and says; "Ok you can have intelligent routing with your current backend stack, but it's going to cost you $25/mo for evert 10 dynos, or you can switch your stack and use randomized routing for free." Then they are empowering their customers to make the choice rather than dictating to them what they should do.
    - chc 14y ago
      $25 a month? Did you forget a few zeroes?
      - ebbv 14y ago
streptomycin 14y ago
He forgot to explain why they won't be refunding customers who were defrauded.
- res0nat0r 14y ago
  Because they weren't victims of fraud.
  - dangrossman 14y ago
    Is promising one service but delivering another not fraud? Promises from the Heroku website pre-Rap Genius posts: "Incoming web traffic is automatically routed to web dynos, with intelligent distribution of load instantly" "Intelligent routing: The routing mesh tracks the availability of each dyno and balances load accordingly. Requests are routed to a dyno only once it becomes available. If a dyno is tied up due to a long-running request, the request is routed to another dyno instead of piling up on the unavailable dyno’s backlog." "Intelligent routing: The routing mesh tracks the location of all dynos running web processes (web dynos) and routes HTTP traffic to them accordingly." "the routing mesh will never serve more than a single request to a dyno at a time" Actual service provided: requests are routed randomly to dynos regardless of how many requests they are currently handling or their current load.
    - streptomycin 14y ago
      Also the misleading performance metrics which hid the fraud.
    - res0nat0r 14y ago
      Your definition of fraud differs from most: "Wrongful or criminal deception intended to result in financial or personal gain." They weren't intentionally and purposefully misleading people. Not having docs up to date on your website, or you not knowing how the underlying backend works is not fraud. As I've mentioned before if every AWS customer could sue Amazon for not understanding how all of the underlying tech worked, or could sue when some of the docs were out of date, there would be more lawyers working there than engineers.
tvladeck 14y ago
All of the stuff that Heroku are doing now to mitigate the routing issues for their Bamboo customers are things they should have done when it first became an issue. They are not going above and beyond in any way to make up for the lost time and money their customers. Again, this is not about -how advanced Heroku's technology is on an absolute level -how challenging routing is for their scale -what competitors offer in this space and for what prices This is only about the delta between what Heroku sold their customers and what their customers received. They are collapsing the delta now, by being honest about what they are selling (and improving their offering, it seems), but they are doing nothing to address the long time for which the delta was significant for a subset of their customers.
- chc 14y ago
  Your complaints are all about the past. What are you looking for here? For Heroku to invent a time machine?
  - ambrice 14y ago
    How about refunds? Isn't that the normal response when a company screws up in a way that causes the customers to pay more money?
    - adamwiggins 14y ago
      Yes, we've given credits (or a refund, at the customer's preference) in cases where lack of visibility or inaccurate docs led to over-provisioning of dynos. There's actually very few cases where people paid more money than they would have otherwise. Heroku is a service with your bill prorated to the second. For the most part, if people don't like the performance (which is measurable externally via benchmarks and browser inspectors), they leave the service. Many people who hit problems with visibility and performance did exactly that. Naturally, we'll be working hard to try to recapture their business, as well as to remove any reasons that existing customers might leave as they hit performance or visibility problems scaling up in the future.
badgar 14y ago
> 1k req/min Also known as <17 requests per second... or a trickle of traffic. Hooray for using bigger numbers and a nonstandard unit to hide inadequacy! Does Heroku use req/min throughout their service? I can't understand why they would, unless they also can't build the infrastructure to measure on a per-second basis. > After extensive research and experimentation, we have yet to find either a theoretical model or a practical implementation that beats the simplicity and robustness of random routing to web backends that can support multiple concurrent connections. Does this CTO think companies like Google and Amazon route their HTTP traffic randomly? No... he knows there are scaleable routing solutions and random routing isn't the best. So he cites "simplicity and robustness." Here, this means "we can't be bothered."
- adamwiggins 14y ago
  Re: requests. RPM is the metric that New Relic reports, and it's the one most of our customers use when they talk about traffic. I try to speak in whatever terms are most familiar to our customers. Re: I can't speak to Google and Amazon, and they aren't representative of the size of our customers anyway. We have discussed with many folks who run ops at many companies that are more on par with the size of our mid- and large-sized customers, and single global request queues are exceedingly rare. The most common setup seems to be clusters of web backends (say, 4 clusters of 40 processes each) that each have their own request queue, but with random load balancing across those clusters. This is a reasonable happy medium between pure random and global request queue, and isn't too different from what you get running (say) 16 processes inside a 2X dyno and 8 web dynos.
- meritt 14y ago
  I too was shocked at that. Also that 6 dynos is apparently the average size to handle that load. It takes $179/mo (6 dynos) to handle 17 requests/second? That's insane.
  - adamwiggins 14y ago
    Didn't intend to imply that. Number of dynos needed varies extremely widely, with the app's response time and language/framework being used as the main variables. There are apps on Heroku that serve 30k–50k reqs/min on 10–20 dynos, typically written in something like Scala/Akka or Node.js and serving incredibly short (~30ms) response times with very little variation. But these are unusual. The more common case of a website, written in non-threadsafe Rails, with median response times of ~200ms but 95th percentile at 3+ seconds, would probably use those same 10 dynos to do only a few thousand requests per minute. Whether or not you use a CDN and page caching also makes a big difference (see Urban Dictionary for an example that does it well). But it really depends. We were trying to quantify when you should be worried. If you're running a blog that serves 600 rpm / 10 reqs/sec off of two dynos, you don't need to sweat it.
bhauer 14y ago
I am surprised they do not moderate the comments on their blog. They have one visible presently that is plain offensive.
cmelbye 14y ago
They still fail to understand that using Unicorn doesn't magically fix this issue. Like, at all. It simply means that the dunk gets tied up when n+1 requests (n is number of unicorn workers) get randomly routed to it instead of just 1. It's in no way comparable to a node.js server that handles thousands of concurrent requests asynchronously. They're simply two different designs, and Ruby's traditional design is fundamentally incompatible with Heroku's router.
- mattsoldo 14y ago
  That's not right. In any configuration the goal is to minimize queue time. What is critical to doing this is having a request queue and a pool of concurrent "workers" (to use the generic queueing theory term) in back of it. Unicorn uses the operating system's TCP connection queue to queue incoming requests that it is not able to immediately server. While n+1 requests can (and will) get routed to a single dyno, this only results in 1 request being queued. It will be queued until the first of the in-process requests is served, which will take ~ the average response time for the app. Given that the other n requests did not get queued (queue time = 0), the average queue time will equal Sum(queue times) / # requests = Average Response Time / n+1.
persei8 14y ago
Is it true that you don't fully buffer body of POST request on router? This limitation works against Unicorn design and makes it hardly a "fix": http://rubyforge.org/pipermail/mongrel-unicorn/2013-April/001743.html http://rubyforge.org/pipermail/mongrel-unicorn/2013-April/00...
- geekylucas 14y ago
  Heroku only buffers the request headers: https://devcenter.heroku.com/articles/http-routing#request-buffering https://devcenter.heroku.com/articles/http-routing#request-b...
- adamwiggins 14y ago
  Plenty of room for improvement here, sure. Some technical discussion in this thread: https://blog.heroku.com/archives/2013/4/3/routing_and_web_performance_on_heroku_a_faq#comment-850974645 https://blog.heroku.com/archives/2013/4/3/routing_and_web_pe...
bradleyjg 14y ago
Random routing to concurrent servers works fairly well if the kind of long running requests you need to worry about spend a lot of time blocking for some external service (e.g. a database call). Then you can get a lot of benefit on the from cooperative or preemptive multitasking on the server, and so the performance characteristics, from the point of view of a new request, of each server is roughly the same, and so going to a random server is pretty good. However, if you have long running requests because they make intensive use of server resources (CPU, RAM) then concurrent servers buys you very little. In that case, sending a new request to a server that is chugging along on a difficult problem is significantly different than sending it to one that isn't. That's where knowing the resource state of each server, and routing accordingly is of huge benefit. While load balancing is a very difficult problem, with some counterintuitive aspects, it is an area of active research, and there are some very clever algorithms out there. For example, this article (http://research.microsoft.com/pubs/153348/idleq.pdf http://research.microsoft.com/pubs/153348/idleq.pdf) from UIUC and Microsoft introduces the Join-Idle-Queue, which is suitable for distributed load balancers, has considerably lower communication overhead than Shortest Queue (AFAICT the original 'intelligent' routing algorithm), and compares its charateristics to both SQ and random routing.
bifrost 14y ago
I'd say there's a major flaw here - "the new world of concurrent web backends" - If now was 1995 I might agree with you, but having concurrency in web-app servers is not new. The lack of concurrency in test/demo app situations is totally understandable, in a production environment you'd have to be completely bonkers to think thats ever useful. I also agree that "having to deal with loadbalancing" is something that most people don't get and shouldn't really have to get, but when the way you do loadbalancing is so fundamentally flawed that its worse than Round Robin DNS, you also clearly don't understand it. To be fair, I have to say I meet people all the time who don't understand it at all and think $randomcrappything is great at loadbalancing. If "connections go to more than one box!" is your metric, then yes, thats loadbalancing. My metric is "Do you send connections to servers that are responsive and not overloaded, maybe with session affinity" and in general most HW loadbalancer products since 1998 have supported that. So if you're not better than 1998 technology, you may want to reevaluate your solution.
- bgentry 14y ago
  I'd say there's a major flaw here - "the new world of concurrent web backends" - If now was 1995 I might agree with you, but having concurrency in web-app servers is not new. Sadly, concurrency is relatively new and unfamiliar territory to many in the Ruby on Rails community.
derengel 14y ago
Heroku's message: "If you have slow clients you are screwed" Unicorn is designed to only serve fast clients on low-latency. And no, they don't do any buffering.
- adamwiggins 14y ago
  I think that overstates it a bit, but yes, there are problems with Unicorn and slow clients. We're investigating: https://blog.heroku.com/archives/2013/4/3/routing_and_web_performance_on_heroku_a_faq#comment-850974645 https://blog.heroku.com/archives/2013/4/3/routing_and_web_pe... If this is an immediate problem for you, it might be worth your while to make your app threadsafe, which gives you more concurrent webserver options.
thatthatis 14y ago
What data do you have to show that the random selection algo has superior performance to a round-robin algo?
- adamwiggins 14y ago
  We investigated round-robin. With N routing (or load balancer) nodes, and any degree of request variance, round robin effectively becomes random very quickly.
deedubaya 14y ago
Haters gonna hate. I wonder how many people bitching here are actual customers who are having problems that haven't been address with a solution. I'm guessing that number is low. Oh, you're a potential customer? That's why you're bitching? About a problem you may or may not have if you actually choose the product? Think about that argument for a second. I've never seen such a transparent response and follow up as I have from Heroku on this issue. Most other companies would have gone into immediate damage mitigation mode and let the wound heal instead of re-opening it and giving feedback on how to fix the problem as Heroku has. I applaud the Heroku team for their effort on their platform and being a kickass company.
- stickfigure 14y ago
  I'm a real customer with real problems. Grep this page for latchkey's description of it. The funny thing is, I don't have much sympathy for Rails users. Scaling problems with a single-threaded, serial request-processing architecture? No surprise there. But we have inexplicable H12 problems with Node.js. There's something broken in the system and it isn't random routing.
  - Cabal 14y ago
    You're talking like Node offers real concurrency. It doesn't.
    - stickfigure 14y ago
      There's nothing wrong with Node's concurrency. Our app, like most webapps, is I/O bound. Any individual instance should be able to handle thousands of concurrent requests as long as they are all blocked on I/O. Being able to process more than one concurrent request (as Node can) is "real concurrency". Java-style native threading is a step above and beyond this, and unnecessary for most web applications.
latchkey 14y ago
I am working for a fairly large heroku app running Node on ~50-100 web dyno's with another 20-50 backends. Here are the problems as I see it: We get H12's all the time. Randomly. The only suggestion we get from Heroku is to make the requests process as fast as possible. Thus, we've spent considerable amount of time going through everything we can possibly do to make all requests respond as fast as possible. I've given up. I see this as a fundamental issue with the routing system. If you are going to use Heroku for a large production deployment, H12's (and your users getting dropped connections) will become a fact of life. There is no auto scaling. We have no idea how many dyno's we actually need. So, we over do it in order to handle peek traffic times. This must be a great money maker for Heroku. There is no incentive for them to build auto scaling into their system because that would mean they wouldn't make as much money. Yes, auto scaling is a hard problem to solve, but there should at least be a plan to start on it and there is none that I have found. Up until someone bitched loudly, nothing was happening to fix any of this. We have an expensive paid support contract with Heroku and before this whole routing issue blew up in public, their only recommendation was to tune the app more and buy into NewRelic for ~$8k / month. We did both and found NewRelic to not give any relevant information to help us. We did a NodeTime trial for ~$49/mo and that actually helped a lot in identifying slow spots in our app. We fixed all the slow spots in our app and still see an endless stream of H12's. Regardless, it shouldn't take a public bitch slapping for a company to listen to their customers. You log into a dyno and see a load average of 30+. Who knows if that number is accurate or how big the underlying box really is, but regardless, I can't imagine that number being good. Am I getting H12's because I'm on an overloaded box or is it because the routing system is fundamentally broken? I don't know and nobody can tell me. This is not a good position to be in. I have heard from several sources that Heroku isn't happy being on AWS and has been wanting to migrate off AWS for a while now. So, if your hosting provider isn't happy on their hosting provider, there must be a reason for that and in the bigger picture, you the customer, is getting screwed. Given these things, I will never recommend that a company use Heroku. It is great if you know you are going to never have more than one dyno, but if you think you are going to go into a large production system with it, it is far better to find something else. Which brings me to another rant... how come none of these other PaaS solutions are as easy as Heroku? The git deploy is seriously the one thing they got mostly right. I'd love to see someone build a layer on top of all the PaaS solutions so that I can just deploy my code to any one of them (or event multiple).
ROFISH 14y ago
Something that's been bugging me about Heroku is that the dyno price has stayed the same ever since launch: $0.05 per hour. Compare to services like Digital Ocean and AWS (who lowered prices significantly in the past few years), Heroku is starting to get very expensive. The 2X dyno at 2x cost doesn't really make me happy, it just invites me to spend more money when it would be more cost-efficient to move.
cjackson27 14y ago
I can't speak to the issues that people are running into when they reach large scale, but I run a small app with two dynos and we've been having issues with H12 request timeout errors for weeks now. This has been bringing down our production app for periods of about fifteen minutes almost daily. I've been completely disappointed with Heroku's support so far. First they obviously skimmed my support request and provided a canned response that was completely off base. Their next response didn't come for four days and only after I called their sales team to see what I could do to get better support. Their only option is a $1k / mo support contract. If you're running a mission critical app, I'd think twice before choosing Heroku.
- adamwiggins 14y ago
  The difficulty of diagnosing H12 errors is really challenging. One thing I can recommend is using the http-request-id labs feature: https://devcenter.heroku.com/articles/http-request-id https://devcenter.heroku.com/articles/http-request-id With this enabled and some code in your app, you can correlate your app's request logs directly against the router logs and trace what happened with any particular H12. I'd be happy to help you do this if you're game. Contact me via adam at heroku dot com. Could you also email me some links to your support tickets so I can check out what happened there?
jholman 14y ago
I'd like to start by acknowledging that I'm one of the "non-customers who are watching from the sidelines". I think Adam's right that this is an important distinction. Adam, there's something that confuses me about this. I'm no expert in routing theory, nor have I done the experiments, so forgive me if my reasoning misses something. I understand why RapGenius took you up on your original promises of "intelligent routing", and I think I understand what you're saying about scaling, and how scaling "intelligent routing" is so far unsolvable, and the motivation for your transition from Bamboo to Cedar, especially in the context of concurrent clients. What I don't understand is this: It seems to me that if you split into two (or more) tiers, and random-load-balance in the front tier (hit first by the customer), and then at the second tier only send requests to unloaded clients, that you eliminate RapGenius's problem for customers who followed your specific recommendations for good performance on Bamboo (to go single-threaded and trust the router). Do you have reason to believe that this doesn't one-shot RapGenius's problem? Do you have strategic/architectural reasons for rejecting this even though it would work? Did you try it and it failed? What's the story there? Maybe I'll write a simulator to (dis)prove my naive theory. :P
- adamwiggins 14y ago
  > It seems to me that if you split into two (or more) tiers, and random-load-balance in the front tier (hit first by the customer), and then at the second tier only send requests to unloaded clients [...] I'm unclear how you'd think introducing a second tier changes things. That tier would need to track dyno availability and then you're right back to the same distributed state problem. Perhaps you mean if the second tier was smaller, or even a single node? In that case, yes, we did try a variation of that. It had some benefits but also some downsides, one being that the extra network hop added latency overhead. We're continuing to explore this and variations of it, but so far we have no evidence that it would provide a major short-term benefit for RG or anyone else. > Do you have reason to believe that this doesn't one-shot RapGenius's problem? As a rule of thumb, I find it's best to avoid one-shots (or "specials"). It's appealing in the short term, but in the medium and long term it creates huge technical debt and almost always results in an upset customer. Products made for, and used by, many people have a level of polish and reliability that will never be matched by one-offs. So if we're going to invest a bunch of energy into trying to solve one (or a handful) of customer's problems, a better investment is to get those customers onto the most recent product, and using all the best practices (e.g. concurrent backend, CDN, asset compilation at build time). That's a more sustainable and long-term solution.
deleted 14y ago
[deleted]
siong1987 14y ago
Since we are on the question of visibility of Heroku dyno, what is the amount of CPU power each dyno has? What about 2Xdyno?
- adamwiggins 14y ago
  This is a tough area. If you go look at various types of infrastructure providers (e.g. EC2, Linode, Rackspace) you'll see that they always end up making up vague units of measurement (e.g. "cores") and then showing all the resources in reference to whatever the base unit is. So there's really no good way to talk about CPU power like there is with memory. That said, I can say that a 1X dyno is not very powerful compared to, say, any server you'd purchase for your own datacenter. Our intention is that 2X dynos will provide twice the CPU horsepower, although CPU and I/O are harder to allocate reliability in virtualized environments.
jordanthoms 14y ago
My biggest issue with heroku is the general slowness with the API - maybe I'm just impatient, but most simple commands like listing releases, viewing logs etc take at least a second, sometimes five before anything happens. Pushes also take quite a while, even the Git push part is much slower than pushing to github. It's just a general sluggishness which gets annoying after a while. If they could get all the API requests down under 500ms I'd be much happier.
- adamwiggins 14y ago
  Yeah, the developer-facing control surfaces on the platform (API calls and git push mainly) have gotten slower over the past year or two. This is on my list of of personal pet peeves, but so far has not made it onto our list of priorities. We try to drive priorities based on what customers want, not what we want: and what we've heard in the last year or so is all about app uptime, security, and now performance and visibility. I'm very much hoping that bringing back "fast is a feature" on the developer-facing portions of the product is something we can work on this year.
j-kidd 14y ago
From the article: > Q. Did the Bamboo router degrade? > A. Yes. Our older router was built and designed during the early years of Heroku to support the Aspen and later the Bamboo stack. These stacks did not support concurrent backends, and thus the router was designed with a per-app global request queue. This worked as designed originally, but then degraded slowly over the course of the next two years. From Adam's message on Feb 17th, 2011 (https://groups.google.com/forum/?fromgroups=#!topic/heroku/8eOosLC5nrw https://groups.google.com/forum/?fromgroups=#!topic/heroku/8...): > You're correct, the routing mesh does not behave in quite the way described by the docs. We're working on evolving away from the global backlog concept in order to provide better support for different concurrency models, and the docs are no longer accurate. The current behavior is not ideal, but we're on our way to a new model which we'll document fully once it's done. It looks like random load balancing was already the expected behavior 2 years ago? The "slow degradation" part seems a bit dishonest to me.
- adamwiggins 14y ago
  There are two separate issues here, and it's easy to get them confused. One is the slow degradation on Bamboo without any change to the routing algorithm code, and the other was the explicit product choice for Cedar with a different code path in the router. Both are described fully here: https://blog.heroku.com/archives/2013/2/16/routing_performance_update https://blog.heroku.com/archives/2013/2/16/routing_performan... The reason it's easy to confuse these two is also part of what confused us at the time. The slow degradation of the Bamboo routing behavior was causing it to gradually become more and more like the explicit choice we had made for our new product. But of course it's up to you (and everyone else observing) to judge whether this was some kind of malicious intent to mislead, versus that we made a series of oversights that added up to some serious problems for our customers. And that we are now doing everything in our power to be fully transparent about, to rectify, and to make sure never happen again.
  - deleted 14y ago
    [deleted]
  - j-kidd 14y ago
randall 14y ago
I think the most annoying thing is they still don't answer Rap Genius's questions about being owed money for paying megabucks for newrelic. I mean If you offer a service that provides incorrect data for two years and you don't offer any sort of framework for reimbursement, that still seems, at best annoying, at worst, dishonest.
- adamwiggins 14y ago
  We spent quite a bit of time trying to find a one-size-fits-all framework. There just wasn't one, so we've done credits on a case-by-case basis. Sorry you find it annoying. It's what was best for our customers.

205 points by adamwiggins 14y ago

scottshea 14y ago

So their solution to the random routing is for the customer to switch to Unicorn/Puma on JRuby. Wow.

adamwiggins 14y ago
Yes, because that is the solution. Empirically. We've run many experiments over the past month to try other approaches to routing, including recreating the exact layout of the Bamboo routing layer (which would never scale to where we are today, but just as a point of reference). None have produced results that are anywhere near as good as using a concurrent backend. (I'd love to publish some of these results so that you don't have to take my word for it.) That said, we're not done. There are changes to the router behavior that could have an additive effect with apps running Unicorn/Puma/etc, and we'll continue to look into those. But concurrent backends are a solution that is ready and fully-baked today.
- habosa 14y ago
  Please publish these results. I think a chart showing that Unicorn + Random routing is better than Thin + Intelligent routing would go a long way to ending this whole thing. That's assuming that you can make deploying a Unicorn app as easy as it was with Thin ('git push heroku')
  - scottshea 14y ago
    Having been part of their efforts to test 2X, and 4X Dynos and using Unicorn long before the RapGenius issue I can tell you the added memory and Unicorn still has issues. We still see periods where queueing is above 500ms. The additional Dyno capacity distributes the chance of queue issues out over a larger numerical set but there is still the possibility of one dyno/Unicorn worker combo getting too much to handle. We use Unicorn Worker Killer to help in that case.
  - scottshea 14y ago
    I should add that using Unicorn Worker Killer to end a process on thresholds is not a solution as much as it is another stopgap.
  - adamwiggins 14y ago
    We might. But what does this actually get us? It helps clear Heroku's name, but it doesn't help our customers at all. I'd prefer to spend our time and energy making customer's lives better. Given the choice between continuing the theoretical debate over routing algorithms vs working on real customer problems (like the H12 visibility problem mentioned elsewhere in this thread), I much prefer the latter.
    - habosa 14y ago
      I respect that mindset, I just don't think it would hurt. Maybe a middle ground would be a full-scale tutorial on how to switch from Thin on Bamboo/Cedar to Unicorn on Cedar for Rails users. It's a non-trivial process and I know I'd like some help with it. And in this same tutorial/article you could throw down the benchmarks you ran as motivation/justification.
- boundlessdreamz 14y ago
  Unless the dyno signals to the router that it is busy, isn't this just postponing the problem? Per dyno, unicorn can handle more requests but requests will still get queued at dyno level if one of the requests is a slow one (say 2-3 seconds)
  - thinkbohemian 14y ago
    If only 1 request is slow and you have 7-8 unicorn workers, only one of them will stay busy. Unicorn knows which of it's workers are busy and does not queue jobs behind individual worker but rather behind the master who delegates the request to the first available worker.
    - adamwiggins 14y ago
      Precisely. As mentioned in the FAQ, putting the queueing logic closer to the process which is ultimately going to serve the request is a more horizontal-scale friendly way of tackling the queueing problem. It works fantastically well for backends that can support 20+ concurrent connections, e.g. Node.js, Twisted, JVM threading, etc. It works less well as you can put fewer connections in each backend, which is part of why we're working on larger dynos.
- tjb 14y ago
  We also adopted Unicorn pretty early on but still suffered issues as dynos simply ran out of memory. In fact with some apps we have seen improvements (but still far from acceptable) in stability by un-adopting this method. The issues raised by scottshea below, as a consequence concern me, what/will be the charge for these as well? To be honest it's the fumbling around in the dark that has annoyed me. I am with you 100% on your manifesto and your points about the type of service you provide. However the time we have spent on this (starting before you came clean about the issues) and the time spent on other increasingly suspicious advice to "up dynos" or spend time "optimising your app" sours this slightly. I accept the "magic black box" comes with its compromises and required understanding at our end but it also means needing to be far more communicative and honest about it at yours. Something which you are putting right I can see. I for one think the premise of Heroku is a great one and you have succeeded for us in many of the things you have set out to achieve. This whole situation has been a real shame, I'm sure this must have a been a pretty shody time for you guys and I hope you come out the better for it. The quicker the better to be honest so you can focus on the new features we'd like to see.
  - adamwiggins 14y ago
    Thanks for your support. Indeed, communication and transparency into how the product works as far as how it affects your app are two things we'd like to get better at. Regarding your app: indeed, Unicorn is a huge improvement, but far from the end of the story. "Performance" is like "security" or "uptime" — it's not a one-time feature, something you check off a list and move on. It's something that requires constant work, and every time you fix one problem or bottleneck that just leads you to the next one. Over time, though, your vigilance pays off with a service that its users deem to be fast or secure or have good uptime. Yet there's no such thing as a finish line on these. Bringing it back to details. Kazuki from Treasure Data made this Unicorn worker killer that might help you: https://github.com/kzk/unicorn-worker-killer https://github.com/kzk/unicorn-worker-killer If you're still not happy with your app's performance, give me a shout at adam at heroku dot com and we'll see if we can help.

hglaser 14y ago

This is a great response. I'm curious why something like this wasn't posted within 24 hours of RapGenius going public? I'd bet that a more thorough, technical reply would have mitigated a lot of the PR issues.

steveklabnik 14y ago
When admitting fault in response to a customer that spends 5 figures/month with you, wise to have a lawyer go over it first.
- hglaser 14y ago
  I would argue that the brand risk -- e.g. the risk of what ended up happening -- outweighs the legal risk. That's not to say I wouldn't send it to a lawyer. But I'd do it with some hustle. Something like: "This is going out in 24 hours. Please comment ASAP!"
  - res0nat0r 14y ago
    As Adam mentioned in so many words: most of the people pissed off about all of this are most likely right here on HN, and not even customers of Heroku. He mentioned most customers were not that mad about the issue overall.
adamwiggins 14y ago
Agreed, I wish we could have done it much sooner. It took a shocking amount of time to sort through all the entangled issues, emotion, and speculation to try to get to the heart of the matter, which ultimately was about web performance and visibility. Also, we wanted to respond to our customers first and foremost, and general community discussion second. So we spent close to a month on skype/hangout/phone with hundreds of customers understanding how and at what magnitude this affected their apps. That was hugely time-consuming, but it gave us the confidence to speak about this in a manner that would be focused on customer needs instead of purely answering community speculation and theoretical discussions.

ebbv 14y ago

It seems to me that Heroku is still failing to understand (or at least cop to) the fact that the switch from intelligent to randomized routing was a loss of a major reason people chose Heroku in the first place. A lot of Heroku's apparent value came from the intelligent routing feature. Everybody knew that it was harder to implement than random routing, that's why they were willing to pay Heroku for it. Nobody's arguing random routing isn't easier and more stable; of course it is. The problem is that by switching over to it, Heroku gave up a major selling point of their platform. Are they really blind enough not to know this? I have a hard time believing that. It seems to me the real way to make people happy is to discount the "base" products which come with random routing and make intelligent routing available as a premium feature. Of course, people who thought they were getting intelligent routing should be credited.

tptacek 14y ago
Is anybody operating at Heroku's scale offering centralized request routing queues? At what price?
- ebbv 14y ago
  Not that I know of but that's why I'm saying it would be a premium product. Likely pricing would have to scale with the number of dynos running behind the router. But that's the service people thought they were getting and what they wanted. If Heroku prices out the intelligent routing and says; "Ok you can have intelligent routing with your current backend stack, but it's going to cost you $25/mo for evert 10 dynos, or you can switch your stack and use randomized routing for free." Then they are empowering their customers to make the choice rather than dictating to them what they should do.
  - chc 14y ago
    $25 a month? Did you forget a few zeroes?
    - ebbv 14y ago

streptomycin 14y ago

He forgot to explain why they won't be refunding customers who were defrauded.

res0nat0r 14y ago
Because they weren't victims of fraud.
- dangrossman 14y ago
  Is promising one service but delivering another not fraud? Promises from the Heroku website pre-Rap Genius posts: "Incoming web traffic is automatically routed to web dynos, with intelligent distribution of load instantly" "Intelligent routing: The routing mesh tracks the availability of each dyno and balances load accordingly. Requests are routed to a dyno only once it becomes available. If a dyno is tied up due to a long-running request, the request is routed to another dyno instead of piling up on the unavailable dyno’s backlog." "Intelligent routing: The routing mesh tracks the location of all dynos running web processes (web dynos) and routes HTTP traffic to them accordingly." "the routing mesh will never serve more than a single request to a dyno at a time" Actual service provided: requests are routed randomly to dynos regardless of how many requests they are currently handling or their current load.
  - streptomycin 14y ago
    Also the misleading performance metrics which hid the fraud.
  - res0nat0r 14y ago
    Your definition of fraud differs from most: "Wrongful or criminal deception intended to result in financial or personal gain." They weren't intentionally and purposefully misleading people. Not having docs up to date on your website, or you not knowing how the underlying backend works is not fraud. As I've mentioned before if every AWS customer could sue Amazon for not understanding how all of the underlying tech worked, or could sue when some of the docs were out of date, there would be more lawyers working there than engineers.

tvladeck 14y ago

All of the stuff that Heroku are doing now to mitigate the routing issues for their Bamboo customers are things they should have done when it first became an issue. They are not going above and beyond in any way to make up for the lost time and money their customers. Again, this is not about -how advanced Heroku's technology is on an absolute level -how challenging routing is for their scale -what competitors offer in this space and for what prices This is only about the delta between what Heroku sold their customers and what their customers received. They are collapsing the delta now, by being honest about what they are selling (and improving their offering, it seems), but they are doing nothing to address the long time for which the delta was significant for a subset of their customers.

chc 14y ago
Your complaints are all about the past. What are you looking for here? For Heroku to invent a time machine?
- ambrice 14y ago
  How about refunds? Isn't that the normal response when a company screws up in a way that causes the customers to pay more money?
  - adamwiggins 14y ago
    Yes, we've given credits (or a refund, at the customer's preference) in cases where lack of visibility or inaccurate docs led to over-provisioning of dynos. There's actually very few cases where people paid more money than they would have otherwise. Heroku is a service with your bill prorated to the second. For the most part, if people don't like the performance (which is measurable externally via benchmarks and browser inspectors), they leave the service. Many people who hit problems with visibility and performance did exactly that. Naturally, we'll be working hard to try to recapture their business, as well as to remove any reasons that existing customers might leave as they hit performance or visibility problems scaling up in the future.

badgar 14y ago

> 1k req/min Also known as <17 requests per second... or a trickle of traffic. Hooray for using bigger numbers and a nonstandard unit to hide inadequacy! Does Heroku use req/min throughout their service? I can't understand why they would, unless they also can't build the infrastructure to measure on a per-second basis. > After extensive research and experimentation, we have yet to find either a theoretical model or a practical implementation that beats the simplicity and robustness of random routing to web backends that can support multiple concurrent connections. Does this CTO think companies like Google and Amazon route their HTTP traffic randomly? No... he knows there are scaleable routing solutions and random routing isn't the best. So he cites "simplicity and robustness." Here, this means "we can't be bothered."

adamwiggins 14y ago
Re: requests. RPM is the metric that New Relic reports, and it's the one most of our customers use when they talk about traffic. I try to speak in whatever terms are most familiar to our customers. Re: I can't speak to Google and Amazon, and they aren't representative of the size of our customers anyway. We have discussed with many folks who run ops at many companies that are more on par with the size of our mid- and large-sized customers, and single global request queues are exceedingly rare. The most common setup seems to be clusters of web backends (say, 4 clusters of 40 processes each) that each have their own request queue, but with random load balancing across those clusters. This is a reasonable happy medium between pure random and global request queue, and isn't too different from what you get running (say) 16 processes inside a 2X dyno and 8 web dynos.
meritt 14y ago
I too was shocked at that. Also that 6 dynos is apparently the average size to handle that load. It takes $179/mo (6 dynos) to handle 17 requests/second? That's insane.
- adamwiggins 14y ago
  Didn't intend to imply that. Number of dynos needed varies extremely widely, with the app's response time and language/framework being used as the main variables. There are apps on Heroku that serve 30k–50k reqs/min on 10–20 dynos, typically written in something like Scala/Akka or Node.js and serving incredibly short (~30ms) response times with very little variation. But these are unusual. The more common case of a website, written in non-threadsafe Rails, with median response times of ~200ms but 95th percentile at 3+ seconds, would probably use those same 10 dynos to do only a few thousand requests per minute. Whether or not you use a CDN and page caching also makes a big difference (see Urban Dictionary for an example that does it well). But it really depends. We were trying to quantify when you should be worried. If you're running a blog that serves 600 rpm / 10 reqs/sec off of two dynos, you don't need to sweat it.

bhauer 14y ago

I am surprised they do not moderate the comments on their blog. They have one visible presently that is plain offensive.

cmelbye 14y ago

They still fail to understand that using Unicorn doesn't magically fix this issue. Like, at all. It simply means that the dunk gets tied up when n+1 requests (n is number of unicorn workers) get randomly routed to it instead of just 1. It's in no way comparable to a node.js server that handles thousands of concurrent requests asynchronously. They're simply two different designs, and Ruby's traditional design is fundamentally incompatible with Heroku's router.

mattsoldo 14y ago
That's not right. In any configuration the goal is to minimize queue time. What is critical to doing this is having a request queue and a pool of concurrent "workers" (to use the generic queueing theory term) in back of it. Unicorn uses the operating system's TCP connection queue to queue incoming requests that it is not able to immediately server. While n+1 requests can (and will) get routed to a single dyno, this only results in 1 request being queued. It will be queued until the first of the in-process requests is served, which will take ~ the average response time for the app. Given that the other n requests did not get queued (queue time = 0), the average queue time will equal Sum(queue times) / # requests = Average Response Time / n+1.

persei8 14y ago

Is it true that you don't fully buffer body of POST request on router? This limitation works against Unicorn design and makes it hardly a "fix": http://rubyforge.org/pipermail/mongrel-unicorn/2013-April/001743.html http://rubyforge.org/pipermail/mongrel-unicorn/2013-April/00...

geekylucas 14y ago
Heroku only buffers the request headers: https://devcenter.heroku.com/articles/http-routing#request-buffering https://devcenter.heroku.com/articles/http-routing#request-b...
adamwiggins 14y ago
Plenty of room for improvement here, sure. Some technical discussion in this thread: https://blog.heroku.com/archives/2013/4/3/routing_and_web_performance_on_heroku_a_faq#comment-850974645 https://blog.heroku.com/archives/2013/4/3/routing_and_web_pe...

bradleyjg 14y ago

Random routing to concurrent servers works fairly well if the kind of long running requests you need to worry about spend a lot of time blocking for some external service (e.g. a database call). Then you can get a lot of benefit on the from cooperative or preemptive multitasking on the server, and so the performance characteristics, from the point of view of a new request, of each server is roughly the same, and so going to a random server is pretty good. However, if you have long running requests because they make intensive use of server resources (CPU, RAM) then concurrent servers buys you very little. In that case, sending a new request to a server that is chugging along on a difficult problem is significantly different than sending it to one that isn't. That's where knowing the resource state of each server, and routing accordingly is of huge benefit. While load balancing is a very difficult problem, with some counterintuitive aspects, it is an area of active research, and there are some very clever algorithms out there. For example, this article (http://research.microsoft.com/pubs/153348/idleq.pdf http://research.microsoft.com/pubs/153348/idleq.pdf) from UIUC and Microsoft introduces the Join-Idle-Queue, which is suitable for distributed load balancers, has considerably lower communication overhead than Shortest Queue (AFAICT the original 'intelligent' routing algorithm), and compares its charateristics to both SQ and random routing.

bifrost 14y ago

I'd say there's a major flaw here - "the new world of concurrent web backends" - If now was 1995 I might agree with you, but having concurrency in web-app servers is not new. The lack of concurrency in test/demo app situations is totally understandable, in a production environment you'd have to be completely bonkers to think thats ever useful. I also agree that "having to deal with loadbalancing" is something that most people don't get and shouldn't really have to get, but when the way you do loadbalancing is so fundamentally flawed that its worse than Round Robin DNS, you also clearly don't understand it. To be fair, I have to say I meet people all the time who don't understand it at all and think $randomcrappything is great at loadbalancing. If "connections go to more than one box!" is your metric, then yes, thats loadbalancing. My metric is "Do you send connections to servers that are responsive and not overloaded, maybe with session affinity" and in general most HW loadbalancer products since 1998 have supported that. So if you're not better than 1998 technology, you may want to reevaluate your solution.

bgentry 14y ago
I'd say there's a major flaw here - "the new world of concurrent web backends" - If now was 1995 I might agree with you, but having concurrency in web-app servers is not new. Sadly, concurrency is relatively new and unfamiliar territory to many in the Ruby on Rails community.

derengel 14y ago

Heroku's message: "If you have slow clients you are screwed" Unicorn is designed to only serve fast clients on low-latency. And no, they don't do any buffering.

adamwiggins 14y ago
I think that overstates it a bit, but yes, there are problems with Unicorn and slow clients. We're investigating: https://blog.heroku.com/archives/2013/4/3/routing_and_web_performance_on_heroku_a_faq#comment-850974645 https://blog.heroku.com/archives/2013/4/3/routing_and_web_pe... If this is an immediate problem for you, it might be worth your while to make your app threadsafe, which gives you more concurrent webserver options.

thatthatis 14y ago

What data do you have to show that the random selection algo has superior performance to a round-robin algo?

adamwiggins 14y ago
We investigated round-robin. With N routing (or load balancer) nodes, and any degree of request variance, round robin effectively becomes random very quickly.

deedubaya 14y ago

Haters gonna hate. I wonder how many people bitching here are actual customers who are having problems that haven't been address with a solution. I'm guessing that number is low. Oh, you're a potential customer? That's why you're bitching? About a problem you may or may not have if you actually choose the product? Think about that argument for a second. I've never seen such a transparent response and follow up as I have from Heroku on this issue. Most other companies would have gone into immediate damage mitigation mode and let the wound heal instead of re-opening it and giving feedback on how to fix the problem as Heroku has. I applaud the Heroku team for their effort on their platform and being a kickass company.

stickfigure 14y ago
I'm a real customer with real problems. Grep this page for latchkey's description of it. The funny thing is, I don't have much sympathy for Rails users. Scaling problems with a single-threaded, serial request-processing architecture? No surprise there. But we have inexplicable H12 problems with Node.js. There's something broken in the system and it isn't random routing.
- Cabal 14y ago
  You're talking like Node offers real concurrency. It doesn't.
  - stickfigure 14y ago
    There's nothing wrong with Node's concurrency. Our app, like most webapps, is I/O bound. Any individual instance should be able to handle thousands of concurrent requests as long as they are all blocked on I/O. Being able to process more than one concurrent request (as Node can) is "real concurrency". Java-style native threading is a step above and beyond this, and unnecessary for most web applications.

latchkey 14y ago

I am working for a fairly large heroku app running Node on ~50-100 web dyno's with another 20-50 backends. Here are the problems as I see it: We get H12's all the time. Randomly. The only suggestion we get from Heroku is to make the requests process as fast as possible. Thus, we've spent considerable amount of time going through everything we can possibly do to make all requests respond as fast as possible. I've given up. I see this as a fundamental issue with the routing system. If you are going to use Heroku for a large production deployment, H12's (and your users getting dropped connections) will become a fact of life. There is no auto scaling. We have no idea how many dyno's we actually need. So, we over do it in order to handle peek traffic times. This must be a great money maker for Heroku. There is no incentive for them to build auto scaling into their system because that would mean they wouldn't make as much money. Yes, auto scaling is a hard problem to solve, but there should at least be a plan to start on it and there is none that I have found. Up until someone bitched loudly, nothing was happening to fix any of this. We have an expensive paid support contract with Heroku and before this whole routing issue blew up in public, their only recommendation was to tune the app more and buy into NewRelic for ~$8k / month. We did both and found NewRelic to not give any relevant information to help us. We did a NodeTime trial for ~$49/mo and that actually helped a lot in identifying slow spots in our app. We fixed all the slow spots in our app and still see an endless stream of H12's. Regardless, it shouldn't take a public bitch slapping for a company to listen to their customers. You log into a dyno and see a load average of 30+. Who knows if that number is accurate or how big the underlying box really is, but regardless, I can't imagine that number being good. Am I getting H12's because I'm on an overloaded box or is it because the routing system is fundamentally broken? I don't know and nobody can tell me. This is not a good position to be in. I have heard from several sources that Heroku isn't happy being on AWS and has been wanting to migrate off AWS for a while now. So, if your hosting provider isn't happy on their hosting provider, there must be a reason for that and in the bigger picture, you the customer, is getting screwed. Given these things, I will never recommend that a company use Heroku. It is great if you know you are going to never have more than one dyno, but if you think you are going to go into a large production system with it, it is far better to find something else. Which brings me to another rant... how come none of these other PaaS solutions are as easy as Heroku? The git deploy is seriously the one thing they got mostly right. I'd love to see someone build a layer on top of all the PaaS solutions so that I can just deploy my code to any one of them (or event multiple).

ROFISH 14y ago

Something that's been bugging me about Heroku is that the dyno price has stayed the same ever since launch: $0.05 per hour. Compare to services like Digital Ocean and AWS (who lowered prices significantly in the past few years), Heroku is starting to get very expensive. The 2X dyno at 2x cost doesn't really make me happy, it just invites me to spend more money when it would be more cost-efficient to move.

cjackson27 14y ago

I can't speak to the issues that people are running into when they reach large scale, but I run a small app with two dynos and we've been having issues with H12 request timeout errors for weeks now. This has been bringing down our production app for periods of about fifteen minutes almost daily. I've been completely disappointed with Heroku's support so far. First they obviously skimmed my support request and provided a canned response that was completely off base. Their next response didn't come for four days and only after I called their sales team to see what I could do to get better support. Their only option is a $1k / mo support contract. If you're running a mission critical app, I'd think twice before choosing Heroku.

adamwiggins 14y ago
The difficulty of diagnosing H12 errors is really challenging. One thing I can recommend is using the http-request-id labs feature: https://devcenter.heroku.com/articles/http-request-id https://devcenter.heroku.com/articles/http-request-id With this enabled and some code in your app, you can correlate your app's request logs directly against the router logs and trace what happened with any particular H12. I'd be happy to help you do this if you're game. Contact me via adam at heroku dot com. Could you also email me some links to your support tickets so I can check out what happened there?

jholman 14y ago

I'd like to start by acknowledging that I'm one of the "non-customers who are watching from the sidelines". I think Adam's right that this is an important distinction. Adam, there's something that confuses me about this. I'm no expert in routing theory, nor have I done the experiments, so forgive me if my reasoning misses something. I understand why RapGenius took you up on your original promises of "intelligent routing", and I think I understand what you're saying about scaling, and how scaling "intelligent routing" is so far unsolvable, and the motivation for your transition from Bamboo to Cedar, especially in the context of concurrent clients. What I don't understand is this: It seems to me that if you split into two (or more) tiers, and random-load-balance in the front tier (hit first by the customer), and then at the second tier only send requests to unloaded clients, that you eliminate RapGenius's problem for customers who followed your specific recommendations for good performance on Bamboo (to go single-threaded and trust the router). Do you have reason to believe that this doesn't one-shot RapGenius's problem? Do you have strategic/architectural reasons for rejecting this even though it would work? Did you try it and it failed? What's the story there? Maybe I'll write a simulator to (dis)prove my naive theory. :P

adamwiggins 14y ago
> It seems to me that if you split into two (or more) tiers, and random-load-balance in the front tier (hit first by the customer), and then at the second tier only send requests to unloaded clients [...] I'm unclear how you'd think introducing a second tier changes things. That tier would need to track dyno availability and then you're right back to the same distributed state problem. Perhaps you mean if the second tier was smaller, or even a single node? In that case, yes, we did try a variation of that. It had some benefits but also some downsides, one being that the extra network hop added latency overhead. We're continuing to explore this and variations of it, but so far we have no evidence that it would provide a major short-term benefit for RG or anyone else. > Do you have reason to believe that this doesn't one-shot RapGenius's problem? As a rule of thumb, I find it's best to avoid one-shots (or "specials"). It's appealing in the short term, but in the medium and long term it creates huge technical debt and almost always results in an upset customer. Products made for, and used by, many people have a level of polish and reliability that will never be matched by one-offs. So if we're going to invest a bunch of energy into trying to solve one (or a handful) of customer's problems, a better investment is to get those customers onto the most recent product, and using all the best practices (e.g. concurrent backend, CDN, asset compilation at build time). That's a more sustainable and long-term solution.

deleted 14y ago

[deleted]

siong1987 14y ago

Since we are on the question of visibility of Heroku dyno, what is the amount of CPU power each dyno has? What about 2Xdyno?

adamwiggins 14y ago
This is a tough area. If you go look at various types of infrastructure providers (e.g. EC2, Linode, Rackspace) you'll see that they always end up making up vague units of measurement (e.g. "cores") and then showing all the resources in reference to whatever the base unit is. So there's really no good way to talk about CPU power like there is with memory. That said, I can say that a 1X dyno is not very powerful compared to, say, any server you'd purchase for your own datacenter. Our intention is that 2X dynos will provide twice the CPU horsepower, although CPU and I/O are harder to allocate reliability in virtualized environments.

jordanthoms 14y ago

My biggest issue with heroku is the general slowness with the API - maybe I'm just impatient, but most simple commands like listing releases, viewing logs etc take at least a second, sometimes five before anything happens. Pushes also take quite a while, even the Git push part is much slower than pushing to github. It's just a general sluggishness which gets annoying after a while. If they could get all the API requests down under 500ms I'd be much happier.

adamwiggins 14y ago
Yeah, the developer-facing control surfaces on the platform (API calls and git push mainly) have gotten slower over the past year or two. This is on my list of of personal pet peeves, but so far has not made it onto our list of priorities. We try to drive priorities based on what customers want, not what we want: and what we've heard in the last year or so is all about app uptime, security, and now performance and visibility. I'm very much hoping that bringing back "fast is a feature" on the developer-facing portions of the product is something we can work on this year.

j-kidd 14y ago

From the article: > Q. Did the Bamboo router degrade? > A. Yes. Our older router was built and designed during the early years of Heroku to support the Aspen and later the Bamboo stack. These stacks did not support concurrent backends, and thus the router was designed with a per-app global request queue. This worked as designed originally, but then degraded slowly over the course of the next two years. From Adam's message on Feb 17th, 2011 (https://groups.google.com/forum/?fromgroups=#!topic/heroku/8eOosLC5nrw https://groups.google.com/forum/?fromgroups=#!topic/heroku/8...): > You're correct, the routing mesh does not behave in quite the way described by the docs. We're working on evolving away from the global backlog concept in order to provide better support for different concurrency models, and the docs are no longer accurate. The current behavior is not ideal, but we're on our way to a new model which we'll document fully once it's done. It looks like random load balancing was already the expected behavior 2 years ago? The "slow degradation" part seems a bit dishonest to me.

adamwiggins 14y ago
There are two separate issues here, and it's easy to get them confused. One is the slow degradation on Bamboo without any change to the routing algorithm code, and the other was the explicit product choice for Cedar with a different code path in the router. Both are described fully here: https://blog.heroku.com/archives/2013/2/16/routing_performance_update https://blog.heroku.com/archives/2013/2/16/routing_performan... The reason it's easy to confuse these two is also part of what confused us at the time. The slow degradation of the Bamboo routing behavior was causing it to gradually become more and more like the explicit choice we had made for our new product. But of course it's up to you (and everyone else observing) to judge whether this was some kind of malicious intent to mislead, versus that we made a series of oversights that added up to some serious problems for our customers. And that we are now doing everything in our power to be fully transparent about, to rectify, and to make sure never happen again.
- deleted 14y ago
  [deleted]
- j-kidd 14y ago

randall 14y ago

I think the most annoying thing is they still don't answer Rap Genius's questions about being owed money for paying megabucks for newrelic. I mean If you offer a service that provides incorrect data for two years and you don't offer any sort of framework for reimbursement, that still seems, at best annoying, at worst, dishonest.

adamwiggins 14y ago
We spent quite a bit of time trying to find a one-size-fits-all framework. There just wasn't one, so we've done credits on a case-by-case basis. Sorry you find it annoying. It's what was best for our customers.