5 ms·

Building a high availability system? The downtime (especially scheduled downtime) of many of these "99%" services is shocking compared to the cutting edge consu

Building a high availability system? The downtime (especially scheduled downtime) of many of these "99%" services is shocking compared to the cutting edge consumer services we're used to. The "hug of death" is an affectionate meme that basically describes the remarkably common scenario where a "99%" website goes down at the precise moment at where it's peaking in popularity and profitability. Or how about high performance? Google famously knows that every millisecond of delay strongly impacts user retention and bounce rate, yet you wouldn't know it browsing many "99%" services. Achieving consistently low end-to-end latency is really hard regardless of scale, it's so easy for performance issues to creep in from every corner of your stack and accumulate. These are hard problems and IMO they are rarely solved well by just stapling together some useful crap.

falcolas 5y ago
Those are usually business decisions. Pay an extra $$$ for an outage scenario that occurs once a month, and doesn't materially impact our users? Hell no. Spend XXX developer hours and/or $$$ on hardware to make the site 50% faster, instead of adding new features our users pay for? Hell no. And let's be honest, for touting speed, Google's awfully slow and bloated these days.
- themacguffinman 5y ago
  A hug of death when your product has peak user interest...doesn't materially impact users? Your users don't care about multi-hour or even multi-day downtimes in "99%" industries like...healthcare and banking? What a perfect demonstration of "Those scenarios are rare if you have gotten used to what 99% can and can't do and therefore just dismiss many solutions without thinking since you know the 99% can't do those thing." Yeah I wonder why these orgs always find these XXX hours & $$$ costs cost-prohibitive even though the benefits have proven to be substantial in cutting edge tech industries. It's almost like they typically dismiss smarter and more cost-effective technical solutions superior to naive "throw hours and hardware at it" approaches, because 99% can't do those things.
  - falcolas 5y ago
    > Your users don't care about multi-hour or even multi-day downtimes Should ask Reddit. And yes, it's a sad state of affairs, but an hour of downtime in healthcare isn't all that uncommon. Convincing administrators to ~triple their hosting costs to avoid the occasional outage is pretty hard to do. I mean, I recently watched a documentary on the 2003 east coast power outage, and one of the primary causes was the single computer in charge of issuing alerts (in an alert-based workflow system) going down.
    - themacguffinman 5y ago
      I'm not entirely sure you're responding to my main point. The point is that tripling hosting costs is the "99%" naive solution to avoiding outages. The point is that smart people invent cost effective solutions to outage issues that don't involve tripling hosting costs. The point is that "99%" are ignoring these smart technical solutions because they can't do it, but they are still problems nonetheless. The point is that these smart inventors are not "circus jugglers" as derisively described by a parent comment, they provide real value that the "99%" reflexively dismiss because apparently the best solution to outages they can muster is to triple hosting costs, real value that you don't get by just stapling useful shit together. The point is that when someone asks "As one of the 99%. What are some of these problems exactly, that me and my extremely large toolkit of cool things I can curl from Github can't solve?", the answer is "so you can build a cost effective system that doesn't take down the east coast power grid because a single computer goes down". Yeah, reddit is notorious for its downtime issues. They're notorious because it's actually rather unusual in the consumer space to be so bad, they kind of suck compared to the majority of their high tech competitors.

falcolas 5y ago

Those are usually business decisions. Pay an extra $$$ for an outage scenario that occurs once a month, and doesn't materially impact our users? Hell no. Spend XXX developer hours and/or $$$ on hardware to make the site 50% faster, instead of adding new features our users pay for? Hell no. And let's be honest, for touting speed, Google's awfully slow and bloated these days.

themacguffinman 5y ago
A hug of death when your product has peak user interest...doesn't materially impact users? Your users don't care about multi-hour or even multi-day downtimes in "99%" industries like...healthcare and banking? What a perfect demonstration of "Those scenarios are rare if you have gotten used to what 99% can and can't do and therefore just dismiss many solutions without thinking since you know the 99% can't do those thing." Yeah I wonder why these orgs always find these XXX hours & $$$ costs cost-prohibitive even though the benefits have proven to be substantial in cutting edge tech industries. It's almost like they typically dismiss smarter and more cost-effective technical solutions superior to naive "throw hours and hardware at it" approaches, because 99% can't do those things.
- falcolas 5y ago
  > Your users don't care about multi-hour or even multi-day downtimes Should ask Reddit. And yes, it's a sad state of affairs, but an hour of downtime in healthcare isn't all that uncommon. Convincing administrators to ~triple their hosting costs to avoid the occasional outage is pretty hard to do. I mean, I recently watched a documentary on the 2003 east coast power outage, and one of the primary causes was the single computer in charge of issuing alerts (in an alert-based workflow system) going down.
  - themacguffinman 5y ago
    I'm not entirely sure you're responding to my main point. The point is that tripling hosting costs is the "99%" naive solution to avoiding outages. The point is that smart people invent cost effective solutions to outage issues that don't involve tripling hosting costs. The point is that "99%" are ignoring these smart technical solutions because they can't do it, but they are still problems nonetheless. The point is that these smart inventors are not "circus jugglers" as derisively described by a parent comment, they provide real value that the "99%" reflexively dismiss because apparently the best solution to outages they can muster is to triple hosting costs, real value that you don't get by just stapling useful shit together. The point is that when someone asks "As one of the 99%. What are some of these problems exactly, that me and my extremely large toolkit of cool things I can curl from Github can't solve?", the answer is "so you can build a cost effective system that doesn't take down the east coast power grid because a single computer goes down". Yeah, reddit is notorious for its downtime issues. They're notorious because it's actually rather unusual in the consumer space to be so bad, they kind of suck compared to the majority of their high tech competitors.