3 ms·

There are a couple of things wrong with the numbers. 1. Traffic is not evenly spread. The figures from the article (400M page loads per month) are subject to r

by bostik 3y ago

There are a couple of things wrong with the numbers. 1. Traffic is not evenly spread. The figures from the article (400M page loads per month) are subject to recursive 80/20 rule. 80% of the requests (320M) are served within 20% of the time (6 days). Within those high-volume days, 80% of the requests (256M) are served in 20% of the time (~29h). And if you're serving particularly spiky traffic patterns, then 80% of the those requests (~205M) come in during just 20% of the time window (5.8h) -- 5.8h is a little shy of 21000 seconds. 205M / 21k is about 9.7k requests per second. That's still doable on a single system, but it's no longer trivial. Especially if you want to run off a single DB with no read replicas. And while the total amount of traffic served remains the same, the necessary bandwidth cap for peak loads gets far above the optimistically averaged 11MB/sec. 2. Unidirectional end-to-end latency only applies to streaming data. A cold start in the real world (no HTTP/3) requires first to establish the underlying TCP connection which is three trips, then the TLS connection which is a minimum of two more trips, and then you get to send the actual HTTP request... which still needs to send the response back. If you want to serve real humans, everything observable has to happen in less than one second[0]. After that there's a steep drop-off as users assume your system is broken and just close the tab. Disclosure: in previous life I helped to run a betting exchange. The traffic patterns are extremely spiky, latency requirements are demanding and trading volume is highly concentrated in just a tiny fraction of the overall event window. For any activity involving live trades, we had to get the results on their screen within 100ms from the moment they initiated the action. That means their network roundtrip latency ate into our event processing budget. 0: https://www.nngroup.com/articles/response-times-3-important-limits/ https://www.nngroup.com/articles/response-times-3-important-...

swiftcoder 3y ago
Point 1 is the kicker, I think, for something like a business insider clone. If a big twitter account links to one of your articles, you may get the whole damn lot of average monthly page loads over the next few minutes. And you absolutely can deal with that sort of load in single large server configurations, but now we're not just building a webserver, we're building a pretty hardcore frontend load balancer (that happens to have an embedded webserver). They may well charge through the nose for it, but a hell of a lot of engineering has gone into AWS' load balancers and network infrastructure, so that the rest of us don't have to become experts in that whole segment of the stack.
- omnimus 3y ago
  Majority of websites that have such viral spikes like businessinsider are content sites. Caching HTML (and everything else) on CDN solves that. Afaik news/political media have to do that anyway because of DDoS. Thats why they can run wordpress (even though thats terribly slow cms/framework). You end up with essentially static website.
  - swiftcoder 3y ago
    Even content sites mostly have user-generated comment feeds (requires a DB connection), and ads (which require processing metrics to justify ad spend). I don't know if the business folks would rather sacrifice those in a load spike, or just load shed at a request level - for an business defined by engagement metrics and ad revenue, it's not clear the former is better than the latter.
    - omnimus 3y ago
      Most of these are handled with outside systems nowdays anyway? Its embedded/loaded using separate API. Separate what can be static and what has to be dynamic. I dont think anyone serious would use comments system from CMS directly. If your comments section dies its a lot better than whole site going down.
      - dzhiurgis 3y ago
        The point of article is to stick to a single system. Comments can be cached too tho.
        omnimus 3y ago
        Article talks about single server. Not a single system. You will not crash your nginx+static website if your nodejs comments api crashes.
- bostik 3y ago
  > a hell of a lot of engineering has gone into AWS' load balancers and network infrastructure Most of it hard earned, after having to deal with thousands of outlier events. An old friend used to run the Finnish election results services' public-facing backend, and was an early AWS customer. He broke their load-balancers - twice. Both times he got in touch with AWS support well in advance, asking to pre-scale and pre-warm the load balancers because he was expecting a major traffic surge once the results started to come in. First time around, he was confidently told that he shouldn't worry, since AWS can take on any amount of traffic. On the election night, AWS load balancers failed to scale in time. The traffic estimates my friend had provided were indeed accurate within an order of magnitude, but AWS hadn't believed his numbers. NOBODY could have a service that legitimately required scaling from a few hundred requests per second to 2M requests per second in about one minute. Apparently their on-call engineers got burned badly that night. Couple of years later, he approached AWS support again, with the same request. That time he was taken more seriously, and was assured that the autoscaling algorithms were now much smarter and their relevant engineering teams had prepared the ground for more rapid autoscaling needs. They were confident they didn't need to pre-scale the fleet. The scaling wasn't still fast enough and their on-call team had to _again_ manually force their balancers to stay just ahead of the demand with the massive surges coming in waves. So much for the fine-tuned scaling algorithms. Fast-forward to today, and AWS have a specialist "hot launch" service offering where they work with the customers up front to make sure their launches have sufficient capacity available and their perimeter load balancers properly pre-warmed to absorb these kinds of supposedly one-off situations.
bcaxis 3y ago
> 2. Unidirectional end-to-end latency only applies to streaming data. Agreed that his "across the world" example is a bit silly. Because he doesn't take into account the connection construction. His primary point is still reasonable. How many services need world wide reach? Did you build it for multiple languages also? If you're in the US. Or you're in the EU. A nice centralized server will have <=30 ms of latency to the entire region you are serving. Edge is over valued unless you do have true global needs and then you have to also manage global database (s).

by bostik 3y ago

swiftcoder 3y ago

Point 1 is the kicker, I think, for something like a business insider clone. If a big twitter account links to one of your articles, you may get the whole damn lot of average monthly page loads over the next few minutes. And you absolutely can deal with that sort of load in single large server configurations, but now we're not just building a webserver, we're building a pretty hardcore frontend load balancer (that happens to have an embedded webserver). They may well charge through the nose for it, but a hell of a lot of engineering has gone into AWS' load balancers and network infrastructure, so that the rest of us don't have to become experts in that whole segment of the stack.

omnimus 3y ago
Majority of websites that have such viral spikes like businessinsider are content sites. Caching HTML (and everything else) on CDN solves that. Afaik news/political media have to do that anyway because of DDoS. Thats why they can run wordpress (even though thats terribly slow cms/framework). You end up with essentially static website.
- swiftcoder 3y ago
  Even content sites mostly have user-generated comment feeds (requires a DB connection), and ads (which require processing metrics to justify ad spend). I don't know if the business folks would rather sacrifice those in a load spike, or just load shed at a request level - for an business defined by engagement metrics and ad revenue, it's not clear the former is better than the latter.
  - omnimus 3y ago
    Most of these are handled with outside systems nowdays anyway? Its embedded/loaded using separate API. Separate what can be static and what has to be dynamic. I dont think anyone serious would use comments system from CMS directly. If your comments section dies its a lot better than whole site going down.
    - dzhiurgis 3y ago
      The point of article is to stick to a single system. Comments can be cached too tho.
      - omnimus 3y ago
        Article talks about single server. Not a single system. You will not crash your nginx+static website if your nodejs comments api crashes.
bostik 3y ago
> a hell of a lot of engineering has gone into AWS' load balancers and network infrastructure Most of it hard earned, after having to deal with thousands of outlier events. An old friend used to run the Finnish election results services' public-facing backend, and was an early AWS customer. He broke their load-balancers - twice. Both times he got in touch with AWS support well in advance, asking to pre-scale and pre-warm the load balancers because he was expecting a major traffic surge once the results started to come in. First time around, he was confidently told that he shouldn't worry, since AWS can take on any amount of traffic. On the election night, AWS load balancers failed to scale in time. The traffic estimates my friend had provided were indeed accurate within an order of magnitude, but AWS hadn't believed his numbers. NOBODY could have a service that legitimately required scaling from a few hundred requests per second to 2M requests per second in about one minute. Apparently their on-call engineers got burned badly that night. Couple of years later, he approached AWS support again, with the same request. That time he was taken more seriously, and was assured that the autoscaling algorithms were now much smarter and their relevant engineering teams had prepared the ground for more rapid autoscaling needs. They were confident they didn't need to pre-scale the fleet. The scaling wasn't still fast enough and their on-call team had to _again_ manually force their balancers to stay just ahead of the demand with the massive surges coming in waves. So much for the fine-tuned scaling algorithms. Fast-forward to today, and AWS have a specialist "hot launch" service offering where they work with the customers up front to make sure their launches have sufficient capacity available and their perimeter load balancers properly pre-warmed to absorb these kinds of supposedly one-off situations.

bcaxis 3y ago

> 2. Unidirectional end-to-end latency only applies to streaming data. Agreed that his "across the world" example is a bit silly. Because he doesn't take into account the connection construction. His primary point is still reasonable. How many services need world wide reach? Did you build it for multiple languages also? If you're in the US. Or you're in the EU. A nice centralized server will have <=30 ms of latency to the entire region you are serving. Edge is over valued unless you do have true global needs and then you have to also manage global database (s).