8 ms·

I run an Alexa top-2000 website. (Mangadex is presently at about 6000.) I spend less than $250 a month. I have loads and loads of thoughts about what they co

by maxk42 5y ago

I run an Alexa top-2000 website. (Mangadex is presently at about 6000.) I spend less than $250 a month. I have loads and loads of thoughts about what they could be doing differently to reduce their costs but I'll just say that the number one thing Mangadex could be doing right now from a cursory glance is to reduce the number of requests. A fresh load of the home page generates over 100 requests. (Mostly images, then javascript fragments.) Mangadex apparently gets over 100 million requests per day. My site - despite ostensibly having more traffic - gets fewer than half that many in a month. (And yes, it's image-heavy.) A couple easy wins would be to reduce the number of images loaded on the front page. (Does the "Seasonal" slider really need 30 images, or would 5 and a link to the "seasonal" page be enough? Same thing with "Recently Added" and the numbers of images on pages in general.) The biggest win would probably be reducing the number of javascript requests. Somehow people seem to think there's some merit to loading javascript which subsequently loads additional javascript. This adds a tremendous amount of latency to your page load and generates needless network traffic. Each request has a tremendous amount of overhead - particularly for dynamically-generated javascript. It's much better to load all of the javascript you need in a single request or a small handful of requests. Unfortunately, this is probably a huge lift for a site already designed in this way, but the improved loading time would be a big UX win. Anyway - best of luck to MangaDex! They've clearly put a lot of thought into this.

radicalbyte 5y ago
Do you manage to get as many buzz-words and OSS products into your system as they do? :) In general the less moving parts you have in a system the more reliable, secure, efficient and cheaper the system becomes. In their case they run a site that is probably under constant attack by the "hired goons", so they're going to need to have more moving parts than others. Plus they will want to optimise for minimal development time (it's a hobby) so just adding another tried and trusted system into the stack to do something you need makes sense.
- starfallg 5y ago
  >In their case they run a site that is probably under constant attack by the "hired goons", so they're going to need to have more moving parts than others. That's taken care of by the DDoS-Guard system they placed fronting their infrastructure. The design of their system has to take this into account, but that is mainly on a IP and DNS level. The design of their stack behind the loadbalancer is mainly driven by their functional and non-functional requirements, rather than by the need to prevent DDoS attacks.
  - radicalbyte 5y ago
    The layering - defence in depth - is very much a security consideration. Especially if you're building a pure request/response/sync system you need that. Or you decouple with a queue for mutations and avoid a lot of issues.
    - starfallg 5y ago
      That may be in terms of managing general security, especially with regards to the attack surface of the solution, but here we are talking about DDoS, which is mostly a separate topic and handled on the network level (for volumetric attacks) and load-balancer level (for non-volumetric attacks) or a combination of both.
- maxk42 5y ago
  lol > In general the less moving parts you have in a system the more reliable, secure, efficient and cheaper the system becomes. 100% agreed. This is not my first high-traffic site, nor even the highest. (I built the analytics system for a an Alexa top-10 site in 2010, reaching some 30 billion writes / day off of a mere 14 small ec2 instances.) I've never seen a k8s implementation in production that was necessary. I will note that my Alexa-2k site is also a personal site (no revenue) and under constant attack. In fact we frequently suffer DDOSes that we don't even notice until reviewing the logs later because it doesn't suffer any latency under pressure.
  - radicalbyte 5y ago
    Interesting, wouldn't mind having a chat outside of HN if you're interested (see my profile for mail). I've spent much of my career working on systems with active users from the hundreds to low thousands, but which process a huge number (50k/sec scale) jobs/tasks. It's a totally different kettle of fish, and if I'm totally honest I'm shocked at how badly "web" scales and how common these naive and super inefficient implementations are (hint: my bare-metal server from 2005 was faster than expensive cloud VMs). Recently I've worked on two high-usage systems (one of which was "handling" 30k requests/second for the first couple of week).
    - deleted 5y ago
      [deleted]
    - golergka 5y ago
      > I've spent much of my career working on systems with active users from the hundreds to low thousands, but which process a huge number (50k/sec scale) jobs/tasks. MMO games, by any chance?
  - Folcon 5y ago
    Would you mind outlining your approach? Really interested to see how you think about this sort of thing =)...
    - 5y ago
tristan9 5y ago
Hi, we're trying to lower the requests:pageview ratio in general, but for what it's worth this article essentially: - ignores the vast majority of "image serving" (most is handled by DDG and our custom CDN) - the JS fragments thankfully should load only on first visit and then get aggressively cached by DDG/your browser One of the pain points is that there are a lot of settings for users to decide what they should or shouldn't see (content rating, original language of origin, search tags, etc) and some are already specifically denormarlized (when querying chapter entities, ES indices for those contain some manga-level properties to avoid needing to dereference that first too) -- however this also makes caching substantially less efficient in many places, alas Thanks!
- jiggawatts 5y ago
  Hi, I'm a performance tuning expert, and this thread piqued my interest. The first thing that I noticed is that even with caching enabled, you're loading "too much data". After loading the main page and then clicking one of the tiles, there are several JSON API calls. Here's an example, 195 kB transferred (528 kB size): https://api.mangadex.org/manga/bbaa17c4-0f36-4bbb-9861-34fc8fdf20fc/feed?limit=96&includes[]=scanlation_group&includes[]=user&order[volume]=desc&order[chapter]=desc&offset=0&contentRating[]=safe&contentRating[]=suggestive&contentRating[]=erotica&contentRating[]=pornographic https://api.mangadex.org/manga/bbaa17c4-0f36-4bbb-9861-34fc8... Oof. Half a megabyte of JSON! Ignore the network traffic for a moment, because GZIP does wonders. The real problem is that generating that much JSON is very "heavy" on servers. Lots and lots of small object allocations, which gives the garbage collector a ton of work to do. It's also expensive to decode on the browser for similar reasons. On my computer, this took a whopping 455ms to transfer, nearly half a second. That results in a noticeable latency hit to the site. In my consulting gig I always give developers the same advice: "Displaying 1 kilobyte of data should take roughly 1 kilobyte of traffic". In other words, there's isn't 500 KB of text anywhere on that page! A quick cut & paste shows about 8 KB of user-visible text in the final HTML rendering. That's a 1:60 ratio of content-to-data, which is very poor. I bet that behind the scenes, this took a heck of a lot more back-end network traffic and in-memory processing to generate. Probably tens to hundreds of megabytes of internal traffic, all up. This is one of the core reasons most sites have difficulty scaling, because for every kilobyte of content output to the screen, they're powering through megabytes or even gigabytes of data behind the scenes. Can this API query be cut down to match what's displayed on the screen? Can it be cached for all users? Can it be cached precompressed? Etc...
polote 5y ago
I don't know about you, but they have 42 average Page-views per visit (HN has 3) so Alexa rank is going to be biased
tyingq 5y ago
>A fresh load of the home page generates over 100 requests. I see 17 requests, all over either h2 or h3. 4 of them JS, and 2 images.
- maxk42 5y ago
  Then you're not doing a fresh load of the page. There are over 30 images visible on the front page, so your measure doesn't pass the smell test, does it?
  - tyingq 5y ago
    >Then you're not doing a fresh load of the page Nope. Different problem. The article was linked to a page under the domain "mangadex.dev". Without any other context, I had assumed "home page" meant http://mangadex.dev http://mangadex.dev , or what I got when clicking "Home" on the linked article. Apparently not.

by maxk42 5y ago

radicalbyte 5y ago

Do you manage to get as many buzz-words and OSS products into your system as they do? :) In general the less moving parts you have in a system the more reliable, secure, efficient and cheaper the system becomes. In their case they run a site that is probably under constant attack by the "hired goons", so they're going to need to have more moving parts than others. Plus they will want to optimise for minimal development time (it's a hobby) so just adding another tried and trusted system into the stack to do something you need makes sense.

starfallg 5y ago
>In their case they run a site that is probably under constant attack by the "hired goons", so they're going to need to have more moving parts than others. That's taken care of by the DDoS-Guard system they placed fronting their infrastructure. The design of their system has to take this into account, but that is mainly on a IP and DNS level. The design of their stack behind the loadbalancer is mainly driven by their functional and non-functional requirements, rather than by the need to prevent DDoS attacks.
- radicalbyte 5y ago
  The layering - defence in depth - is very much a security consideration. Especially if you're building a pure request/response/sync system you need that. Or you decouple with a queue for mutations and avoid a lot of issues.
  - starfallg 5y ago
    That may be in terms of managing general security, especially with regards to the attack surface of the solution, but here we are talking about DDoS, which is mostly a separate topic and handled on the network level (for volumetric attacks) and load-balancer level (for non-volumetric attacks) or a combination of both.
maxk42 5y ago
lol > In general the less moving parts you have in a system the more reliable, secure, efficient and cheaper the system becomes. 100% agreed. This is not my first high-traffic site, nor even the highest. (I built the analytics system for a an Alexa top-10 site in 2010, reaching some 30 billion writes / day off of a mere 14 small ec2 instances.) I've never seen a k8s implementation in production that was necessary. I will note that my Alexa-2k site is also a personal site (no revenue) and under constant attack. In fact we frequently suffer DDOSes that we don't even notice until reviewing the logs later because it doesn't suffer any latency under pressure.
- radicalbyte 5y ago
  Interesting, wouldn't mind having a chat outside of HN if you're interested (see my profile for mail). I've spent much of my career working on systems with active users from the hundreds to low thousands, but which process a huge number (50k/sec scale) jobs/tasks. It's a totally different kettle of fish, and if I'm totally honest I'm shocked at how badly "web" scales and how common these naive and super inefficient implementations are (hint: my bare-metal server from 2005 was faster than expensive cloud VMs). Recently I've worked on two high-usage systems (one of which was "handling" 30k requests/second for the first couple of week).
  - deleted 5y ago
    [deleted]
  - golergka 5y ago
    > I've spent much of my career working on systems with active users from the hundreds to low thousands, but which process a huge number (50k/sec scale) jobs/tasks. MMO games, by any chance?
- Folcon 5y ago
  Would you mind outlining your approach? Really interested to see how you think about this sort of thing =)...
  - 5y ago

tristan9 5y ago

Hi, we're trying to lower the requests:pageview ratio in general, but for what it's worth this article essentially: - ignores the vast majority of "image serving" (most is handled by DDG and our custom CDN) - the JS fragments thankfully should load only on first visit and then get aggressively cached by DDG/your browser One of the pain points is that there are a lot of settings for users to decide what they should or shouldn't see (content rating, original language of origin, search tags, etc) and some are already specifically denormarlized (when querying chapter entities, ES indices for those contain some manga-level properties to avoid needing to dereference that first too) -- however this also makes caching substantially less efficient in many places, alas Thanks!

jiggawatts 5y ago
Hi, I'm a performance tuning expert, and this thread piqued my interest. The first thing that I noticed is that even with caching enabled, you're loading "too much data". After loading the main page and then clicking one of the tiles, there are several JSON API calls. Here's an example, 195 kB transferred (528 kB size): https://api.mangadex.org/manga/bbaa17c4-0f36-4bbb-9861-34fc8fdf20fc/feed?limit=96&includes[]=scanlation_group&includes[]=user&order[volume]=desc&order[chapter]=desc&offset=0&contentRating[]=safe&contentRating[]=suggestive&contentRating[]=erotica&contentRating[]=pornographic https://api.mangadex.org/manga/bbaa17c4-0f36-4bbb-9861-34fc8... Oof. Half a megabyte of JSON! Ignore the network traffic for a moment, because GZIP does wonders. The real problem is that generating that much JSON is very "heavy" on servers. Lots and lots of small object allocations, which gives the garbage collector a ton of work to do. It's also expensive to decode on the browser for similar reasons. On my computer, this took a whopping 455ms to transfer, nearly half a second. That results in a noticeable latency hit to the site. In my consulting gig I always give developers the same advice: "Displaying 1 kilobyte of data should take roughly 1 kilobyte of traffic". In other words, there's isn't 500 KB of text anywhere on that page! A quick cut & paste shows about 8 KB of user-visible text in the final HTML rendering. That's a 1:60 ratio of content-to-data, which is very poor. I bet that behind the scenes, this took a heck of a lot more back-end network traffic and in-memory processing to generate. Probably tens to hundreds of megabytes of internal traffic, all up. This is one of the core reasons most sites have difficulty scaling, because for every kilobyte of content output to the screen, they're powering through megabytes or even gigabytes of data behind the scenes. Can this API query be cut down to match what's displayed on the screen? Can it be cached for all users? Can it be cached precompressed? Etc...

polote 5y ago

I don't know about you, but they have 42 average Page-views per visit (HN has 3) so Alexa rank is going to be biased

tyingq 5y ago

>A fresh load of the home page generates over 100 requests. I see 17 requests, all over either h2 or h3. 4 of them JS, and 2 images.

maxk42 5y ago
Then you're not doing a fresh load of the page. There are over 30 images visible on the front page, so your measure doesn't pass the smell test, does it?
- tyingq 5y ago
  >Then you're not doing a fresh load of the page Nope. Different problem. The article was linked to a page under the domain "mangadex.dev". Without any other context, I had assumed "home page" meant http://mangadex.dev http://mangadex.dev , or what I got when clicking "Home" on the linked article. Apparently not.