25 ms·

MangaDex infrastructure overview

567 points by m45t3r 5y ago

jiggawatts 5y ago
What kills me is that this was a rather pedestrian outcome on a much cheaper 2-core virtual machine back in 2007 or so. I easily got 3K requests / sec out of my laptop at the same time, and it was not a trivial app! People's expectations have shifted so much it's absurd. If you look at the TechEmpower benchmarks, ordinary VMs can easily push 100K requests per second, no sweat, even with managed languages. Trivial stuff like static content being treated as static content (files on the disk!) not as a distributed cache in front of a database can do wonders. Am I just old and jaded?
- genewitch 5y ago
  In 2011 a company i contracted for was testing some new dell 1U servers with around 1-2TB of ram. There was a postgres database with 4000qps that could fit into tmpfs, and so i restricted postgres to 640Kb of memory and we got replication working, it took about 6 hours of babysitting. We threw the switch and watched as postgres, with 640Kb of ram and a tmpfs backed store proceeded to handle all of the query traffic. There were some stored procedures or something that were long-querying or whatever - i'm not a DB person at all, so we switched back to the regular production server about 8 minutes later. Yes, we did it in production.
  - bagels 5y ago
    For real, 640 kilobits?
    - sigstoat 5y ago
      K isn't the abbreviation for kilo, so if you're going to rag on the fellow for the 'b', then you should at least be asking what a Kelvin*bit is.
      - iandinwoodie 5y ago
        “The binary meaning of the kilobyte for 1024 bytes typically uses the symbol KB, with an uppercase letter K.” [0] 0. https://en.m.wikipedia.org/wiki/Kilobyte https://en.m.wikipedia.org/wiki/Kilobyte
      - jmiserez 5y ago
        640KiB is very little and I'm wondering if it's a typo, given that the servers had 1-2TiB available. Postgres 9.0 released in 2010 already had 32MiB as the default for shared_buffers (with a minimum of 128KiB): https://www.postgresql.org/docs/9.0/runtime-config-resource.html https://www.postgresql.org/docs/9.0/runtime-config-resource.... and 8.1 released in 2005 used 8MB (1000*8KiB): https://www.postgresql.org/docs/8.1/runtime-config-resource.html https://www.postgresql.org/docs/8.1/runtime-config-resource....
        sigstoat 5y ago
        i interpreted it as "we wanted to turn the shared buffers ~off, but in a hilarious way that would suggest to someone reading the configuration file that something was going on" (bill gates, mumble mumble) but, wtf do i know, i'm the crazy guy who tries to interpret comments generously.
        genewitch 5y ago
        Yes, it was a direct reference to Bill Gates "640 kilobytes is enough for anyone" and i typed the comment right before i fell asleep.
      - bagels 5y ago
        The question was more about the kilo part, even though I didn't clarify. Seems orders of magnitude too small?
  - winrid 5y ago
    Postgres handles low memory situations well. It'll kill memory intensive queries before it crashes. I wonder if your application was getting a lot of errors back instead of successful queries :)
    - genewitch 5y ago
      the application performed fine, even though we made the switch around 15:00 PST. The DBA was concerned because of the few long queries. Obviously the tmpfs was doing the heavy lifting, there - and if i had to do a postmortem, i'd wager that filling the OS caches was the main reason the long queries took so long. We didn't do any sort of performance tracing. The main purpose was to show that these $35k servers could essentially replace the older machines if need be, even though the old ones had FusionIO. I just removed the middleman of the PCIe bus between the application and the memory. It was a near constant argument on the floor about whether or not we could feasibly switch to SSDs in some configuration over spinning rust or even FusionIO, i wanted a third option. Basically, serve out of registered, ECC memory in front, replicate to the fusionIO and let those handle the spindled backups, which iirc was a pain point.
- huijzer 5y ago
  To add to that, in 2005, a Cloudflare engineer showed that you can receive 1 million packages per second (https://blog.cloudflare.com/how-to-receive-a-million-packets/ https://blog.cloudflare.com/how-to-receive-a-million-packets...). Without processing though.
- kragen 5y ago
  I think even a distributed cache in front of a database shouldn't have any trouble handling 2000 requests per second. The issue is not really the number of requests per second, probably, but the number of bytes, which they don't talk about at all in the article; reading manga with no ads is a pretty static kind of application, which could be satisfied amply with a web browser or even a much simpler program loading images from a filesystem directory. Valgrind claims httpdito runs a few thousand instructions per request, but that's not really accurate; what happens is that the kernel is doing all the work. httpdito on Linux can handle about 4000 requests per second per core, nearly a million clock cycles per request, almost all of which is in the kernel. Of course it doesn't ship its logs off to Grafana. In fact, it doesn't have logs at all. But it would work fine for reading manga.
  - antupis 5y ago
    Also that 2000 request per second has to happen 24/7 not only quick demo session.
    - kragen 5y ago
      httpdito is nothing if not consistent in its performance. It doesn't have enough state to have many ways to perform well at first and then more poorly later, or for that matter vice versa. (Not saying it couldn't happen, but it's not that likely.) Linux is pretty good about consistent performance, too, though it has more state.
  - buzer 5y ago
    > The issue is not really the number of requests per second, probably, but the number of bytes, which they don't talk about at all in the article; reading manga with no ads is a pretty static kind of application, which could be satisfied amply with a web browser or even a much simpler program loading images from a filesystem directory. I assume they are talking about their more dynamic content serving in this post (for things like search, tracking which chapters are read, new chapter listing based on what user follows etc.). They have a custom CDN that is hosted by volunteers to serve the images for the manga pages. They provide some metrics for that at https://mangadex.network https://mangadex.network, there are also some older screenshots where they hit 3.2GB/s.
- deleted 5y ago
  [deleted]
- z3t4 5y ago
  2k sockets on a test bed vs 2k real user request in production is very different. I doubt you ran a top 1000 Alexa site on your laptop. Today we need to deal with SSL which eats from the performance budget.
  - jiggawatts 5y ago
    > SSL which eats from the performance budget. That was a short-lived thing, and has now become a myth perpetuated by companies like Citrix and F5 that sell "SSL offload" appliances for $$$. Have you benchmarked the overhead of TLS? In my experience, a single CPU core can easily put out multiple gigabytes of AES-256 (tens of gigabits). This benchmark shows 3 GB/s (24 Gbps) for recent AMD CPUs, and nearly 40 Gbps per core for an Intel CPU: https://calomel.org/aesni_ssl_performance.html https://calomel.org/aesni_ssl_performance.html A multi-core server is very unlikely to have more than a 1-5% overheard due to TLS. Even connection set up is a minor overhead with elliptic curve certificates. This is thanks to the AES offload instructions, which are present in all server CPUs made any time in the last 5-7 years or so. As long as the modern Galois Counter Mode (GCM) is used with AES, performance should be great. Meanwhile, Citrix ADC v13 with a hardware "SSL offload card" actually slows down connections! I had a very hard time getting more than 600 Mbps through one. It seems to be the way the ASIC offload chip is architected: it seems to use a large number of slow cores, a bit like a GPU. This means that any one TLS stream will have its bandwidth capped!
    - tpetry 5y ago
      The problem with these benchmarks is they measure the bandwidth you can push through an established tls connection. Try to build 2000 new tls connections a seconds (yes many are still active and dont need to be restarted) that is what is the really slow part. Not sending the data over already established channels.
      - lossolo 5y ago
        This exactly. Most CPU time in our Haproxy is spent on crypto/new TLS sessions handling thousands of new connections per second.
- adreamingsoul 5y ago
  No. I think you have a healthy perspective and we should all be questioning if current trends are beneficial/sustainable. I haven’t read the article, but the headline alone to me seems alarming, $1,500 a month is a lot of money for only 2k rps.
  - reilly3000 5y ago
    Maybe its a lot of money just for the web servers, but for the entire infrastructure stack its pretty reasonable IMHO.
  - rk06 5y ago
    There is more to it that http request response. Mangadex also need to store a lot of images and distribute them.
    - hnlmorg 5y ago
      CDNs have already solved this problem and are much cheaper than $1500/month. I've ran far more complex sites with much higher traffic for less.
      - rk06 5y ago
        Mangadex can't use cloudflare because of privacy reasons. They may be facing similar issues with other popular CDNs. I am sure they must be using some kind of CDN for sure, however, those options are unlikely to be free
        hnlmorg 5y ago
        Privacy reasons? It's all static content that is publicly accessible. I don't understand what the privacy reasons could be under this context. Are they worried about CDNs logging the images their visitors access? Seems like an absurd edge case to worry about in my opinion. > however, those options are unlikely to be free I wasn't even talking about free CDNs :)
- willvarfar 5y ago
  Excellent post, good technical content, amazing feat. That said, I echo that the amazing feat is that they can fit modern inefficient tool choices with poor mechanical sympathy into that budget. The last decade of web-dev tooling has been pushing the TCO of systems through the roof and this post is all about how to struggle against that whilst using those tools. If they went old-school they'd get another order of magnitude savings. Many veterans know of systems doing 10x that in 10x less cost. Remember C10K was in 1999.
  - vymague 5y ago
    > If they went old-school they'd get another order of magnitude savings. Many veterans know of systems doing 10x that in 10x less cost. Remember C10K was in 1999. How to learn more about the old-school way without getting a job related to it? Like, topic or book recommendations.
    - thoughtFrame 5y ago
      I'm also curious about this, since in many areas of computing (not only webdev) the old-school guys take some stuff as so obvious that they don't even bother writing about it or explaining it beyond "This is obviously possible, dude". They know how to achieve this level of performance, but for everyone else, we have to cobble together fragmented insights. So if anyone out there reads this and thinks like the GP, please do write about it ;)
      - hsn915 5y ago
        What topics specifically are you interested in? And where do all the people like you hang out? I'm not an old-school guy by any means .. but I might have something to contribute.
        vymague 5y ago
        > What topics specifically are you interested in? Well, for example, what's the old-school alternative to mangadex's solution? > And where do all the people like you hang out? We are here on HN.
- colesantiago 5y ago
  Not just you, but if it works for them, that's completely fine. But there are many ways to achieve 20K RPS without this type architecture and especially without k8s, for less than $1,500.
  - bob1029 5y ago
    >20k RPS. If this metric is what you are chasing, there are ways to reliably break 1 million RPS using a single box if you don't play the shiny BS tech game. The moment you involve multiple computers and containers, you are typically removed from this level of performance. Going from 2,000 to 2,000,000 RPS (serialized throughput) requires many ideological sacrifices. Mechanical sympathy (ring buffers, batching, minimizing latency) can save you unbelievable amounts of margin and time when properly utilized.
    - nine_k 5y ago
      I frankly don't see where containers could lower the performance. Basically a container is a glorified chroot. It has the same networking unless you asked for isolation, then packets have to follow a local (inside the host) route. It has exactly no CPU or kernel interface penalty. Maybe you wanted to say about container orchestration like k8s, with its custom network fabric, etc.
      - bob1029 5y ago
        Even loopback is significantly slower than a direct method invocation.
      - krageon 5y ago
        > I frankly don't see where containers could lower the performance. Have you seen most k8s deployments? It's not the containers, it's the thoughtspace that comes with them. Even just using bare containers invites a level of abstraction and generally comes with a type of developer that just isn't desirable.
        5y ago
- deleted 5y ago
  [deleted]
- cthalupa 5y ago
  You're not alone in making comments like this, and as I read them I'm not sure if people are missing the point or being disingenuous. Much of the work I've done professionally has happened in the webhosting space, and I'm not a new hand at it - the first "professional" website I ever ran was hosted off of a machine running on a Quantum R5000. I have served (and still serve) plenty of static content as files off the disk. My own first impulse when building anything is to use as few moving parts and as simple of a setup as possible. Requests per second are not a good metric for the amount of work being done by a system, because not all requests are made equal. You say you got 3k requests out of your laptop and that it wasn't a trivial app, but you have only provided a trivial amount of information. Taking a quick look at the features provided by the site, they offer quite a bit of flexibility on both what is and how it's displayed. Filtering options based on original and translated language, adult content filtering, multiple types of search, request routing to low quality images to save on data, category and tag filtering, tracking of what you have read and what your progress is on it, follows and notifications, permissions systems for uploading and updating content, etc. This is all on top of the basic "Display images and metadata about the images" functionality. They have concerns around privacy and being able to withstand active attackers due to the content they host. They have their own requirements about logging, analytics, etc. Their own concerns about data availability. You could not meet the design requirements of this website and serve 2k requests per second on a 2 core VM in 2007. I'm not saying there aren't inefficiencies in their architecture and further places where they could save money/increase performance/etc., but acting like two cores of lower-clockrate lower-ipc compute from a decade and a half ago could do all of the work needed to support the features and design requirements they have for this website is pretty disparaging towards the people who built this infrastructure.
snypher 5y ago
>more than 10 million unique monthly visitors >our ~$1500/month budget I understand not wanting to show ads, but is there no way for the users to contribute to hosting costs?
- whateveracct 5y ago
  A $5/mo premium plan would break even so quickly
  - tommica 5y ago
    Premium plan on content that can be considered as dubious in copyright context? Seems like a quick way to get shut down.
    - hrnn 5y ago
      premium plan on a virtual badge, NFT or whatever crap you want. content would still be freely available for everybody but the infra costs would be a bit less.
  - novok 5y ago
    Or even a patreon style 'you get nothing but a supporter badge' with that kind of traffic levels.
- m45t3r 5y ago
  There is the MangaDex@Home, where users can serve part of disk space/bandwidth to help serve (mainly old) manga chapters. It does need to be something that is running 24/7 (e.g.: not a PC that is shutdown frequently), so something like a VPS or a service is recommended.
  - AviKav 5y ago
    Virtually every chapter is served via MD@H now. Client doesn't really need much availability, as long as it can do a graceful shutdown. Even in the event of a sudden shutdown, the trust penalties are much lower than H@H and in practice go away after a trickle of traffic to raise your score
deathanatos 5y ago
It's a nice article, I guess, but the site is down (the one discussed in the article, not the blog post itself) for me.
- Semaphor 5y ago
  It works fine for me: https://mangadex.org/ https://mangadex.org/
- bytearray64 5y ago
  You probably have Verizon - they've started null routeing traffic to sites "like this". https://old.reddit.com/r/mangadex/comments/nvj7qf/is_verizon_blocking_mangadex/ https://old.reddit.com/r/mangadex/comments/nvj7qf/is_verizon...
  - ApolloFortyNine 5y ago
    Ah shucks I thought we had mostly avoided that stuff in the US. I'm guessing though they're using some old spam ip/block though, there's a lot more obvious piracy sites then a Manga site. For instance, I can access all the major torrent sites.
    - bytearray64 5y ago
      They also block nyaa, an Anime/Manga focused tracker. It's not a very aggressive list though, as you're right that major torrent sites are still accessible.
  - deathanatos 5y ago
    Huh, right you are, on both accounts, it seems. That's disappointing. If only I had some choice to ISPs, then I could express my disappointment by voting with my wallet…
    - bebna 5y ago
      Just call their service often enough, and tell them internet isnt working.
urlgrey 5y ago
I'm amazed that their architecture doesn't include a CDN. These days I expect nearly all high traffic websites to make use of a CDN for all kinds of content, even content that's not cached. They cited Cloudflare not being used due to privacy concerns. It'd be interesting to hear more about that, as well as why other CDNs weren't worth evaluating too.
- Sebguer 5y ago
  I think the usual argument re: Cloudflare on the privacy front is the fact that they pretty aggressively fingerprint users, and will downgrade or block traffic originating from VPNs or some countries. This is a natural side effect of those things often being tied to abusive traffic, and a lot of it is likely configurable (at least on their paid plans) but it often comes up around this.
- ev1 5y ago
  It's effectively a warez site. There's a reason why they host in the places they do and can't be too picky about providers. CF will also pass through things like DMCAs easily. Based on their sidebar, it's probably hosted at Ecatel or whatever they are called now (cybercrime host) via Epik as a reseller, the provider famous for hosting far-right stuff.
  - rovr138 5y ago
    What’s the reason behind where they host and having issues with providers? I haven’t heard this before Regarding DMCA’s, as an entity doing business where they’re legal, what should they do as a middle man?
    - ev1 5y ago
      > Regarding DMCA’s, as an entity doing business where they’re legal, what should they do as a middle man? Don't use them and instead have your middleman be in a country that ignores intellectual property rights and copyright? I'm not saying CF is wrong to pass them through. I'm just saying CF is not the right choice for a warez site for longevity.
mdoms 5y ago
I loaded the front page of Mangadex and it made 114 web requests including 10 first-party XHR requests, 30(!!!!) Javascript resource requests and somehow 4 font requests, without me interacting with the page. Clicking one of the titles on the front page resulted in nearly 40 additional requests. Perhaps if you are limited by requests per second you could consider how many requests a single user is making per interaction, and if this is a reasonable number. The website is impressively fast though, I'll give you that.
- m45t3r 5y ago
  They're probably more limited by bandwidth than requests per second, but anyway you look the number of requests are still impressive considered the budget. BTW, the site is not just fast: they serve images on high quality (same as the original [1], that can be multiple MBs per page [2]) at an pretty impressive speed too. [1]: before someone asks why they don't optimize the images, this is by design since they want to serve high quality images. There is an optional toggle to reduce the image size, but this is disabled by default. [2]: for those not familiar, the average number of pages on a manga is something like ~20, and this can be read in ~5 minutes depending on the density of the text. So you can easily consume 50MB+ per chapter.
  - deleted 5y ago
    [deleted]
- watermelon0 5y ago
  Frontend framework they use (NUXT) uses code splitting [1], which means that: - first request is fast, because you only need to download chunks required for a single page/controller (and you prefetch others in the background) - changing some parts of codebase requires to re-download only affected chunks, instead of the whole bundle [1] https://www.telerik.com/blogs/what-you-should-know-code-splitting-nuxtjs https://www.telerik.com/blogs/what-you-should-know-code-spli...
deleted 5y ago
[deleted]
jhgg 5y ago
They mention a $1,500 budget per month but then omit things critical to understanding how they achieve that cost point. What is actually more interesting is to understand what portion is spent on servers versus bandwidth - and what hardware configuration they use to host the site. For example, Is $1,500/mo just paying for colo costs + bandwidth, with already owned recycled hardware (think last gen hardware that you can get at steep discounts from eBay / used hardware resellers...) That would have been way more interesting to know given the blog title than the choice of infrastructure software they use.
chime 5y ago
Not familiar with the project but it is great to see a counterpart to over-provisioned enterprise infrastructure. $10 in 2021 can do what $100 in 2011 did, what $1000 in 2001 did, and that is not solely due to hardware. Well-designed deployments of K8s, KVM/LXC, Ceph, LBs like this project can handle so much more traffic than poorly configured Wordpress storefronts. They're using battle-tested tech from Redis and RabbitMQ to Ansible and Grafana. Nothing super fancy, nothing used just for the sake of being modern. Not sure how long it took them to end up with this architecture but it doesn't look like a new dev would have a hard time getting familiar with how everything works. Would definitely like to hear more about their dev environment, how it is different from prod, and how they handle the differences.
- novok 5y ago
  I think enterprise and more optimize for business flexibility and ability to A/B test very rapidly vs a finely crafted piece of efficiency, for better or worse. The people behind this probably do this for their day job, or are teens that are about to do it for their day job.
  - chime 5y ago
    I agree with you. I mostly work in enterprise and understand that it has different needs and ROI requirements. However, my personal mindset is that computers and networks are really really fast now and it's a tragedy that most of these gains are nullified due to unoptimized layers of abstraction or over-architecting. So it's a welcome sight to read about well-designed infrastructure like this.
    - novok 5y ago
      It happens because business are optimizing for resources that are ultimately more expensive or slower, which is staffing levels and the ability to respond to the market so the business can grow or survive longer. Inefficient computing architecture as a side effect is a worthwhile tradeoff in light of that to them. But as a craftsman, it is definitely nice :)
- 5y ago
latch 5y ago
I've done things at scale (5-10K req/s) on a budget ($1000 USD) and I've done things at much smaller scales that required a much larger budget. _How_ you hit scale on a budget is one part of the equation. The other part is: what you're doing. Off the top of my head, the "how" will often involve the following (just to list a few): 1 - Baremetal 2 - Cache 3 - Denormalize 4 - Append-only 5 - Shard 6 - Performance focused clients/api 7 - Async / background everything These strategies work _really_ well for catalog-type systems: amazon.com, wiki, shopify, spotify, stackoverflow. The list is virtually endless. But it doesn't take much more complexity for it to become more difficult/expensive. Twitter's a good example. Forget twitter-scale, just imagine you've outgrown what 1 single DB server can do, how do you scale? You can't shard on the `author_id` because the hot path isn't "get all my tweets", the hot path is "get all the tweets of the people I follow". If you shard on `author_id`, you now need to visit N shards. To optimize the hot path, you need to duplicate tweets into each "recipient" shard so that you can do: "select tweet from tweets where recipient_id = $1 order by created desc limit 50". But this duplication is never going to be cheap (to compute or store). (At twitter's scale, even though it's a simple graph, you have the case of people with millions of followers which probably need special handling. I assume this involves a server-side merge of "tweets from normal people" & RAM["tweets from the popular people"].)
- 29athrowaway 5y ago
  Try to convert as much content as you can into static content, and serve it via CDN. Then, use your servers only for dynamic stuff. Also, put the browser to work for you, caching via Cache-Control, ETag, etc. Only then, optimize your server...
- ignoramous 5y ago
  > Twitter's a good example. Mike Cvet's talk about Twitter's fan-in/fan-out problem and its solution makes for a fascinating watch: https://www.youtube-nocookie.com/embed/WEgCjwyXvwc https://www.youtube-nocookie.com/embed/WEgCjwyXvwc
  - Cipater 5y ago
    I appreciate the no-cookie embed. Learned something new today.
robertwt7 5y ago
I had nothing but respect for the whole team. Dedicating their time to build everything from scratch, not to mention that they maintain everything for free.. It's a cool project, not sure if there's a way for anyone to contribute. I"ll join the discord afterwork to see if they need any extra hand. Gee, how do these people find other people online to work on all of the cool projects. I would love to join rather than playing games after WFH on the same pc over and over again lol
- codewithcheese 5y ago
  Find cool project. Contribute. :)
  - robertwt7 5y ago
    I do on some open source projects on github. Sorry what I meant is not just some open source projects but working products like this driven by volunteers / teams like theirs.
- IncRnd 5y ago
  Okay, but isn't most of their content stolen? Why would you want to contribute to that?
  - dinobones 5y ago
    Maybe because they enjoy the interesting domain and challenges of the area, look at a project like Dolphin for example. Also, some people hold the view that things like information, media, code can not be “stolen” in the traditional sense, so that further reduces any qualms about associating themselves with it.
  - cyborgx7 5y ago
    No, intellectual "property" can not be stolen. You are thinking of copyright infringement.
    - deleted 5y ago
TekMol 5y ago
My cheap $20/month VPS serves tens of thousands a user per day without breaking much of a sweat. Using a good old LAMP stack (Linux, Apache, MariaDB, PHP). I don't know how many requests per second it can handle. Trying a guess via curl: time curl --insecure --header 'Host: www.mysite.com' https://127.0.0.1 https://127.0.0.1 > test This gives me 0.03s So it could handle about 30 requests per second? Or 30x the number of CPUs? What do you guys think?
- dharmab 5y ago
  Does it serve 20-40 hi resolution images and uploads per user?
  - TekMol 5y ago
    I wanted to start a discussion about how to estimate the number of requests a given server can handle per second. So when I read "x requests/s" I can put that into perspective. But it seems you think I wanted to start a dick measuring contest? If your question is genuine: I would serve images via a CDN. The above timing is for assembling a page by doing an auth check, a bunch of database queries and templating the result.
  - lostmsu 5y ago
    I am not sure how to interpret this para: > In practice, we currently see peaks of above 2000 requests every single second during prime time. That is multiple billions of requests per month, or more than 10 million unique monthly visitors. And all of this before actually serving images. If I am reading that correctly, 2000r/s does not include images, and makes it unclear if $1500/month does.
    - cinntaile 5y ago
      I'm pretty sure that includes images, that's why people visit the site. Prime time happens when a very popular manga gets released at around the same time every week.
  - Hamuko 5y ago
holoduke 5y ago
I am running an app with 10.000 incoming rq/s on AVG. It's running on 8, 8 core Hetzner VMs. Most request are static data calls like images, JSON and text. About 5% is MySQL and other IO operations. I pay about 300 euros a month for this setup. Quite happy with it.
FpUser 5y ago
I wrote business backend server that calculates various things and returns results as json. It serves in average up to 5000 requests/s for about $220CDN / month. Architecture - single executable written in C++ () running on rented dedicated server from OVH. 16 cores, 128GB RAM and couple of SSDs. It can do much higher requests per second wise on simple requests but most common requests are actually heavy iterative calculations so hence the average of 5000 requests/s
jprupp 5y ago
This is complexity for complexity's sake. Pay no attention to the disclaimer at the start of the article. They threw every buzzword-heavy bit of tech they could find at it, creating a Frankenstein monster.
- sofixa 5y ago
  Completely disagree. How would you do it in a simpler way, while keeping the features like redundancy ( including storage), logs, metrics, etc?
  - pahae 5y ago
    Looking at their diagrams it seems that the k8s cluster exists solely to handle their monitoring and logging needs which would be extreme overkill, especially since 18k metrics/samples and 7k logs per second are nothing. Plus you now suddenly need a S3-compatible storage backend for all your logs and metrics. Good thing Ceph comes 'free' with Proxmox, I guess. Deploying an instance of Prometheus with *every host is also unusual, to say the least and I don't quite understand their comment to that. If you don't like a pull-based architecture (which is a valid point) why use one at all!? There are many more push-based setups out there that are simpler to set up and less complex.
    - tristan9 5y ago
      > k8s cluster exists solely to handle their monitoring and logging Does image processing, runs our analytics, runs our Sentry, runs our gitlab-ci runners, and quite a few other things not mentioned expressly > which would be extreme overkill That's an interesting argument against k8s ; if anything I find it much easier to work with -- once accustomed to its idioms, ofc -- than alternatives like dedicated VMs, Docker Swarm etc Getting HA and auto-healing for free is possible without it, of course, but does require much more work, especially if you aim for a somewhat minimised amount of statefulness (as in deviation from the template of your system) Also S3-compatible storage backends are really aplenty, from commercial offering to simpler ones like MinIO. Ceph just happens to be a bit higher of a deployment investment with the benefit of fantastic performance, flexibility and resiliency. Somewhat like k8s itself, it's a bit daunting at first but does actually make things simpler in the long run (imo) > 18k metrics/samples [...] are nothing Well yes and no, the number of metrics isn't relevant per se, but its cardinality is very relevant, and managing that in a single prometheus instance will quickly require some serious vertical scaling, especially if you want to look at data on longer ranges (which, in contrary to logs, we are interested in) > 7k logs per second [...] are nothing That's an interesting take . Surely this isn't a world-record-shattering amount indeed, but no one seems to have such a great non-SaaS-or-cheap solution to storing, sorting and querying this amount of logs either (at the resource efficiency of Loki anyway), so maybe we just have a different set of expectations for log management > If you don't like a pull-based architecture [...] why use one at all!? There are many more push-based setups out there that are simpler to set up and less complex. Are there really? That is non-SaaS and with as widespread 3rd-party software support as Prometheus does? ie great integration with essentially any database, webserver, runtime, OS, etc? Because if we talk only about node metrics like CPU etc then yeah, sure there are plenty of options. But (maybe not so) obviously the diagram showing only node exporter doesn't mean that this is the only integration we use -- we collect prometheus metrics for MySQL, PHP-FPM, Varnish, Nginx, HAProxy, Elasticsearch, Redis, RabbitMQ etc (essentially every single piece of software we use). Fwiw I found very little in the way of open-source solutions to that problem that ticked as many boxes as Prometheus. As for "simpler to set up and less complex", both Cortex and Loki would be really annoying to manage outside of Kubernetes, I'll happily give you that. But... being able to easily deploy and manage such systems once you have Kubernetes is precisely one of the reasons to use it. You can't say it's complex to deploy itself but then ignore the fact that it largely outweighs this by making reliable operation of complex-but-powerful software on top it, that is precisely one of the upsides of using it in the first place :)
angarg12 5y ago
> The only missing bit would be the ability to replicate production traffic, as some bugs only happen under very high traffic by a large number of concurrent users. This is however at best difficult or nearly impossible to do. Not sure I'm missing something here. Surely you could sample some prod traffic and then replay it with one of the many load test tools out there. You might lose in the geographical distribution, but load testing a web server with 2k TPS sounds a bit trivial.
hsn915 5y ago
I don't understand. Why is 2k requests/sec supposed to be massive? Try this yourself: write a simple web server in Go, host it on a cheap VPS provider, let's say at the option that costs $20/mo. Your website will be able to handle more than 1k/s requests with hardly any resource usage. ok, let's assume you're doing some complicated things. So what? You can scale vertically, upgrade to the $120/mo server. Your website now should be able to comfortably handle 5k req/s Looking at the website itself, mangadex.org, it doesn't even host the manga itself. The whole website is just an index that links to manga on external websites. All you are doing is storing metadata and displaying it as a webpage. The easiest problem on the web. So, I really don't understand the premise behind the whole post. The problem statement is: > In practice, we currently see peaks of above 2000 requests every single second during prime time. This is great in terms of success as a website, but it's underwhelming in terms of describing a technical problem.
- colesantiago 5y ago
  I agree, just had to read the article again, and took it as a fancy way of wasting money really.
- AviKav 5y ago
  The only releases that link to external websites are the ones from sites such as MangaPlus and BiliBili (And delayed releases if you count those)
- Tenoke 5y ago
  >Looking at the website itself, mangadex.org, it doesn't even host the manga itself. They do seem to. Clicking on a random manga on there the images are hosted on their server[0]. Also I guess some of those are much bigger images which is less trivial to serve at that rate than a 10kb static page. 0. blob:https://mangadex.org/e78bd61a-e761-4a73-a27c-5f58394e7ea4 https://mangadex.org/e78bd61a-e761-4a73-a27c-5f58394e7ea4
  - akx 5y ago
    Blob links are scoped to your browser tab, they're not real internet URLs.
germanjoey 5y ago
Many manga fans have a love/hate relationship with mangadex. On one hand, it's provided hosting for countless hours of entertainment over the years. Their "v3" version of the site was basically perfect from a usability point of view, to the point that the entire community chose to unite itself under its flag. On the other hand, directly because of the above, their hasty self-inflicted take down earlier this year nearly killed the entire hobby. Many series essentially stopped updating for the ~5 months the site was down, and many more are likely never coming back again. The decision to suddenly take the site down for a full site rewrite feels completely inexplicable from the outside. (A writeup the above or the previous one[1], both of which read like they were written by a Google Product Manager, especially don't help as they conspicuously avoid any comment to the one question on everyone's mind: "leaving aside the supposed security issues with the backend, why on earth also rewrite and redesign the entire front end from scratch at the same time?") [1] https://mangadex.dev/why-rebuild/ https://mangadex.dev/why-rebuild/
- nvarsj 5y ago
  I didn't get that take at all from the why-rebuild link. It seems reasonable to me - legacy code base, hard to maintain, with security problems that led to the massive data leak a while back. They also don't owe anything to anyone, and as a hobbyist project, they wanted to try something new. I'm impressed as they seem to have managed it - and the new site feels a lot more responsive than the old one.
- creshal 5y ago
  Yeah. This whole mess pushed me to moving everything I had (or could remember, anyway) to Tachiyomi¹, so I can hop between hosting websites freely without losing progress or access to old chapters (as long as I don't run out of local storage). And while it works fine for reading, it kills any interaction with the hosting sites. No chance for monetization, socialization or anything else that can help sites survive long-term. [1] https://tachiyomi.org/ https://tachiyomi.org/
  - m45t3r 5y ago
    Before MangaDex we had Batoto (the old Batoto before some sketchy company bought the name), that was kinda of the same: serving high quality manga for most scanlators that wanted it (and also avoiding hosting pirated chapters from official sources, so kinda the same as MangaDex nowadays). As far I remember Batoto closed because of pressure from companies and also because of the high costs related to run the site. So yeah, considering how fragile maintaining a site like this is, it is always a good idea to sync your progress in a third party so it is easier to migrate if something goes wrong. > And while it works fine for reading, it kills any interaction with the hosting sites. No chance for monetization, socialization or anything else that can help sites survive long-term. BTW, MangaDex doesn't have monetization because it is strict a hobby and also because it is a gray area to monetize about this kinda of work [1]. Also, their Tachiyomi client is official (MangaDex v5 API was tested primarily via their Tachiyomi client before they finished the Web interface). [1]: both for companies (that has the copyright from the works hosted on those sites) and the scanlators (the fans that does actual work of translating those chapters). Sites that host those chapters and monetize are pretty much monetizing on work from other people.
maxk42 5y ago
I run an Alexa top-2000 website. (Mangadex is presently at about 6000.) I spend less than $250 a month. I have loads and loads of thoughts about what they could be doing differently to reduce their costs but I'll just say that the number one thing Mangadex could be doing right now from a cursory glance is to reduce the number of requests. A fresh load of the home page generates over 100 requests. (Mostly images, then javascript fragments.) Mangadex apparently gets over 100 million requests per day. My site - despite ostensibly having more traffic - gets fewer than half that many in a month. (And yes, it's image-heavy.) A couple easy wins would be to reduce the number of images loaded on the front page. (Does the "Seasonal" slider really need 30 images, or would 5 and a link to the "seasonal" page be enough? Same thing with "Recently Added" and the numbers of images on pages in general.) The biggest win would probably be reducing the number of javascript requests. Somehow people seem to think there's some merit to loading javascript which subsequently loads additional javascript. This adds a tremendous amount of latency to your page load and generates needless network traffic. Each request has a tremendous amount of overhead - particularly for dynamically-generated javascript. It's much better to load all of the javascript you need in a single request or a small handful of requests. Unfortunately, this is probably a huge lift for a site already designed in this way, but the improved loading time would be a big UX win. Anyway - best of luck to MangaDex! They've clearly put a lot of thought into this.
- radicalbyte 5y ago
  Do you manage to get as many buzz-words and OSS products into your system as they do? :) In general the less moving parts you have in a system the more reliable, secure, efficient and cheaper the system becomes. In their case they run a site that is probably under constant attack by the "hired goons", so they're going to need to have more moving parts than others. Plus they will want to optimise for minimal development time (it's a hobby) so just adding another tried and trusted system into the stack to do something you need makes sense.
  - starfallg 5y ago
    >In their case they run a site that is probably under constant attack by the "hired goons", so they're going to need to have more moving parts than others. That's taken care of by the DDoS-Guard system they placed fronting their infrastructure. The design of their system has to take this into account, but that is mainly on a IP and DNS level. The design of their stack behind the loadbalancer is mainly driven by their functional and non-functional requirements, rather than by the need to prevent DDoS attacks.
deleted 5y ago
[deleted]
cyberpsybin 5y ago
At some point in new re-design, they started to load full size images for thumbnails. The whole site feels slower due to that. Need an automatic re-scaler service.
- tristan9 5y ago
  Not correct, we generate 2 thumbnails sizes for every cover -- if the site loads full-size anywhere by default (rather than when you expand it) it's definitely a bug!
probotect0r 5y ago
How is the Wireguard VPN set up? Has anyone used Wireguard to set up a VPN into AWS VPCs?
Shadonototro 5y ago
How can this be legal? It's basically pirating content
sxhunga 5y ago
Interesting to know more about news.ycombinator.com !

567 points by m45t3r 5y ago

What kills me is that this was a rather pedestrian outcome on a much cheaper 2-core virtual machine back in 2007 or so. I easily got 3K requests / sec out of my laptop at the same time, and it was not a trivial app! People's expectations have shifted so much it's absurd. If you look at the TechEmpower benchmarks, ordinary VMs can easily push 100K requests per second, no sweat, even with managed languages. Trivial stuff like static content being treated as static content (files on the disk!) not as a distributed cache in front of a database can do wonders. Am I just old and jaded?

genewitch 5y ago
In 2011 a company i contracted for was testing some new dell 1U servers with around 1-2TB of ram. There was a postgres database with 4000qps that could fit into tmpfs, and so i restricted postgres to 640Kb of memory and we got replication working, it took about 6 hours of babysitting. We threw the switch and watched as postgres, with 640Kb of ram and a tmpfs backed store proceeded to handle all of the query traffic. There were some stored procedures or something that were long-querying or whatever - i'm not a DB person at all, so we switched back to the regular production server about 8 minutes later. Yes, we did it in production.
- bagels 5y ago
  For real, 640 kilobits?
  - sigstoat 5y ago
    K isn't the abbreviation for kilo, so if you're going to rag on the fellow for the 'b', then you should at least be asking what a Kelvin*bit is.
    - iandinwoodie 5y ago
      “The binary meaning of the kilobyte for 1024 bytes typically uses the symbol KB, with an uppercase letter K.” [0] 0. https://en.m.wikipedia.org/wiki/Kilobyte https://en.m.wikipedia.org/wiki/Kilobyte
    - jmiserez 5y ago
      640KiB is very little and I'm wondering if it's a typo, given that the servers had 1-2TiB available. Postgres 9.0 released in 2010 already had 32MiB as the default for shared_buffers (with a minimum of 128KiB): https://www.postgresql.org/docs/9.0/runtime-config-resource.html https://www.postgresql.org/docs/9.0/runtime-config-resource.... and 8.1 released in 2005 used 8MB (1000*8KiB): https://www.postgresql.org/docs/8.1/runtime-config-resource.html https://www.postgresql.org/docs/8.1/runtime-config-resource....
      - sigstoat 5y ago
        i interpreted it as "we wanted to turn the shared buffers ~off, but in a hilarious way that would suggest to someone reading the configuration file that something was going on" (bill gates, mumble mumble) but, wtf do i know, i'm the crazy guy who tries to interpret comments generously.
        genewitch 5y ago
        Yes, it was a direct reference to Bill Gates "640 kilobytes is enough for anyone" and i typed the comment right before i fell asleep.
    - bagels 5y ago
      The question was more about the kilo part, even though I didn't clarify. Seems orders of magnitude too small?
- winrid 5y ago
  Postgres handles low memory situations well. It'll kill memory intensive queries before it crashes. I wonder if your application was getting a lot of errors back instead of successful queries :)
  - genewitch 5y ago
    the application performed fine, even though we made the switch around 15:00 PST. The DBA was concerned because of the few long queries. Obviously the tmpfs was doing the heavy lifting, there - and if i had to do a postmortem, i'd wager that filling the OS caches was the main reason the long queries took so long. We didn't do any sort of performance tracing. The main purpose was to show that these $35k servers could essentially replace the older machines if need be, even though the old ones had FusionIO. I just removed the middleman of the PCIe bus between the application and the memory. It was a near constant argument on the floor about whether or not we could feasibly switch to SSDs in some configuration over spinning rust or even FusionIO, i wanted a third option. Basically, serve out of registered, ECC memory in front, replicate to the fusionIO and let those handle the spindled backups, which iirc was a pain point.
huijzer 5y ago
To add to that, in 2005, a Cloudflare engineer showed that you can receive 1 million packages per second (https://blog.cloudflare.com/how-to-receive-a-million-packets/ https://blog.cloudflare.com/how-to-receive-a-million-packets...). Without processing though.
kragen 5y ago
I think even a distributed cache in front of a database shouldn't have any trouble handling 2000 requests per second. The issue is not really the number of requests per second, probably, but the number of bytes, which they don't talk about at all in the article; reading manga with no ads is a pretty static kind of application, which could be satisfied amply with a web browser or even a much simpler program loading images from a filesystem directory. Valgrind claims httpdito runs a few thousand instructions per request, but that's not really accurate; what happens is that the kernel is doing all the work. httpdito on Linux can handle about 4000 requests per second per core, nearly a million clock cycles per request, almost all of which is in the kernel. Of course it doesn't ship its logs off to Grafana. In fact, it doesn't have logs at all. But it would work fine for reading manga.
- antupis 5y ago
  Also that 2000 request per second has to happen 24/7 not only quick demo session.
  - kragen 5y ago
    httpdito is nothing if not consistent in its performance. It doesn't have enough state to have many ways to perform well at first and then more poorly later, or for that matter vice versa. (Not saying it couldn't happen, but it's not that likely.) Linux is pretty good about consistent performance, too, though it has more state.
- buzer 5y ago
  > The issue is not really the number of requests per second, probably, but the number of bytes, which they don't talk about at all in the article; reading manga with no ads is a pretty static kind of application, which could be satisfied amply with a web browser or even a much simpler program loading images from a filesystem directory. I assume they are talking about their more dynamic content serving in this post (for things like search, tracking which chapters are read, new chapter listing based on what user follows etc.). They have a custom CDN that is hosted by volunteers to serve the images for the manga pages. They provide some metrics for that at https://mangadex.network https://mangadex.network, there are also some older screenshots where they hit 3.2GB/s.
deleted 5y ago
[deleted]
z3t4 5y ago
2k sockets on a test bed vs 2k real user request in production is very different. I doubt you ran a top 1000 Alexa site on your laptop. Today we need to deal with SSL which eats from the performance budget.
- jiggawatts 5y ago
  > SSL which eats from the performance budget. That was a short-lived thing, and has now become a myth perpetuated by companies like Citrix and F5 that sell "SSL offload" appliances for $$$. Have you benchmarked the overhead of TLS? In my experience, a single CPU core can easily put out multiple gigabytes of AES-256 (tens of gigabits). This benchmark shows 3 GB/s (24 Gbps) for recent AMD CPUs, and nearly 40 Gbps per core for an Intel CPU: https://calomel.org/aesni_ssl_performance.html https://calomel.org/aesni_ssl_performance.html A multi-core server is very unlikely to have more than a 1-5% overheard due to TLS. Even connection set up is a minor overhead with elliptic curve certificates. This is thanks to the AES offload instructions, which are present in all server CPUs made any time in the last 5-7 years or so. As long as the modern Galois Counter Mode (GCM) is used with AES, performance should be great. Meanwhile, Citrix ADC v13 with a hardware "SSL offload card" actually slows down connections! I had a very hard time getting more than 600 Mbps through one. It seems to be the way the ASIC offload chip is architected: it seems to use a large number of slow cores, a bit like a GPU. This means that any one TLS stream will have its bandwidth capped!
  - tpetry 5y ago
    The problem with these benchmarks is they measure the bandwidth you can push through an established tls connection. Try to build 2000 new tls connections a seconds (yes many are still active and dont need to be restarted) that is what is the really slow part. Not sending the data over already established channels.
    - lossolo 5y ago
      This exactly. Most CPU time in our Haproxy is spent on crypto/new TLS sessions handling thousands of new connections per second.
adreamingsoul 5y ago
No. I think you have a healthy perspective and we should all be questioning if current trends are beneficial/sustainable. I haven’t read the article, but the headline alone to me seems alarming, $1,500 a month is a lot of money for only 2k rps.
- reilly3000 5y ago
  Maybe its a lot of money just for the web servers, but for the entire infrastructure stack its pretty reasonable IMHO.
- rk06 5y ago
  There is more to it that http request response. Mangadex also need to store a lot of images and distribute them.
  - hnlmorg 5y ago
    CDNs have already solved this problem and are much cheaper than $1500/month. I've ran far more complex sites with much higher traffic for less.
    - rk06 5y ago
      Mangadex can't use cloudflare because of privacy reasons. They may be facing similar issues with other popular CDNs. I am sure they must be using some kind of CDN for sure, however, those options are unlikely to be free
      - hnlmorg 5y ago
        Privacy reasons? It's all static content that is publicly accessible. I don't understand what the privacy reasons could be under this context. Are they worried about CDNs logging the images their visitors access? Seems like an absurd edge case to worry about in my opinion. > however, those options are unlikely to be free I wasn't even talking about free CDNs :)
willvarfar 5y ago
Excellent post, good technical content, amazing feat. That said, I echo that the amazing feat is that they can fit modern inefficient tool choices with poor mechanical sympathy into that budget. The last decade of web-dev tooling has been pushing the TCO of systems through the roof and this post is all about how to struggle against that whilst using those tools. If they went old-school they'd get another order of magnitude savings. Many veterans know of systems doing 10x that in 10x less cost. Remember C10K was in 1999.
- vymague 5y ago
  > If they went old-school they'd get another order of magnitude savings. Many veterans know of systems doing 10x that in 10x less cost. Remember C10K was in 1999. How to learn more about the old-school way without getting a job related to it? Like, topic or book recommendations.
  - thoughtFrame 5y ago
    I'm also curious about this, since in many areas of computing (not only webdev) the old-school guys take some stuff as so obvious that they don't even bother writing about it or explaining it beyond "This is obviously possible, dude". They know how to achieve this level of performance, but for everyone else, we have to cobble together fragmented insights. So if anyone out there reads this and thinks like the GP, please do write about it ;)
    - hsn915 5y ago
      What topics specifically are you interested in? And where do all the people like you hang out? I'm not an old-school guy by any means .. but I might have something to contribute.
      - vymague 5y ago
        > What topics specifically are you interested in? Well, for example, what's the old-school alternative to mangadex's solution? > And where do all the people like you hang out? We are here on HN.
colesantiago 5y ago
Not just you, but if it works for them, that's completely fine. But there are many ways to achieve 20K RPS without this type architecture and especially without k8s, for less than $1,500.
- bob1029 5y ago
  >20k RPS. If this metric is what you are chasing, there are ways to reliably break 1 million RPS using a single box if you don't play the shiny BS tech game. The moment you involve multiple computers and containers, you are typically removed from this level of performance. Going from 2,000 to 2,000,000 RPS (serialized throughput) requires many ideological sacrifices. Mechanical sympathy (ring buffers, batching, minimizing latency) can save you unbelievable amounts of margin and time when properly utilized.
  - nine_k 5y ago
    I frankly don't see where containers could lower the performance. Basically a container is a glorified chroot. It has the same networking unless you asked for isolation, then packets have to follow a local (inside the host) route. It has exactly no CPU or kernel interface penalty. Maybe you wanted to say about container orchestration like k8s, with its custom network fabric, etc.
    - bob1029 5y ago
      Even loopback is significantly slower than a direct method invocation.
    - krageon 5y ago
      > I frankly don't see where containers could lower the performance. Have you seen most k8s deployments? It's not the containers, it's the thoughtspace that comes with them. Even just using bare containers invites a level of abstraction and generally comes with a type of developer that just isn't desirable.
      - 5y ago
deleted 5y ago
[deleted]
cthalupa 5y ago
You're not alone in making comments like this, and as I read them I'm not sure if people are missing the point or being disingenuous. Much of the work I've done professionally has happened in the webhosting space, and I'm not a new hand at it - the first "professional" website I ever ran was hosted off of a machine running on a Quantum R5000. I have served (and still serve) plenty of static content as files off the disk. My own first impulse when building anything is to use as few moving parts and as simple of a setup as possible. Requests per second are not a good metric for the amount of work being done by a system, because not all requests are made equal. You say you got 3k requests out of your laptop and that it wasn't a trivial app, but you have only provided a trivial amount of information. Taking a quick look at the features provided by the site, they offer quite a bit of flexibility on both what is and how it's displayed. Filtering options based on original and translated language, adult content filtering, multiple types of search, request routing to low quality images to save on data, category and tag filtering, tracking of what you have read and what your progress is on it, follows and notifications, permissions systems for uploading and updating content, etc. This is all on top of the basic "Display images and metadata about the images" functionality. They have concerns around privacy and being able to withstand active attackers due to the content they host. They have their own requirements about logging, analytics, etc. Their own concerns about data availability. You could not meet the design requirements of this website and serve 2k requests per second on a 2 core VM in 2007. I'm not saying there aren't inefficiencies in their architecture and further places where they could save money/increase performance/etc., but acting like two cores of lower-clockrate lower-ipc compute from a decade and a half ago could do all of the work needed to support the features and design requirements they have for this website is pretty disparaging towards the people who built this infrastructure.

snypher 5y ago

>more than 10 million unique monthly visitors >our ~$1500/month budget I understand not wanting to show ads, but is there no way for the users to contribute to hosting costs?

whateveracct 5y ago
A $5/mo premium plan would break even so quickly
- tommica 5y ago
  Premium plan on content that can be considered as dubious in copyright context? Seems like a quick way to get shut down.
  - hrnn 5y ago
    premium plan on a virtual badge, NFT or whatever crap you want. content would still be freely available for everybody but the infra costs would be a bit less.
- novok 5y ago
  Or even a patreon style 'you get nothing but a supporter badge' with that kind of traffic levels.
m45t3r 5y ago
There is the MangaDex@Home, where users can serve part of disk space/bandwidth to help serve (mainly old) manga chapters. It does need to be something that is running 24/7 (e.g.: not a PC that is shutdown frequently), so something like a VPS or a service is recommended.
- AviKav 5y ago
  Virtually every chapter is served via MD@H now. Client doesn't really need much availability, as long as it can do a graceful shutdown. Even in the event of a sudden shutdown, the trust penalties are much lower than H@H and in practice go away after a trickle of traffic to raise your score

deathanatos 5y ago

It's a nice article, I guess, but the site is down (the one discussed in the article, not the blog post itself) for me.

Semaphor 5y ago
It works fine for me: https://mangadex.org/ https://mangadex.org/
bytearray64 5y ago
You probably have Verizon - they've started null routeing traffic to sites "like this". https://old.reddit.com/r/mangadex/comments/nvj7qf/is_verizon_blocking_mangadex/ https://old.reddit.com/r/mangadex/comments/nvj7qf/is_verizon...
- ApolloFortyNine 5y ago
  Ah shucks I thought we had mostly avoided that stuff in the US. I'm guessing though they're using some old spam ip/block though, there's a lot more obvious piracy sites then a Manga site. For instance, I can access all the major torrent sites.
  - bytearray64 5y ago
    They also block nyaa, an Anime/Manga focused tracker. It's not a very aggressive list though, as you're right that major torrent sites are still accessible.
- deathanatos 5y ago
  Huh, right you are, on both accounts, it seems. That's disappointing. If only I had some choice to ISPs, then I could express my disappointment by voting with my wallet…
  - bebna 5y ago
    Just call their service often enough, and tell them internet isnt working.

urlgrey 5y ago

I'm amazed that their architecture doesn't include a CDN. These days I expect nearly all high traffic websites to make use of a CDN for all kinds of content, even content that's not cached. They cited Cloudflare not being used due to privacy concerns. It'd be interesting to hear more about that, as well as why other CDNs weren't worth evaluating too.

Sebguer 5y ago
I think the usual argument re: Cloudflare on the privacy front is the fact that they pretty aggressively fingerprint users, and will downgrade or block traffic originating from VPNs or some countries. This is a natural side effect of those things often being tied to abusive traffic, and a lot of it is likely configurable (at least on their paid plans) but it often comes up around this.
ev1 5y ago
It's effectively a warez site. There's a reason why they host in the places they do and can't be too picky about providers. CF will also pass through things like DMCAs easily. Based on their sidebar, it's probably hosted at Ecatel or whatever they are called now (cybercrime host) via Epik as a reseller, the provider famous for hosting far-right stuff.
- rovr138 5y ago
  What’s the reason behind where they host and having issues with providers? I haven’t heard this before Regarding DMCA’s, as an entity doing business where they’re legal, what should they do as a middle man?
  - ev1 5y ago
    > Regarding DMCA’s, as an entity doing business where they’re legal, what should they do as a middle man? Don't use them and instead have your middleman be in a country that ignores intellectual property rights and copyright? I'm not saying CF is wrong to pass them through. I'm just saying CF is not the right choice for a warez site for longevity.

mdoms 5y ago

I loaded the front page of Mangadex and it made 114 web requests including 10 first-party XHR requests, 30(!!!!) Javascript resource requests and somehow 4 font requests, without me interacting with the page. Clicking one of the titles on the front page resulted in nearly 40 additional requests. Perhaps if you are limited by requests per second you could consider how many requests a single user is making per interaction, and if this is a reasonable number. The website is impressively fast though, I'll give you that.

m45t3r 5y ago
They're probably more limited by bandwidth than requests per second, but anyway you look the number of requests are still impressive considered the budget. BTW, the site is not just fast: they serve images on high quality (same as the original [1], that can be multiple MBs per page [2]) at an pretty impressive speed too. [1]: before someone asks why they don't optimize the images, this is by design since they want to serve high quality images. There is an optional toggle to reduce the image size, but this is disabled by default. [2]: for those not familiar, the average number of pages on a manga is something like ~20, and this can be read in ~5 minutes depending on the density of the text. So you can easily consume 50MB+ per chapter.
- deleted 5y ago
  [deleted]
watermelon0 5y ago
Frontend framework they use (NUXT) uses code splitting [1], which means that: - first request is fast, because you only need to download chunks required for a single page/controller (and you prefetch others in the background) - changing some parts of codebase requires to re-download only affected chunks, instead of the whole bundle [1] https://www.telerik.com/blogs/what-you-should-know-code-splitting-nuxtjs https://www.telerik.com/blogs/what-you-should-know-code-spli...

deleted 5y ago

[deleted]

jhgg 5y ago

They mention a $1,500 budget per month but then omit things critical to understanding how they achieve that cost point. What is actually more interesting is to understand what portion is spent on servers versus bandwidth - and what hardware configuration they use to host the site. For example, Is $1,500/mo just paying for colo costs + bandwidth, with already owned recycled hardware (think last gen hardware that you can get at steep discounts from eBay / used hardware resellers...) That would have been way more interesting to know given the blog title than the choice of infrastructure software they use.

chime 5y ago

Not familiar with the project but it is great to see a counterpart to over-provisioned enterprise infrastructure. $10 in 2021 can do what $100 in 2011 did, what $1000 in 2001 did, and that is not solely due to hardware. Well-designed deployments of K8s, KVM/LXC, Ceph, LBs like this project can handle so much more traffic than poorly configured Wordpress storefronts. They're using battle-tested tech from Redis and RabbitMQ to Ansible and Grafana. Nothing super fancy, nothing used just for the sake of being modern. Not sure how long it took them to end up with this architecture but it doesn't look like a new dev would have a hard time getting familiar with how everything works. Would definitely like to hear more about their dev environment, how it is different from prod, and how they handle the differences.

novok 5y ago
I think enterprise and more optimize for business flexibility and ability to A/B test very rapidly vs a finely crafted piece of efficiency, for better or worse. The people behind this probably do this for their day job, or are teens that are about to do it for their day job.
- chime 5y ago
  I agree with you. I mostly work in enterprise and understand that it has different needs and ROI requirements. However, my personal mindset is that computers and networks are really really fast now and it's a tragedy that most of these gains are nullified due to unoptimized layers of abstraction or over-architecting. So it's a welcome sight to read about well-designed infrastructure like this.
  - novok 5y ago
    It happens because business are optimizing for resources that are ultimately more expensive or slower, which is staffing levels and the ability to respond to the market so the business can grow or survive longer. Inefficient computing architecture as a side effect is a worthwhile tradeoff in light of that to them. But as a craftsman, it is definitely nice :)
5y ago

latch 5y ago

I've done things at scale (5-10K req/s) on a budget ($1000 USD) and I've done things at much smaller scales that required a much larger budget. _How_ you hit scale on a budget is one part of the equation. The other part is: what you're doing. Off the top of my head, the "how" will often involve the following (just to list a few): 1 - Baremetal 2 - Cache 3 - Denormalize 4 - Append-only 5 - Shard 6 - Performance focused clients/api 7 - Async / background everything These strategies work _really_ well for catalog-type systems: amazon.com, wiki, shopify, spotify, stackoverflow. The list is virtually endless. But it doesn't take much more complexity for it to become more difficult/expensive. Twitter's a good example. Forget twitter-scale, just imagine you've outgrown what 1 single DB server can do, how do you scale? You can't shard on the `author_id` because the hot path isn't "get all my tweets", the hot path is "get all the tweets of the people I follow". If you shard on `author_id`, you now need to visit N shards. To optimize the hot path, you need to duplicate tweets into each "recipient" shard so that you can do: "select tweet from tweets where recipient_id = $1 order by created desc limit 50". But this duplication is never going to be cheap (to compute or store). (At twitter's scale, even though it's a simple graph, you have the case of people with millions of followers which probably need special handling. I assume this involves a server-side merge of "tweets from normal people" & RAM["tweets from the popular people"].)

29athrowaway 5y ago
Try to convert as much content as you can into static content, and serve it via CDN. Then, use your servers only for dynamic stuff. Also, put the browser to work for you, caching via Cache-Control, ETag, etc. Only then, optimize your server...
ignoramous 5y ago
> Twitter's a good example. Mike Cvet's talk about Twitter's fan-in/fan-out problem and its solution makes for a fascinating watch: https://www.youtube-nocookie.com/embed/WEgCjwyXvwc https://www.youtube-nocookie.com/embed/WEgCjwyXvwc
- Cipater 5y ago
  I appreciate the no-cookie embed. Learned something new today.

robertwt7 5y ago

I had nothing but respect for the whole team. Dedicating their time to build everything from scratch, not to mention that they maintain everything for free.. It's a cool project, not sure if there's a way for anyone to contribute. I"ll join the discord afterwork to see if they need any extra hand. Gee, how do these people find other people online to work on all of the cool projects. I would love to join rather than playing games after WFH on the same pc over and over again lol

codewithcheese 5y ago
Find cool project. Contribute. :)
- robertwt7 5y ago
  I do on some open source projects on github. Sorry what I meant is not just some open source projects but working products like this driven by volunteers / teams like theirs.
IncRnd 5y ago
Okay, but isn't most of their content stolen? Why would you want to contribute to that?
- dinobones 5y ago
  Maybe because they enjoy the interesting domain and challenges of the area, look at a project like Dolphin for example. Also, some people hold the view that things like information, media, code can not be “stolen” in the traditional sense, so that further reduces any qualms about associating themselves with it.
- cyborgx7 5y ago
  No, intellectual "property" can not be stolen. You are thinking of copyright infringement.
  - deleted 5y ago

TekMol 5y ago

My cheap $20/month VPS serves tens of thousands a user per day without breaking much of a sweat. Using a good old LAMP stack (Linux, Apache, MariaDB, PHP). I don't know how many requests per second it can handle. Trying a guess via curl: time curl --insecure --header 'Host: www.mysite.com' https://127.0.0.1 https://127.0.0.1 > test This gives me 0.03s So it could handle about 30 requests per second? Or 30x the number of CPUs? What do you guys think?

dharmab 5y ago
Does it serve 20-40 hi resolution images and uploads per user?
- TekMol 5y ago
  I wanted to start a discussion about how to estimate the number of requests a given server can handle per second. So when I read "x requests/s" I can put that into perspective. But it seems you think I wanted to start a dick measuring contest? If your question is genuine: I would serve images via a CDN. The above timing is for assembling a page by doing an auth check, a bunch of database queries and templating the result.
- lostmsu 5y ago
  I am not sure how to interpret this para: > In practice, we currently see peaks of above 2000 requests every single second during prime time. That is multiple billions of requests per month, or more than 10 million unique monthly visitors. And all of this before actually serving images. If I am reading that correctly, 2000r/s does not include images, and makes it unclear if $1500/month does.
  - cinntaile 5y ago
    I'm pretty sure that includes images, that's why people visit the site. Prime time happens when a very popular manga gets released at around the same time every week.
- Hamuko 5y ago

holoduke 5y ago

I am running an app with 10.000 incoming rq/s on AVG. It's running on 8, 8 core Hetzner VMs. Most request are static data calls like images, JSON and text. About 5% is MySQL and other IO operations. I pay about 300 euros a month for this setup. Quite happy with it.

FpUser 5y ago

I wrote business backend server that calculates various things and returns results as json. It serves in average up to 5000 requests/s for about $220CDN / month. Architecture - single executable written in C++ () running on rented dedicated server from OVH. 16 cores, 128GB RAM and couple of SSDs. It can do much higher requests per second wise on simple requests but most common requests are actually heavy iterative calculations so hence the average of 5000 requests/s

jprupp 5y ago

This is complexity for complexity's sake. Pay no attention to the disclaimer at the start of the article. They threw every buzzword-heavy bit of tech they could find at it, creating a Frankenstein monster.

sofixa 5y ago
Completely disagree. How would you do it in a simpler way, while keeping the features like redundancy ( including storage), logs, metrics, etc?
- pahae 5y ago
  Looking at their diagrams it seems that the k8s cluster exists solely to handle their monitoring and logging needs which would be extreme overkill, especially since 18k metrics/samples and 7k logs per second are nothing. Plus you now suddenly need a S3-compatible storage backend for all your logs and metrics. Good thing Ceph comes 'free' with Proxmox, I guess. Deploying an instance of Prometheus with *every host is also unusual, to say the least and I don't quite understand their comment to that. If you don't like a pull-based architecture (which is a valid point) why use one at all!? There are many more push-based setups out there that are simpler to set up and less complex.
  - tristan9 5y ago
    > k8s cluster exists solely to handle their monitoring and logging Does image processing, runs our analytics, runs our Sentry, runs our gitlab-ci runners, and quite a few other things not mentioned expressly > which would be extreme overkill That's an interesting argument against k8s ; if anything I find it much easier to work with -- once accustomed to its idioms, ofc -- than alternatives like dedicated VMs, Docker Swarm etc Getting HA and auto-healing for free is possible without it, of course, but does require much more work, especially if you aim for a somewhat minimised amount of statefulness (as in deviation from the template of your system) Also S3-compatible storage backends are really aplenty, from commercial offering to simpler ones like MinIO. Ceph just happens to be a bit higher of a deployment investment with the benefit of fantastic performance, flexibility and resiliency. Somewhat like k8s itself, it's a bit daunting at first but does actually make things simpler in the long run (imo) > 18k metrics/samples [...] are nothing Well yes and no, the number of metrics isn't relevant per se, but its cardinality is very relevant, and managing that in a single prometheus instance will quickly require some serious vertical scaling, especially if you want to look at data on longer ranges (which, in contrary to logs, we are interested in) > 7k logs per second [...] are nothing That's an interesting take . Surely this isn't a world-record-shattering amount indeed, but no one seems to have such a great non-SaaS-or-cheap solution to storing, sorting and querying this amount of logs either (at the resource efficiency of Loki anyway), so maybe we just have a different set of expectations for log management > If you don't like a pull-based architecture [...] why use one at all!? There are many more push-based setups out there that are simpler to set up and less complex. Are there really? That is non-SaaS and with as widespread 3rd-party software support as Prometheus does? ie great integration with essentially any database, webserver, runtime, OS, etc? Because if we talk only about node metrics like CPU etc then yeah, sure there are plenty of options. But (maybe not so) obviously the diagram showing only node exporter doesn't mean that this is the only integration we use -- we collect prometheus metrics for MySQL, PHP-FPM, Varnish, Nginx, HAProxy, Elasticsearch, Redis, RabbitMQ etc (essentially every single piece of software we use). Fwiw I found very little in the way of open-source solutions to that problem that ticked as many boxes as Prometheus. As for "simpler to set up and less complex", both Cortex and Loki would be really annoying to manage outside of Kubernetes, I'll happily give you that. But... being able to easily deploy and manage such systems once you have Kubernetes is precisely one of the reasons to use it. You can't say it's complex to deploy itself but then ignore the fact that it largely outweighs this by making reliable operation of complex-but-powerful software on top it, that is precisely one of the upsides of using it in the first place :)

angarg12 5y ago

> The only missing bit would be the ability to replicate production traffic, as some bugs only happen under very high traffic by a large number of concurrent users. This is however at best difficult or nearly impossible to do. Not sure I'm missing something here. Surely you could sample some prod traffic and then replay it with one of the many load test tools out there. You might lose in the geographical distribution, but load testing a web server with 2k TPS sounds a bit trivial.

hsn915 5y ago

I don't understand. Why is 2k requests/sec supposed to be massive? Try this yourself: write a simple web server in Go, host it on a cheap VPS provider, let's say at the option that costs $20/mo. Your website will be able to handle more than 1k/s requests with hardly any resource usage. ok, let's assume you're doing some complicated things. So what? You can scale vertically, upgrade to the $120/mo server. Your website now should be able to comfortably handle 5k req/s Looking at the website itself, mangadex.org, it doesn't even host the manga itself. The whole website is just an index that links to manga on external websites. All you are doing is storing metadata and displaying it as a webpage. The easiest problem on the web. So, I really don't understand the premise behind the whole post. The problem statement is: > In practice, we currently see peaks of above 2000 requests every single second during prime time. This is great in terms of success as a website, but it's underwhelming in terms of describing a technical problem.

colesantiago 5y ago
I agree, just had to read the article again, and took it as a fancy way of wasting money really.
AviKav 5y ago
The only releases that link to external websites are the ones from sites such as MangaPlus and BiliBili (And delayed releases if you count those)
Tenoke 5y ago
>Looking at the website itself, mangadex.org, it doesn't even host the manga itself. They do seem to. Clicking on a random manga on there the images are hosted on their server[0]. Also I guess some of those are much bigger images which is less trivial to serve at that rate than a 10kb static page. 0. blob:https://mangadex.org/e78bd61a-e761-4a73-a27c-5f58394e7ea4 https://mangadex.org/e78bd61a-e761-4a73-a27c-5f58394e7ea4
- akx 5y ago
  Blob links are scoped to your browser tab, they're not real internet URLs.

germanjoey 5y ago

Many manga fans have a love/hate relationship with mangadex. On one hand, it's provided hosting for countless hours of entertainment over the years. Their "v3" version of the site was basically perfect from a usability point of view, to the point that the entire community chose to unite itself under its flag. On the other hand, directly because of the above, their hasty self-inflicted take down earlier this year nearly killed the entire hobby. Many series essentially stopped updating for the ~5 months the site was down, and many more are likely never coming back again. The decision to suddenly take the site down for a full site rewrite feels completely inexplicable from the outside. (A writeup the above or the previous one[1], both of which read like they were written by a Google Product Manager, especially don't help as they conspicuously avoid any comment to the one question on everyone's mind: "leaving aside the supposed security issues with the backend, why on earth also rewrite and redesign the entire front end from scratch at the same time?") [1] https://mangadex.dev/why-rebuild/ https://mangadex.dev/why-rebuild/

nvarsj 5y ago
I didn't get that take at all from the why-rebuild link. It seems reasonable to me - legacy code base, hard to maintain, with security problems that led to the massive data leak a while back. They also don't owe anything to anyone, and as a hobbyist project, they wanted to try something new. I'm impressed as they seem to have managed it - and the new site feels a lot more responsive than the old one.
creshal 5y ago
Yeah. This whole mess pushed me to moving everything I had (or could remember, anyway) to Tachiyomi¹, so I can hop between hosting websites freely without losing progress or access to old chapters (as long as I don't run out of local storage). And while it works fine for reading, it kills any interaction with the hosting sites. No chance for monetization, socialization or anything else that can help sites survive long-term. [1] https://tachiyomi.org/ https://tachiyomi.org/
- m45t3r 5y ago
  Before MangaDex we had Batoto (the old Batoto before some sketchy company bought the name), that was kinda of the same: serving high quality manga for most scanlators that wanted it (and also avoiding hosting pirated chapters from official sources, so kinda the same as MangaDex nowadays). As far I remember Batoto closed because of pressure from companies and also because of the high costs related to run the site. So yeah, considering how fragile maintaining a site like this is, it is always a good idea to sync your progress in a third party so it is easier to migrate if something goes wrong. > And while it works fine for reading, it kills any interaction with the hosting sites. No chance for monetization, socialization or anything else that can help sites survive long-term. BTW, MangaDex doesn't have monetization because it is strict a hobby and also because it is a gray area to monetize about this kinda of work [1]. Also, their Tachiyomi client is official (MangaDex v5 API was tested primarily via their Tachiyomi client before they finished the Web interface). [1]: both for companies (that has the copyright from the works hosted on those sites) and the scanlators (the fans that does actual work of translating those chapters). Sites that host those chapters and monetize are pretty much monetizing on work from other people.

maxk42 5y ago

I run an Alexa top-2000 website. (Mangadex is presently at about 6000.) I spend less than $250 a month. I have loads and loads of thoughts about what they could be doing differently to reduce their costs but I'll just say that the number one thing Mangadex could be doing right now from a cursory glance is to reduce the number of requests. A fresh load of the home page generates over 100 requests. (Mostly images, then javascript fragments.) Mangadex apparently gets over 100 million requests per day. My site - despite ostensibly having more traffic - gets fewer than half that many in a month. (And yes, it's image-heavy.) A couple easy wins would be to reduce the number of images loaded on the front page. (Does the "Seasonal" slider really need 30 images, or would 5 and a link to the "seasonal" page be enough? Same thing with "Recently Added" and the numbers of images on pages in general.) The biggest win would probably be reducing the number of javascript requests. Somehow people seem to think there's some merit to loading javascript which subsequently loads additional javascript. This adds a tremendous amount of latency to your page load and generates needless network traffic. Each request has a tremendous amount of overhead - particularly for dynamically-generated javascript. It's much better to load all of the javascript you need in a single request or a small handful of requests. Unfortunately, this is probably a huge lift for a site already designed in this way, but the improved loading time would be a big UX win. Anyway - best of luck to MangaDex! They've clearly put a lot of thought into this.

radicalbyte 5y ago
Do you manage to get as many buzz-words and OSS products into your system as they do? :) In general the less moving parts you have in a system the more reliable, secure, efficient and cheaper the system becomes. In their case they run a site that is probably under constant attack by the "hired goons", so they're going to need to have more moving parts than others. Plus they will want to optimise for minimal development time (it's a hobby) so just adding another tried and trusted system into the stack to do something you need makes sense.
- starfallg 5y ago
  >In their case they run a site that is probably under constant attack by the "hired goons", so they're going to need to have more moving parts than others. That's taken care of by the DDoS-Guard system they placed fronting their infrastructure. The design of their system has to take this into account, but that is mainly on a IP and DNS level. The design of their stack behind the loadbalancer is mainly driven by their functional and non-functional requirements, rather than by the need to prevent DDoS attacks.

deleted 5y ago

[deleted]

cyberpsybin 5y ago

At some point in new re-design, they started to load full size images for thumbnails. The whole site feels slower due to that. Need an automatic re-scaler service.

tristan9 5y ago
Not correct, we generate 2 thumbnails sizes for every cover -- if the site loads full-size anywhere by default (rather than when you expand it) it's definitely a bug!

probotect0r 5y ago

How is the Wireguard VPN set up? Has anyone used Wireguard to set up a VPN into AWS VPCs?

Shadonototro 5y ago

How can this be legal? It's basically pirating content

sxhunga 5y ago

Interesting to know more about news.ycombinator.com !