9 ms·
> The question: how does OpenTTD’s infrastructure look, or even: why is it so complex, is a rather complicated question to answer in a few words. [...] Was wai
by paxys 3y ago
> The question: how does OpenTTD’s infrastructure look, or even: why is it so complex, is a rather complicated question to answer in a few words. [...]
Was waiting for them to get to the "why is it so complex" part, but after all the details of Cloudflare Pages, Cloudflare R2, Cloudflare Workers, Cloudflare Access, EC2 instances, multiple CDNs, hosted Redis, Nomad, Pulumi, web of proxies and APIs and front doors, a dozen microservices and an IaaS repo to make sense of all of this, it came down to:
> In total, we store over 150GiB of data, transfer over 6TiB of data monthly, have more than 10M requests a month, and serve thousands of unique visitors every week.
Basically my MacBook Pro from 2019 could host all their infra and data and serve the entire load (~3 RPS) with room to spare for my day-to-day work.
For anyone else who is reading the post looking to get inspired – ignore everything they did and start small. A single web service to handle all business logic, hosted on two rented VPS instances which split traffic. Data stored in mysql or postgres with some regular backup. Start scaling only when the load from this setup overwhelms you (and I can guarantee it won't for 99.9% of cases, including the one in this post).
- mynameisvlad 3y agoAnd if your MBP were to die today? Or if a burst of traffic came through? Or if you closed the lid and forgot to turn Caffeinate on? Surely you can understand why people don't routinely host all their project's infrastructure on their laptops, even if the technical specs are enough.
- LordShredda 3y agoIt doesn't have to be _your_ macbook, it could be someone else's, like a vps somewhere else with a 99.9 uptime. That would be more than enough to serve 150GB of static data + a forum. You don't need to involve amazon and cloudflare for a 90's game.
- Macha 3y agoNote that the forum is a separate entity to the openttd project (though there is some overlap in people), and while openttd is the most commonly played variety these days, was started to discuss TTD and TTDPatch
- bshipp 3y agoI think the point is that most of those requirements could be met with a $60/month hetzner server containing a 512 GB raid array. I think that comes with 100GB free backup and if you pay another 10 bucks or so you can upgrade to a terabyte. But it's been a while since I checked out those costs.
- lelanthran 3y ago> And if your MBP were to die today? Or if a burst of traffic came through? Or if you closed the lid and forgot to turn Caffeinate on? So ... two macbooks then? > Surely you can understand why people don't routinely host all their project's infrastructure on their laptops, even if the technical specs are enough. I think the poster was simply highlighting that 150GiB of storage, 6TiB of transfer and around 4 requests per second to an average of maybe[1] 26 users per minute might not necessarily need a K8 cluster and 8 different cloud services. [1] I'm taking "thousands of users" to mean "up to 19000", otherwise they would have said tens of thousands, dozens of thousands, etc.
- mynameisvlad 3y agoThen they should have said that. But even then, you’re just pushing the cost onto the person maintaining the system. Cloudflare R2 => Now you have to bootstrap and maintain your own high availability Minio cluster. On multiple servers in multiple data centers for redundancy, of course. Cloudflare Workers/Pages => Now you have to maintain your own compute runner (granted, could be as simple as a docker container but that still both requires work to set up and transition over as well as maintain over time) and load balancer (once again, just Nginx or Apache but that requires setup and maintenance) to execute and serve this content. Cloudflare Access => Now you have to maintain your own access control system like ory. Cloudflare Tunnels => If you’re only running one node, this isn’t needed, so congrats I guess. You’d still need to provide internal access if you have multiple nodes in a HA environment, though. AWS EC2 => Now you need to maintain your own VMs. Etc. This is a volunteer project. Having someone maintain all these things may not be even remotely practical. It’s all the rage to hate on cloud services in 2023 but they abstract away a lot of operational work and that’s not something to be blindly discounted.
- TylerE 3y agoFor some values of "half to". It's an open source game... if the site goes down for 10 minutes every now and then, would anyone even notice?
- 3y ago
- metalforever 3y agoIt does not happen. Just round robin two $10 vps. Have backups of the database.
- klysm 3y agoSounds like a very reliable system. I’m sure the latencies would be excellent everywhere.
- Macha 3y agoI'm not sure even something ludicrous like a 10s latency would actually be a problem for OpenTTD. The uses of their hosting are: 1. The web page you're reading 2. Mod downloading and server listings. Sure, less latency would be nice, but is it vital? In particular, if you join a multiplayer game, that's the end of their server involvement, they're not hosting the multiplayer servers, so it won't have a gameplay effect. Add on to that that a lot of their players get the game from Steam, linux package repositories, or JGR's github releases page anyway.
- cdchn 3y ago10s of latency is a problem for any web page, OpenTTD or not.
- Macha 3y agoI just went to new.reddit.com and it's 9s for a full render. YouTube is 5s. Given openttd.org renders in a single request unlike those two sites, it actually wouldn't be that much of an outlier.
- cdchn 3y agoI wouldn't hold either of those up as good counter examples: 1) Single Page Apps vs. mostly static HTML 2) Not great exemplars of quickly loading pages 3) Your bandwidth might be bad- I was able to load youtube.com in <2s.
- Macha 3y ago> 1) Single Page Apps vs. mostly static HTML Users don't care about tech choice. > 2) Not great exemplars of quickly loading pages And yet they're some of the most popular web sites on the internet, so does that not indicate that they do so despite the performance and therefore users find their performance at least acceptable? > 3) Your bandwidth might be bad- I was able to load youtube.com in Good for you? I'm on a 2GBit/s fiber connection with 3ms ping to youtube.com . I am however, likely located further away from their actual servers doing the processing (while ping measures the response from their edge servers).
- cdchn 3y agoNow you just need to globally distribute your MacBook Pro.
- rhn_mk1 3y agoDoes latency matter that much for OpenTTD visitors? If not, what's the reason to distribute anything?
- cdchn 3y agoBetter performance, reliability, protection from DDOS, lots of reasons.
- rhn_mk1 3y agoWhile I'm not so confident that it's a given that performance or reliability increase after a system is distributed, let's assume that it's true. This turns my question into: does OpenTTD need more performance or reliability for its website? I can kind of see how DDOS protection might be useful, but... I don't protect my stuff against DDOS: the loss of service is nullified by the effort and risk required to set it up and maintain. What would that calculation look like for a random forum?
- cdchn 3y ago>While I'm not so confident that it's a given that performance or reliability increase after a system is distributed, let's assume that it's true. I think you can unequivocally agree that a distributed service thats designed to be fault tolerant is going to more more reliable than your MacBook Pro sitting in you closet on your home Internet connection. >I can kind of see how DDOS protection might be useful, but... I don't protect my stuff against DDOS If you don't care about your stuff going down, then it doesn't matter. If you don't care about it, then comparing it to a setup where that is a feature, isn't an even comparison. Even if you hosted this in your own closet on your MacBook Pro, OpenTTD's setup is still somewhat competitive. You might say "Oh I get that all for free" well a MacBook Pro costs money, your home internet costs money (although most consumer Internet is going to push back if you do more than 1Tb up a month- this is the cue for everybody who wants to rave about how great their Internet is to be contrarian below), you're paying electricity, and rent- even hosting your MacBook Pro is a marginal benefit from other expensive you already have, it's still not _free_.
- mcpackieh 3y agoEverybody wants to larp like they're operating at Google scale. I've even heard people refer to 1GB csv files as "big data"; that file fits in the RAM of my laptop from 20 years ago! But I've stopped fighting it when I have no stake in it. If it's good for their CV and they can afford the expense, then good for them I guess. It's not my problem. Maybe it's problematic from an environmental standard I suppose, but that could impeach many hobbies.
- thesnide 3y agoI name it HDA: Hype Driven Architecture. It is much more driven by personal career goals than real business or technical incentives. I also don't usually care as everyone is free, unless they are on my payroll ;-)
- ajsnigrutin 3y ago> I've even heard people refer to 1GB csv files as "big data"; Same here... once it wouldn't fit in excel (65535 lines + ), it became "big data" :)
- _boffin_ 3y agoYou… can easily fit over a billion rows of data into excel. That 65k is so last decade.
- ASalazarMX 3y agoI'm glad to know you can do BIG DATA in Excel these days.
- Noughmad 3y agoIt kinda makes more sense for OpenTTD. The game is famous for creating huge complicated networks for moving cargo and passengers, it is natural that the devs gravitate towards huge complicated computer networks too.
- pessimizer 3y ago>> In total, we store over 150GiB of data, transfer over 6TiB of data monthly, have more than 10M requests a month, and serve thousands of unique visitors every week. > Basically my MacBook Pro from 2019 could host all their infra and data and serve the entire load (~3 RPS) with room to spare for my day-to-day work. Yeah, their requirements remind me of somebody on a bittorrent forum describing their minimum acceptable seedbox.
- doctorpangloss 3y agoHmm, the web services interact with OpenTTD the game/application, across a few versions, so it's understandable to me that their infrastructure is complex. > To keep the AWS infrastructure as cheap as possible, we wanted to avoid needing a NAT gateway: if you use IPv4, you need something that allows you to talk with the outside world. On AWS you do this by installing NAT gateways. Sadly, those are (relatively speaking) rather expensive. So instead, we run as much as we can IPv6-only. If the OpenTTD folks are reading this: this is not true. You can assign public IP addresses to your t4g instances by using a public subnet in the VPC they are deployed to. They will correctly reach WAN addresses via their public interface and you will not need a NAT gateway. This incurs no additional cost. It sounds like your current instances were deployed to a private subnet. You will not need to do any additional configuration on Amazon Linux 2 or the distribution of Ubuntu those instances deploy with using the Launch Wizard - both from the point of view of the Linux firewall and from the point of view of security group configuration. This will greatly simplify your configuration. > Nomad is similar to Kubernetes, AWS ECS, AWS EKS, Azure AKS, etc, but a bit simpler to work with in a day-to-day... For this we run nginx on all clients in the cluster. Nginx uses Nomad’s service discovery information to forward traffic to the right instance. You are reinventing Kubernetes. This is okay. At the time that these decisions were made, ChatGPT 4 didn't exist - nowadays, if you want, you can "just" ask for Kubernetes manifests, and you will get correct ones, and you will see the light. The keywords for what you should ask for are `flux` and `eksctl`. You can create your Kubernetes cluster with one file & one command with `eksctl`, then `flux bootstrap` a Git repo that will contain all your YAML files describing your application. Another perspective is, to ameliorate your NAT gateway mistake, you have to be pretty familiar with the AWS and Cloudflare networking details. If you used eksctl, you wouldn't.
- paxys 3y agoPeople have been running reliable and performant multiplayer game servers for decades on potato-quality hardware. It isn't some insurmountable problem that only became possible in the cloud computing age. Heck popular minecraft servers get orders of magnitude more traffic than OpenTTD, and they all run on things like DigitalOcean droplets and $5 VPS's. You don't need relationships with account managers at AWS and Cloudflare and an alphabet soup of proprietary products to serve a few thousand gamers playing a 30 year old game.
- throw2413123123 3y ago> Cloudflare Pages, Cloudflare R2, Cloudflare Workers, Cloudflare Access These are actually fully managed services that require very little maintenance and tend to reduce complexity versus non managed services. > A single web service to handle all business logic This probably isn't an ideal architecture for someone building a backend for a multiplayer video game. > ignore everything they did you seem to be ignoring the fact that their architecture evolved into this state over the course of a few years.
- bob1029 3y ago> hosted on two rented VPS instances which split traffic I would push back here. I think even this is ridiculous complexity in 2023 if you just need to serve a webapp or API. Managing VMs is a mistake now. Serverless is an extreme competitive advantage when you are small and trying to stay focused on the customers. I don't have time to babysit self-serving technological curiosities as we try to ramp. There definitely isn't any money in pet problems. There is also a benefit to noob developers - if you constrain yourself to shipping serverless function code, you can't possibly be tempted and fall into weird infra rabbit holes that rob you of your ability to deliver near-term value and learn about practical software work. Our next-gen architecture consists of exactly 2 things. Azure SQL Database Hyperscale and Azure Functions. We deliver server-rendered HTML directly to the client from the HTTP trigger functions and they in turn connect directly to the DB. I almost suspect Microsoft doesn't like us doing it this way (e.g. mandatory URL route prefixes). But, too bad for them - we worked around it & our hosting model is effectively a prerequisite for the complexity circus everyone else employs. That's it. We are about to have a zero VM cloud infrastructure and someone with 2 days worth of YouTube training could become semi-effective at monitoring it all. Become the cockroach of technology users. Use the barest subset of what is needed to get the job done, but do it in a clever way. Infest someone's cloud so you don't have to screw with boring things like compliance, audits, power supply replacements, deployments, etc. Use their products in ways that feel manipulative, but are still strongly within the lines of ToS and the barycenter of the overall crowd. Stay away from bleeding edge technology. Imagine if you literally only had to push code (aka not declarative infra) to GitHub and pay one cloud bill. Certs, networking, patching, backups, monitoring, recovery, scaling up/out, etc all completely handled for you. Why wouldn't you want your life to be this easy? Is it because "fuck Microsoft[0]" that we continue to dig the technological equivalents of ditches by hand all day? Are we just poorly incentivized? [0] To be clear, you could replace "Microsoft" with "Cloudflare" or a number of other hyperscalars. I am tempted to play around with the CF offerings again. I like the idea of an "emergency backup vendor" if things ever get spicy with Microsoft.
- scapegoat2 3y agoI would definitely not call serving HTML from Azure Functions next-gen. You are increasing the response time by 20-30ms just by using HTTP-Functions instead of plain ASP.NET (lots of abstractions and grpc channel to worker processes). Beyond that you will get long cold starts, no response-streaming and slow scaling. The cost of the consumption-based plan will skyrocket after small number of requests compared to serving from dedicated App Service.
- terom 3y agoI think it would be a very bad idea to host the entire OpenTTD infrastructure on your MacBook Pro.