22 ms·

Like many others have pointed out: Cost. I'm the CTO of a moderately sized gaming community, Hypixel Minecraft, who operates about 700 rented dedicated machine

Like many others have pointed out: Cost. I'm the CTO of a moderately sized gaming community, Hypixel Minecraft, who operates about 700 rented dedicated machines to service 70k-100k concurrent players. We push about 4PB/mo in egress bandwidth, something along the lines of 32gbps 95th-percentile. The big cloud providers have repeatedly quoted us an order of magnitude more than our entire fleet's cost....JUST in bandwidth costs. Even if we bring our own ISPs and cross-connect to just use cloud's compute capacity, they still charge stupid high costs to egress to our carriers. Even if bandwidth were completely free, at any timescale above 1-2 years purchasing your own hardware, LTO-ing, or even just renting will be cheaper. Cloud is great if your workload is variable and erratic and you're unable to reasonably commit to year+ terms, or if your team is so small that you don't have the resources to manage infrastructure yourself, but at a team size of >10 your sysadmins running on bare metal will pay their own salaries in cloud savings.

MCthrowaway12 6y ago
[offtopic] Awesome seeing you here! I used to be super active on Hypixel when I was in high school (I was actually #1 on the leaderboards for one of the games for over a year). One of my first large scale programming projects ever was creating a Hypixel mod for my guild. Years later I am now a software engineer working at Google on the YouTube algorithm!
- AgentK20 6y ago
  Great to hear! Lots of the people who learned how to code because of Minecraft have ended up all over the industry. It's where I got my start, as well as many of my coworkers. That's part of our passion for making Hytale, to empower the next generation of youth who want a game they can tinker with and use to learn. If you haven't seen it yet definitely go take a look, got lots of cool plans for moddability and customization :)
  - kldavis4 6y ago
    Also offtopic, but my 12 year old is obsessed with Hytale.. Loves to read the blog updates and watch the videos and can't wait to eventually play it :)
    - djrogers 6y ago
      As is my 11yr old daughter. Cannot wait to play it. Actually, she’s a budding designer, and all she wants to do is mod it!
      - Jowsey 6y ago
        Haha same, admittedly I don't have a child, I am the child, I can't wait to get into Hytale and work on mods, servers, whatever. Hypixel is singlehandedly the reason I got into Minecraft and Minecraft is the reason I ended up learning Java. That Java knowledge has earned me a respectable income as I go through college. I owe a lot to Hypixel/Studios man :D
  - BrandoElFollito 6y ago
    This is the reason I like your site. It not only drags my son into coding (which neither the school nor me to a certain extend could not) but also since he was admitted as a helper (or something like that) he is also learning other skills. The downside is that this competes with school (thanks god he is very good so that's fine-ish) The upside is that it is quarantine in France so I can manage this. Thanks for all of that!
  - Yukonv 6y ago
    Little late but though I would say hi. I too got started programming thanks to Minecraft. My first real job was working at Overcast Network (oc.tc). I remember having to scale out our infrastructure to seven dedicated servers after a popular YouTuber featured us. At the time that felt crazy for a Minecraft server and here you are now with hundreds of servers. Huge congrats on scaling to where you are today. Have lots of fond memories of those early years, especially Minecon 2013.
- localhost 6y ago
  Since we're OT anyway, thank you for all the work you do on the YouTube algorithm. It has brought a tremendous amount of value into my life!
  - mnky9800n 6y ago
    people like the youtube algorithm?
    - hn_check 6y ago
      This is remarkable. The "YouTube algorithm" is responsible for Donald Trump's ascension, Pizzagate, the belief that 5G towers cause Coronavirus, anti-vaccers, and on and on. It has been the most destructive force in the modern era. Personally I find it unbelievably stupid -- the videos it presents to me are just a terrible collection of trash, and if I happen to watch one video about someone shooting arrows, suddenly YouTube thinks I am an extreme bow and arrow aficionado. This post will be auto-flagged given that dang prefers to run an AGW-denial website that spends far too much time apologizing and appeasing the US president.
- notwedtm 6y ago
  I'll chime in and say hello as well! Great seeing Hypixel staying on top after all these years!
RhodesianHunter 6y ago
> Cloud is great if your workload is variable and erratic Or if you're just constantly iterating on a large product with many engineers. Those engineers' salaries almost always outweigh all of your cloud costs and so making them productive is cost effective. Things like SNS/SQS/S3/VPC/ELB/etc. save you countless hours and often make up for the increased cloud costs with increased developer productivity.
- YarickR2 6y ago
  No they don't . Things you mentioned are very easy to deploy on-premises, and they won't take more than one man month to deploy and maintain.
  - kelnos 6y ago
    That's an incredibly naive assessment and is absolutely not true in practice, except for perhaps for some orgs where the stars just happen to align to make that a reality. Most places heavily build "cloud" into their frameworks, tooling, hiring, and, well, everything. Even just swapping out a single hosted/managed component (as my company did a few years ago, replacing Kinesis with Kafka) can require a lot of up-front work[0] and on-going maintenance. [0] Especially if you're already in production with the thing you want to replace, and want to transition without downtime for your customers.
  - RhodesianHunter 6y ago
    I love that you think mature products that took teams of highly skilled people at Amazon years to refine can just be slapped together in a month in your colo. What a joke.
- dahfizz 6y ago
  > Those engineers' salaries almost always outweigh all of your cloud costs I think there is a divide here. Most people in this thread that mention cost as an issue are running some serious gear, and that does not come cheap. Your parent is running 70 servers plus some serious networking equipment, which is easily a couple million dollars. And he said that a cloud provider would 10x that cost. If all you need is a little VM to run a website, cloud hosting is cheap. If you are running real infrastructure, a couple hours of developer time will never out-weight the astronomical costs of cloud hosting.
bob33212 6y ago
That is where the margins come from on AWS. They are charging 10x the cost of the hardware. Even the AWS well architected framework says that there are always tradeoffs between disaster recovery and cost. For your business cost is more important than having your servers replicated in many time zones.
- jbverschoor 6y ago
  Servers aren’t replicated. As a matter of fact.. aws began out of unused machines for ephemeral servers
JoshTriplett 6y ago
Who are you getting your bandwidth from?
- kjsthree 6y ago
  This! I very much want to hear about how you get your bandwidth.
  - AgentK20 6y ago
    We actually just get a datacenter blend, but I _believe_ they've got us cordoned off on our own dedicated 40gbps Level3 (now Centurylink) line to prevent us getting attacked from harming other customers. It's not so much about which provider you go to, it's more just about the economies of scale. Our entire fleet exists in one physical DC location due to the requirements of Minecraft's netcode, so rather than having five 10gbps lines at five facilities, we've got one 40gbps line. Because of that the ISPs don't meter by the GB, but by the 95th percentile traffic. At those scales, we're paying pennies on the dollar compared to public "by-the-GB" pricing.
    - lewis1028282 6y ago
      What do you do about backups? I saw you only have one DC, are you worried about that DC being flooded etc and losing all your data?
      - AgentK20 6y ago
        Offsite AWS Glacier backups, as well as a warm redundancy in another DC.
koheripbal 6y ago
Same here - CTO of medium sized company. Our IT infra costs are 1/10th the cost of cloud, simply because I happen to be comfortable having on-premise machine and working on them (sometime myself). We have two dozen servers in two locations. It's more time to setup, but maintenance is actually quite low.
- dkdk8283 6y ago
  > maintenance is actually quite low. I do hope you regularly patch and reboot your systems.
  - biztos 6y ago
    I hope so too, but it's not like a fleet of fine-tuned EC2 instances running CentOS 6.5 is going to be patched and rebooted regularly without a similar amount of sysadmin attention. (Not picking on CentOS, just using it as a placeholder for $REQUIRED_OS_BECAUSE_OF_CUSTOM_SOFTWARE.)
    - mr_toad 6y ago
      Automatic unattended updates are totally possible (I haven’t tried it in CentOS but it works fine on Ubuntu). Even kernel updates can now be done without a reboot. And with the proper setup even reboots can be automated.
      - mrweasel 6y ago
        But that's not limited to cloud, you can do that for on-premise stuff.
        twic 6y ago
        There's a cognitive failure i see a lot where people incorrectly conflate two different new hotnesses like this. Cloud? Automatically updating operating systems. On-prem? Must be stuck on an old version. Microservices? You can release to production in minutes. Monolith? Must take you weeks to get a release out. Go? Simple self-contained deployables. Java? Must need a JDK and Tomcat installed on the server. It always seem to come up when the leading term is something dubious (cloud, microservices, Go), but the following term is something unequivocally good. Or maybe that's just when i notice it.
mmmBacon 6y ago
A few years ago I was trying to start a company and get it off the ground. We had to make decisions on our tech stack and whether we were going to use AWS and build around their infra. Our business was very data heavy and required transferring large datasets from outside to our databases. Even in our early prototypes, we realized that we couldn’t scale cost-effectively on AWS. I figured out that we could colocate and rent racks, install HW, hire people to maintain, etc... for way less than we could use the cloud for. I was shocked at the difference. I remember saying to my cofounder why does anyone use AWS, you can do this on your own way cheaper. Later I worked at a FAANG and remember when Snap filed their S1 when they were going public they disclosed that they were paying Google $5B and we were totally shocked at the cost compared to our own spend on significantly larger infra. I think people don’t realize this is doable and it’s great to hear stories like yours showing the possibilities.
- RachelF 6y ago
  Dropbox did the same thing a few years back - moved everything from Amazon S3 to their own storage. My guess is they did it for cost reasons.
  - llarsson 6y ago
    That S3 is eventually consistent with object updates (HTTP PUT) might also screw up things for a company whose core value is synchronized storage.
    - boredpudding 6y ago
      Since the move to S3 is only 'a few years ago', my guess is this was not the reason for them to move away from S3 since service at Dropbox is pretty much the same.
    - RachelF 6y ago
      I suspect that is behind many problems with Google Drive. Sometimes old versions seems to take precedence over newer ones, for some reason.
YarickR2 6y ago
Gaming company senior reliability engineer/officer here, with the same tune. We're operating thousands of servers on three continents (NA, Europe, Asia), running our own game distribution network and basically doing as much as possible on prem. Every time we're trying to play by the cloud provider rules, we're getting stung with outrageous storage and bandwidth bills. I'm pushing for in-house k8s deployment to increase hardware utilisation and drive our price/performance ratio even lower, but even now we're much better financially purchasing hardware, putting it in leased racks, doing networking ourselves and going full cycle. Probably, if we would be a smaller shop, we'd outsource networking . That's it.
- SteveNuts 6y ago
  Are you looking at a specific vendor for your on prem k8s?
  - SteveNuts 6y ago
    Not sure why I was downvoted here, I'm in the same boat looking for a solution, not trying to sell something.
jread 6y ago
I’d be interested in your take on AWS Outposts - do you see potential in this type of offering?
- AgentK20 6y ago
  I'll be totally honest, I hadn't heard of Outposts till you mentioned it, but from my quick glance I'd say there's definitely potential in it, but I don't think it fits our use case particularly well. One of the challenges for us is, to be honest, we're rather spoiled having run exclusively off-the-shelf open-source tech on our own hardware. It's difficult to start paying per million DB queries, for example, when we've been paying a flat rate of $X/mo for the past 7 years. On top of that, our team is very comfortable operating and managing these services in-house, and while it would free them up to focus more on dev tasks, it's a tiny gain compared to the cost increase of moving to SaaS model. Having said that, though, I'm definitely going to look into Outposts more since it seems much more usable than full-cloud, so thanks for bringing it to my attention!
legionof7 6y ago
Ooh completely unrelated to the thread but thanks for offering my friends and I countless hours of fun during our middle school years!
andrenotgiant 6y ago
It sounds like you've got a good setup going with colo, but just as a way of illustrating some of the small providers lower bandwidth costs: DigitalOcean gives 3TB bandwidth on a $15 2CPU/2GBMEM/60GBSSD instance. If you ran 1350 of them it'd cost you ~$20k/month and get you your 4PB egress within bandwidth allowance.
- dragonwriter 6y ago
  > DigitalOcean gives 3TB bandwidth on a $15 2CPU/2GBMEM/60GBSSD instance. DigitalOcean is a VPS provider, not a cloud provider, and that's pretty similar to pricing on the nearest comparable AWS service (Amazon LightSail), not something showing DO as notably better (1Core/2GBRAM/60GBSSD/3TB transfer @ $10/mo or 2Core/4GBRAM/80GBSSD/4TB transfer @ $20/mo.) Though if you are optimizing price per TB of transfer quota, LightSail does best with the 1Core/1GB/40GBSSD/2TB transfer @ $5 instance size.
  - kortilla 6y ago
    > DigitalOcean is a VPS provider, not a cloud provider From the first paragraph in https://en.wikipedia.org/wiki/DigitalOcean https://en.wikipedia.org/wiki/DigitalOcean > DigitalOcean, Inc. is an American cloud infrastructure provider[2] headquartered in New York City with data centers worldwide.[3] DigitalOcean provides developers cloud services that help to deploy and scale applications that run simultaneously on multiple computers. You might have your own cute definition of “cloud”, but it doesn’t match the industry so get over it and stop correcting people.
- tpetry 6y ago
  Bandwidth Alliance is only for routing http traffic, but a game server will have udp traffic.
  - dylz 6y ago
    Bandwidth Alliance has nothing to do with this, afaik. Minecraft is a TCP game.
- 6y ago
Animats 6y ago
Yes. I wonder about Second Life, which is doing a "cloud uplift" to AWS. The parts of the system that are variable-load and web-like are already on AWS. But the region servers (one CPU for each 256m^2) are owned outright, and in a colo in Phoenix. They're obsolete machines. But they are compute bound, with a constant load 24/7. Even when no user is in a region, the simulated world continues to run. Uses a bit less CPU, but the physics engine is still doing 45 cycles every second, and all scripted objects are still running. Leaves fall from the trees, plants grow, animals graze, trains run, whether or not any human is watching. They think AWS will be cheaper. I hope they are right, but have doubts. Fortunately, they're doing this slowly and carefully, and if it turns out that AWS is too expensive, they should be able to move back or to elsewhere. Since what they're doing isn't even close to what AWS normally does, they're not that tied to AWS features.
- lifeisstillgood 6y ago
  >> Leaves fall from the trees, plants grow, animals graze, trains run, whether or not any human is watching. Well that solves the age old question about trees falling in forests :-)
  - rbanffy 6y ago
    That's a poorly implemented universe. Unless information propagates, it didn't happen ;-) You could always fast forward the current state when the first external observer arrives. But I get there are time constraints - you can't just stop the real universe until it converges to a consensual state.
    - lifeisstillgood 6y ago
      This does bother me about the whole "likely we are living in the matrix" thing. If we want to design a universe that does not waste a lot of processing power we need to do things like have a speed of light limitation (keep it fairly slow too). But the sensible idea is to only process lazily - don't calculate till the observer observes - but it is very much like a dependency hell.
  - 6y ago
cwojno 6y ago
I worked for a boss that pretty much demanded that we move to the cloud. I showed them the costs for the then 2 providers (GCP/AWS) and arrived at the exact same conclusion on server hosting alone, as bandwidth wasn't the main driver of our application. The rationale was that we'd save so much money by not having to manage the servers ourselves, but we honestly spent much, much more time in software deployments than managing hardware migrations and failures.
- abathur 6y ago
  To be clear, it took much more time to deploy your own software to the cloud relative to on-prem?
  - smolder 6y ago
    I interpreted it as: prior to moving to the cloud, server maintenance was not a very significant cost, so the justification for moving to the cloud was weak.
brownbat 6y ago
I remember a post here a while back from a guy running Bitcoin mining in his dorm room. One day he realized he could offer his spare cycles to grad students with high computing workloads and undercut cloud computing prices while increasing what he made far over just mining. It's very weird that we haven't fully arbitraged $/instruction to a single (low) price yet (or storage/hosting, whichever). If only there were an Uber for unused cycles or storage... let everyone turn their unused capacity into mini AWSs with a common interface and safety and reliability guarantees. Maybe there are too many barriers, like security.
- akg_67 6y ago
  > If only there were an Uber for unused cycles or storage... let everyone turn their unused capacity into mini AWSs with a common interface and safety and reliability guarantees. Funny, this was our idea with first startup 18 years ago, federation of unused storage and redistribution. There were no takers as no one wanted to contribute their unused storage but wanted others. It was much easier to centrally locate our own storage and allocate to users who required it.
- Fornax96 6y ago
  > If only there were an Uber for unused cycles or storage... let everyone turn their unused capacity into mini AWSs with a common interface and safety and reliability guarantees This is exactly what Sia.tech is building for storage. A competitive marketplace for storage providers where anyone can set up a host and start selling their spare storage space. It's not completely finished yet, but it's pretty close. A bunch of companies are already building products on top of it (including me)
  - swyx 6y ago
    how much replication do you need to make this kind of thing reliable? since anyone can pull storage at any time. does that then make transactions super slow? any numbers you can put on this would be helpful. also, whats your product? (just curious)
    - 6y ago
nikanj 6y ago
Are you actually on-premises at your HQ, or are you colocated to a datacenter?
- AgentK20 6y ago
  Neither. We've got a long-standing positive relationship with our data center and are operating on the same hardware that we started renting 4 years ago. Because of our initial investment to build up the fleet, we're now basically paying just the power and networking, plus a small markup, for our core fleet and fairly standard pricing for additional month-to-month machines that we spin up and down based on seasonal demand.
eb0la 6y ago
> Cloud is great if your workload is variable and erratic I would say also, the cloud is cheap if you can shut down or scale-down services when you need it. IMHO if you ow your software stack, you can use the cloud to rationalize your spending (for instance, Netflix changed their default codec some weeks ago which made them save a lot of monet in egress bandwith)... but nobody can do this. If your company runs prepackaged software, like an SAP, Manhattan, you don't have margin to shutdown services when they're not needed (at least by now). Also, after a long time working in a datacenter, I am still ashamed how powerful bare-metal has become: For less than 30K you can get a 1TB server (without storage) with at least 40 cores. Some vendors are even offering hardware as Opex in a pay-as-you-go seting to compete against the cloud. (disclaimer I work for a Google Cloud partner)
- gwd 6y ago
  > Some vendors are even offering hardware as Opex in a pay-as-you-go seting to compete against the cloud. Isn't that how IBM ran their mainframe business 50 years ago? The more things change...
- twic 6y ago
  > IMHO if you ow your software stack, you can use the cloud to rationalize your spending (for instance, Netflix changed their default codec some weeks ago which made them save a lot of monet in egress bandwith)... but nobody can do this. Netflix's content serving is done from physical hardware they own and place in datacentres. I don't understand your point.
- sokoloff 6y ago
  Netflix does not push their videos streams out of AWS (because that would be stratospherically unaffordable for them). They run their apps and systems on the cloud (search and preferences), but the video bits come from Netflix racks distributed to a variety of DCs and ISPs.
fxtentacle 6y ago
Cost. We currently pay less than €5000 monthly for 500TB/month in traffic and 50 Ryzen CPUs. Amazon would be $30,000 traffic + $100,000 compute.
- machiaweliczny 6y ago
  I think you forgot to include cost of engineers in this calculation.
  - sgtfrankieboy 6y ago
    Engineers cost $130,000/month? Thats a fulltime salary for 14 network engineers (salary from Indeed). There is no way you need that many for maintaining a fleet of 50 CPUs (chance is that in OPs case they got duel socket servers as well).
    - HelloNurse 6y ago
      On the other hand, you also need to pay engineers, and rather senior ones, to monitor and upgrade cloud solutions. They are just less visible: typically an extra duty for "devops" programmers and project managers rather than dedicated server farm staff.
    - kaydub 6y ago
      $130,000 a month is not 14 network engineers. Salary is not the full cost of an employee to the business.
      - kitotik 6y ago
        Even fully loaded it should cover at least 8-9 engineers.
  - twic 6y ago
    An iron rule of these discussions is that when someone objects to non-cloud solutions on the grounds of staff costs, they have not actually calculated the staff costs.
sandGorgon 6y ago
what i have generally seen is that storage - blob (s3) or database - is the blocker. Even if you go to k8s, the hassle to manage storage is non-trivial. And because of this singular fact, startups cannot move to on-premise. You really dont want to manage snapshots, restores, backups. Anyone can manage application servers.
- davedx 6y ago
  I don't understand. You're talking about a filesystem, with an option for backing that filesystem up?
- rhlsthrm 6y ago
  s3 (or similar) and managed database seem to be the most "commoditized" (and cheapest) pieces of the cloud services stack. So it seems reasonable to move everything else to cloud-agnostic/on prem while still leaving those pieces in the cloud.
  - sandGorgon 6y ago
    how do you solve for egress traffic ? the traffic between the server and the database starts being the most expensive part of the stack
winter_blue 6y ago
> about 700 rented dedicated machines to service 70k-100k concurrent players Slightly off-topic, but it sounds like a single machine can't handle more than 150 concurrent players. How is this so? Is Hypixel Minecraft that resource intensive? What's the bottleneck — it is the CPU, or RAM, or the network latency(?) per machine, or is it something else? > We push about 4PB/mo in egress bandwidth With 70k players, 4PB / 70k is approximately ~57 GB per player per month. That divided by the number of seconds in a month (2.628e+6) is: 57 x 10^9 / 2.628e+6 = ~21.7 kB. Over a month, on average, a single player consumes a bandwidth of ~21.7 kB/sec, or about 174 kbps. This is an extraordinarily low per-player bandwidth consumption. At 150 players, each machine would average 26 Mbps of network traffic. This is fairly low as well. Of course, the machines have to be capable of handling of possibly much higher peak usage, but even an order of magnitude more — 260 Mbps, is something a Raspberry Pi 4 (which support full-throughput Gigabit Ethernet) can do.
- AgentK20 6y ago
  About 60 of those machines are custom L7 frontend load balancers, and an additional 40-50 are database and dev machines. As for the actual gameservers, that's about the numbers we expect on average (150/box), although it varies wildly by gametype. Simple games of Skywars or Murder Mystery with no AI and an ultra-slimmed-down world we can fit 300+ players per box and are CPU-bound. Housing or Skyblock servers with full AI, dynamic worlds, and player-built structures we might only be able to fit 100 players per box and are RAM-bound. You're pretty accurate with the per player bandwidth. We measure it at an average of 200kbps/online player. We are incredibly compute-intensive, though, and these machines are all operating with E3-1271v3 CPUs, 32GB of RAM, and ~100GB SSD.
  - winter_blue 6y ago
    Thanks for the insightful comment. I saw you mentioned in another thread Minecraft performs better when the CPU has better single-threaded performance. I'm guessing, going forward into the future, you'd probably want to build machines with Zen 2 AMD Ryzen CPUs (or future Zen 3 Ryzens), like the Threadripper 3960X (which has a base clock of 3.8 GHz, 24 physical CPU cores, and costs ~$1200 — so $50 for each of those cores). Or, the AMD Ryzen 9 3950X, which is about $40 per core (when you get it on sale), and despite a lower nominal base clock actually performs better on single-threaded benchmarks (e.g. https://www.cpubenchmark.net/singleThread.html https://www.cpubenchmark.net/singleThread.html).
weinzierl 6y ago
Sorry, but what is LTO in this context? Google turns up Link Time Optimization and Legal Tribune Online and both seem not quite right....
- michaelt 6y ago
  Probably "Lease To Own" a.k.a. "hire purchase" which is kinda similar to buying something on long-term credit.
- plq 6y ago
  Not OP but he most probably meant lease-to-own. In my experience it's a financing model that spans the equipment's economical lifetime. Let's say 3 years for a server box. In the end of the contract you either pay the last instalment, which is bigger (5x-10x) than usual and keep it, or you return the equipment and get a new one. There are many variations of this scheme but that's more or less the idea.
swyx 6y ago
thanks for all the detailed info! i find this hard to square with the evidence that the Netflixes of the world still find it worthwhile to pay AWS then. theres no way that they dont have the same problems you do. care to speculate on why they prefer to pay a vendor? (it seems the Dropbox story is the exception that proves the rule)
- Strom 6y ago
  Netflix doesn't host its actual content on AWS. They built their own CDN and have their own dedicated servers all over the world. https://openconnect.netflix.com/en/ https://openconnect.netflix.com/en/
- deleted 6y ago
  [deleted]
anoraca 6y ago
Thank you for all of the great times playing Sky Wars!
neeraga 6y ago
Which dedicated infra host did you use?
- AgentK20 6y ago
  We bounced around through a few different hosts in the first two or three years of our operation, but for the past four years we've been consistently with SingleHop, now an INAP company. They've been great through the entire time that we've worked with them and have saved our asses on more than one occasion, especially back before mitigation tech like CloudFlare Spectrum was available.
  - ksec 6y ago
    So what is going to happen now that iNAP has filed for Chapter 11? ( Interesting there is no official words on the website. But that is what Wiki shows. )
StreamBright 6y ago
This is one of the valid use cases of on-prem, predictable high network bandwidth workload. This is the exception though. An average company has the exact opposite. Non-predictable, low bandwidth workloads that are different (like data warehouse vs. website).
tomohawk 6y ago
Agree on the cost for a sane organization. One draw with the cloud is that not all organizations are sane, so doing the work in house / on prem can be more expensive in terms of money, time, and politics than outsourcing to the cloud.
dbancajas 6y ago
off topic: could you share how you started in the business? I thought running gaming servers was a low margin business.. Is there a particular size you have to be for it to be worth the pay off?
dsieczko2 6y ago
Let's not forget about Packet! ;-)
dsieczko2 6y ago
To elaborate, we've worked with many customers looking to get out of "slinging metal" and focus on what matters to their business most. Some have said that the revenue generated justifies the cost of <enter one of big 3 here> BUT others (and I think you'll see this more over the next decade) are looking at automated bare metal. You get the primitives of the cloud but on physical servers, and more importantly with economics closer to running your own gear. Especially if you consider TCO - power, network, ops people, etc
- dsieczko2 6y ago
  Packet ^^
bytematic 6y ago
I'm super curious how you scale these Minecraft servers. I have a 60$ google cloud instance and it can barely handle 3 players with a few mods without lagging a few times per hour.
- AgentK20 6y ago
  It's a bit off topic, but the challenge there is similar to a reply I wrote elsewhere in this thread about clock speeds. Minecraft is almost entirely single threaded, which means that you could get a $400 Google cloud instance and still only be able to hold those three players. To scale past that you're going to need higher clock speed processors. Take a look at OVH's "GAME" series offering or any dedicated host with E3's or i7s.
pixtemp7384 6y ago
The Hypixel servers are at maximum capacity during the peak 4+ hours every day, meaning additional players cannot join. This is an obvious issue for other applications.
- AgentK20 6y ago
  This isn't a limit of our ability to procure hardware, but instead a limit of one of our 7-year-old monolithic applications. We're working on increasing its stability at high player counts, but it's a totally separate issue from server provisioning and cloud vs bare metal.
  - pixtemp7384 6y ago
    I see. Out of curiosity, what is the limiting application?
    - AgentK20 6y ago
      An in-house application that we (I) developed in order to handle game balancing, queuing, and overall network state for a multi-LB Minecraft network. The application's name is MasterControl, and it's architecture and design (or lack thereof, it was literally the first program I ever wrote) has been the source of many headaches over the last few years. Had I known years back that we would see the kind of player numbers that we have this year, we 100% would have completely rewritten it into our modern microservice architecture.
mister_hn 6y ago
What about cost of personal (operations/administration) and the cost of maintaining+electricity bill?
tekknik 6y ago
What about the other services? You’re running a Minecraft cluster so you’re fairly detached from what running a traditional service is like these days. Will those same sysadmins make API gateway for me? Or SNS? If you look at AWS and see only it’s EC2 offering then that’s part of the problem. I just moved a customer from onprem to AWS and they were flabbergasted at the performance difference (their processors being 3-4 yrs old at this point) and changed some of their processes from running for hours to a few minutes for zero capex. Then theres adding capacity. Need a new server to spin up your app? Brb while I go call dell, ETA 2 months....
ksec 6y ago
Very late into the discussion. One thing that bothers me a lot is Server Performance / Price hasn't dropped much over the last 5+ years. Memory prices over a long period has essentially been flat. The only thing that has gotten cheaper is NAND. I kind of expect of hoping to have double the core, and memory of a Server for the same price in 5 - 6 years time. And this hasn't happened.
hey_ross 6y ago
Disclosure: I work at Oracle Cloud on the product management side (not sales ;-) ) I'd like to chat with you about this; we charge a flat rate for dedicated connections and about 95-97% less than AWS on egress and are just starting to talk to people about it - it's how we picked up Zoom and 8x8 (bi-directional video has huge egress charges). Let me know if you are open to chatting, there wasn't a "exec team" link on your webpage to reach you directly.

MCthrowaway12 6y ago

[offtopic] Awesome seeing you here! I used to be super active on Hypixel when I was in high school (I was actually #1 on the leaderboards for one of the games for over a year). One of my first large scale programming projects ever was creating a Hypixel mod for my guild. Years later I am now a software engineer working at Google on the YouTube algorithm!

AgentK20 6y ago
Great to hear! Lots of the people who learned how to code because of Minecraft have ended up all over the industry. It's where I got my start, as well as many of my coworkers. That's part of our passion for making Hytale, to empower the next generation of youth who want a game they can tinker with and use to learn. If you haven't seen it yet definitely go take a look, got lots of cool plans for moddability and customization :)
- kldavis4 6y ago
  Also offtopic, but my 12 year old is obsessed with Hytale.. Loves to read the blog updates and watch the videos and can't wait to eventually play it :)
  - djrogers 6y ago
    As is my 11yr old daughter. Cannot wait to play it. Actually, she’s a budding designer, and all she wants to do is mod it!
    - Jowsey 6y ago
      Haha same, admittedly I don't have a child, I am the child, I can't wait to get into Hytale and work on mods, servers, whatever. Hypixel is singlehandedly the reason I got into Minecraft and Minecraft is the reason I ended up learning Java. That Java knowledge has earned me a respectable income as I go through college. I owe a lot to Hypixel/Studios man :D
- BrandoElFollito 6y ago
  This is the reason I like your site. It not only drags my son into coding (which neither the school nor me to a certain extend could not) but also since he was admitted as a helper (or something like that) he is also learning other skills. The downside is that this competes with school (thanks god he is very good so that's fine-ish) The upside is that it is quarantine in France so I can manage this. Thanks for all of that!
- Yukonv 6y ago
  Little late but though I would say hi. I too got started programming thanks to Minecraft. My first real job was working at Overcast Network (oc.tc). I remember having to scale out our infrastructure to seven dedicated servers after a popular YouTuber featured us. At the time that felt crazy for a Minecraft server and here you are now with hundreds of servers. Huge congrats on scaling to where you are today. Have lots of fond memories of those early years, especially Minecon 2013.
localhost 6y ago
Since we're OT anyway, thank you for all the work you do on the YouTube algorithm. It has brought a tremendous amount of value into my life!
- mnky9800n 6y ago
  people like the youtube algorithm?
  - hn_check 6y ago
    This is remarkable. The "YouTube algorithm" is responsible for Donald Trump's ascension, Pizzagate, the belief that 5G towers cause Coronavirus, anti-vaccers, and on and on. It has been the most destructive force in the modern era. Personally I find it unbelievably stupid -- the videos it presents to me are just a terrible collection of trash, and if I happen to watch one video about someone shooting arrows, suddenly YouTube thinks I am an extreme bow and arrow aficionado. This post will be auto-flagged given that dang prefers to run an AGW-denial website that spends far too much time apologizing and appeasing the US president.
notwedtm 6y ago
I'll chime in and say hello as well! Great seeing Hypixel staying on top after all these years!

RhodesianHunter 6y ago

> Cloud is great if your workload is variable and erratic Or if you're just constantly iterating on a large product with many engineers. Those engineers' salaries almost always outweigh all of your cloud costs and so making them productive is cost effective. Things like SNS/SQS/S3/VPC/ELB/etc. save you countless hours and often make up for the increased cloud costs with increased developer productivity.

YarickR2 6y ago
No they don't . Things you mentioned are very easy to deploy on-premises, and they won't take more than one man month to deploy and maintain.
- kelnos 6y ago
  That's an incredibly naive assessment and is absolutely not true in practice, except for perhaps for some orgs where the stars just happen to align to make that a reality. Most places heavily build "cloud" into their frameworks, tooling, hiring, and, well, everything. Even just swapping out a single hosted/managed component (as my company did a few years ago, replacing Kinesis with Kafka) can require a lot of up-front work[0] and on-going maintenance. [0] Especially if you're already in production with the thing you want to replace, and want to transition without downtime for your customers.
- RhodesianHunter 6y ago
  I love that you think mature products that took teams of highly skilled people at Amazon years to refine can just be slapped together in a month in your colo. What a joke.
dahfizz 6y ago
> Those engineers' salaries almost always outweigh all of your cloud costs I think there is a divide here. Most people in this thread that mention cost as an issue are running some serious gear, and that does not come cheap. Your parent is running 70 servers plus some serious networking equipment, which is easily a couple million dollars. And he said that a cloud provider would 10x that cost. If all you need is a little VM to run a website, cloud hosting is cheap. If you are running real infrastructure, a couple hours of developer time will never out-weight the astronomical costs of cloud hosting.

bob33212 6y ago

That is where the margins come from on AWS. They are charging 10x the cost of the hardware. Even the AWS well architected framework says that there are always tradeoffs between disaster recovery and cost. For your business cost is more important than having your servers replicated in many time zones.

jbverschoor 6y ago
Servers aren’t replicated. As a matter of fact.. aws began out of unused machines for ephemeral servers

JoshTriplett 6y ago

Who are you getting your bandwidth from?

kjsthree 6y ago
This! I very much want to hear about how you get your bandwidth.
- AgentK20 6y ago
  We actually just get a datacenter blend, but I _believe_ they've got us cordoned off on our own dedicated 40gbps Level3 (now Centurylink) line to prevent us getting attacked from harming other customers. It's not so much about which provider you go to, it's more just about the economies of scale. Our entire fleet exists in one physical DC location due to the requirements of Minecraft's netcode, so rather than having five 10gbps lines at five facilities, we've got one 40gbps line. Because of that the ISPs don't meter by the GB, but by the 95th percentile traffic. At those scales, we're paying pennies on the dollar compared to public "by-the-GB" pricing.
  - lewis1028282 6y ago
    What do you do about backups? I saw you only have one DC, are you worried about that DC being flooded etc and losing all your data?
    - AgentK20 6y ago
      Offsite AWS Glacier backups, as well as a warm redundancy in another DC.

koheripbal 6y ago

Same here - CTO of medium sized company. Our IT infra costs are 1/10th the cost of cloud, simply because I happen to be comfortable having on-premise machine and working on them (sometime myself). We have two dozen servers in two locations. It's more time to setup, but maintenance is actually quite low.

dkdk8283 6y ago
> maintenance is actually quite low. I do hope you regularly patch and reboot your systems.
- biztos 6y ago
  I hope so too, but it's not like a fleet of fine-tuned EC2 instances running CentOS 6.5 is going to be patched and rebooted regularly without a similar amount of sysadmin attention. (Not picking on CentOS, just using it as a placeholder for $REQUIRED_OS_BECAUSE_OF_CUSTOM_SOFTWARE.)
  - mr_toad 6y ago
    Automatic unattended updates are totally possible (I haven’t tried it in CentOS but it works fine on Ubuntu). Even kernel updates can now be done without a reboot. And with the proper setup even reboots can be automated.
    - mrweasel 6y ago
      But that's not limited to cloud, you can do that for on-premise stuff.
      - twic 6y ago
        There's a cognitive failure i see a lot where people incorrectly conflate two different new hotnesses like this. Cloud? Automatically updating operating systems. On-prem? Must be stuck on an old version. Microservices? You can release to production in minutes. Monolith? Must take you weeks to get a release out. Go? Simple self-contained deployables. Java? Must need a JDK and Tomcat installed on the server. It always seem to come up when the leading term is something dubious (cloud, microservices, Go), but the following term is something unequivocally good. Or maybe that's just when i notice it.

mmmBacon 6y ago

A few years ago I was trying to start a company and get it off the ground. We had to make decisions on our tech stack and whether we were going to use AWS and build around their infra. Our business was very data heavy and required transferring large datasets from outside to our databases. Even in our early prototypes, we realized that we couldn’t scale cost-effectively on AWS. I figured out that we could colocate and rent racks, install HW, hire people to maintain, etc... for way less than we could use the cloud for. I was shocked at the difference. I remember saying to my cofounder why does anyone use AWS, you can do this on your own way cheaper. Later I worked at a FAANG and remember when Snap filed their S1 when they were going public they disclosed that they were paying Google $5B and we were totally shocked at the cost compared to our own spend on significantly larger infra. I think people don’t realize this is doable and it’s great to hear stories like yours showing the possibilities.

RachelF 6y ago
Dropbox did the same thing a few years back - moved everything from Amazon S3 to their own storage. My guess is they did it for cost reasons.
- llarsson 6y ago
  That S3 is eventually consistent with object updates (HTTP PUT) might also screw up things for a company whose core value is synchronized storage.
  - boredpudding 6y ago
    Since the move to S3 is only 'a few years ago', my guess is this was not the reason for them to move away from S3 since service at Dropbox is pretty much the same.
  - RachelF 6y ago
    I suspect that is behind many problems with Google Drive. Sometimes old versions seems to take precedence over newer ones, for some reason.

YarickR2 6y ago

Gaming company senior reliability engineer/officer here, with the same tune. We're operating thousands of servers on three continents (NA, Europe, Asia), running our own game distribution network and basically doing as much as possible on prem. Every time we're trying to play by the cloud provider rules, we're getting stung with outrageous storage and bandwidth bills. I'm pushing for in-house k8s deployment to increase hardware utilisation and drive our price/performance ratio even lower, but even now we're much better financially purchasing hardware, putting it in leased racks, doing networking ourselves and going full cycle. Probably, if we would be a smaller shop, we'd outsource networking . That's it.

SteveNuts 6y ago
Are you looking at a specific vendor for your on prem k8s?
- SteveNuts 6y ago
  Not sure why I was downvoted here, I'm in the same boat looking for a solution, not trying to sell something.

jread 6y ago

I’d be interested in your take on AWS Outposts - do you see potential in this type of offering?

AgentK20 6y ago
I'll be totally honest, I hadn't heard of Outposts till you mentioned it, but from my quick glance I'd say there's definitely potential in it, but I don't think it fits our use case particularly well. One of the challenges for us is, to be honest, we're rather spoiled having run exclusively off-the-shelf open-source tech on our own hardware. It's difficult to start paying per million DB queries, for example, when we've been paying a flat rate of $X/mo for the past 7 years. On top of that, our team is very comfortable operating and managing these services in-house, and while it would free them up to focus more on dev tasks, it's a tiny gain compared to the cost increase of moving to SaaS model. Having said that, though, I'm definitely going to look into Outposts more since it seems much more usable than full-cloud, so thanks for bringing it to my attention!

legionof7 6y ago

Ooh completely unrelated to the thread but thanks for offering my friends and I countless hours of fun during our middle school years!

andrenotgiant 6y ago

It sounds like you've got a good setup going with colo, but just as a way of illustrating some of the small providers lower bandwidth costs: DigitalOcean gives 3TB bandwidth on a $15 2CPU/2GBMEM/60GBSSD instance. If you ran 1350 of them it'd cost you ~$20k/month and get you your 4PB egress within bandwidth allowance.

dragonwriter 6y ago
> DigitalOcean gives 3TB bandwidth on a $15 2CPU/2GBMEM/60GBSSD instance. DigitalOcean is a VPS provider, not a cloud provider, and that's pretty similar to pricing on the nearest comparable AWS service (Amazon LightSail), not something showing DO as notably better (1Core/2GBRAM/60GBSSD/3TB transfer @ $10/mo or 2Core/4GBRAM/80GBSSD/4TB transfer @ $20/mo.) Though if you are optimizing price per TB of transfer quota, LightSail does best with the 1Core/1GB/40GBSSD/2TB transfer @ $5 instance size.
- kortilla 6y ago
  > DigitalOcean is a VPS provider, not a cloud provider From the first paragraph in https://en.wikipedia.org/wiki/DigitalOcean https://en.wikipedia.org/wiki/DigitalOcean > DigitalOcean, Inc. is an American cloud infrastructure provider[2] headquartered in New York City with data centers worldwide.[3] DigitalOcean provides developers cloud services that help to deploy and scale applications that run simultaneously on multiple computers. You might have your own cute definition of “cloud”, but it doesn’t match the industry so get over it and stop correcting people.
tpetry 6y ago
Bandwidth Alliance is only for routing http traffic, but a game server will have udp traffic.
- dylz 6y ago
  Bandwidth Alliance has nothing to do with this, afaik. Minecraft is a TCP game.
6y ago

Animats 6y ago

Yes. I wonder about Second Life, which is doing a "cloud uplift" to AWS. The parts of the system that are variable-load and web-like are already on AWS. But the region servers (one CPU for each 256m^2) are owned outright, and in a colo in Phoenix. They're obsolete machines. But they are compute bound, with a constant load 24/7. Even when no user is in a region, the simulated world continues to run. Uses a bit less CPU, but the physics engine is still doing 45 cycles every second, and all scripted objects are still running. Leaves fall from the trees, plants grow, animals graze, trains run, whether or not any human is watching. They think AWS will be cheaper. I hope they are right, but have doubts. Fortunately, they're doing this slowly and carefully, and if it turns out that AWS is too expensive, they should be able to move back or to elsewhere. Since what they're doing isn't even close to what AWS normally does, they're not that tied to AWS features.

lifeisstillgood 6y ago
>> Leaves fall from the trees, plants grow, animals graze, trains run, whether or not any human is watching. Well that solves the age old question about trees falling in forests :-)
- rbanffy 6y ago
  That's a poorly implemented universe. Unless information propagates, it didn't happen ;-) You could always fast forward the current state when the first external observer arrives. But I get there are time constraints - you can't just stop the real universe until it converges to a consensual state.
  - lifeisstillgood 6y ago
    This does bother me about the whole "likely we are living in the matrix" thing. If we want to design a universe that does not waste a lot of processing power we need to do things like have a speed of light limitation (keep it fairly slow too). But the sensible idea is to only process lazily - don't calculate till the observer observes - but it is very much like a dependency hell.
- 6y ago

cwojno 6y ago

I worked for a boss that pretty much demanded that we move to the cloud. I showed them the costs for the then 2 providers (GCP/AWS) and arrived at the exact same conclusion on server hosting alone, as bandwidth wasn't the main driver of our application. The rationale was that we'd save so much money by not having to manage the servers ourselves, but we honestly spent much, much more time in software deployments than managing hardware migrations and failures.

abathur 6y ago
To be clear, it took much more time to deploy your own software to the cloud relative to on-prem?
- smolder 6y ago
  I interpreted it as: prior to moving to the cloud, server maintenance was not a very significant cost, so the justification for moving to the cloud was weak.

brownbat 6y ago

I remember a post here a while back from a guy running Bitcoin mining in his dorm room. One day he realized he could offer his spare cycles to grad students with high computing workloads and undercut cloud computing prices while increasing what he made far over just mining. It's very weird that we haven't fully arbitraged $/instruction to a single (low) price yet (or storage/hosting, whichever). If only there were an Uber for unused cycles or storage... let everyone turn their unused capacity into mini AWSs with a common interface and safety and reliability guarantees. Maybe there are too many barriers, like security.

akg_67 6y ago
> If only there were an Uber for unused cycles or storage... let everyone turn their unused capacity into mini AWSs with a common interface and safety and reliability guarantees. Funny, this was our idea with first startup 18 years ago, federation of unused storage and redistribution. There were no takers as no one wanted to contribute their unused storage but wanted others. It was much easier to centrally locate our own storage and allocate to users who required it.
Fornax96 6y ago
> If only there were an Uber for unused cycles or storage... let everyone turn their unused capacity into mini AWSs with a common interface and safety and reliability guarantees This is exactly what Sia.tech is building for storage. A competitive marketplace for storage providers where anyone can set up a host and start selling their spare storage space. It's not completely finished yet, but it's pretty close. A bunch of companies are already building products on top of it (including me)
- swyx 6y ago
  how much replication do you need to make this kind of thing reliable? since anyone can pull storage at any time. does that then make transactions super slow? any numbers you can put on this would be helpful. also, whats your product? (just curious)
  - 6y ago

nikanj 6y ago

Are you actually on-premises at your HQ, or are you colocated to a datacenter?

AgentK20 6y ago
Neither. We've got a long-standing positive relationship with our data center and are operating on the same hardware that we started renting 4 years ago. Because of our initial investment to build up the fleet, we're now basically paying just the power and networking, plus a small markup, for our core fleet and fairly standard pricing for additional month-to-month machines that we spin up and down based on seasonal demand.

eb0la 6y ago

> Cloud is great if your workload is variable and erratic I would say also, the cloud is cheap if you can shut down or scale-down services when you need it. IMHO if you ow your software stack, you can use the cloud to rationalize your spending (for instance, Netflix changed their default codec some weeks ago which made them save a lot of monet in egress bandwith)... but nobody can do this. If your company runs prepackaged software, like an SAP, Manhattan, you don't have margin to shutdown services when they're not needed (at least by now). Also, after a long time working in a datacenter, I am still ashamed how powerful bare-metal has become: For less than 30K you can get a 1TB server (without storage) with at least 40 cores. Some vendors are even offering hardware as Opex in a pay-as-you-go seting to compete against the cloud. (disclaimer I work for a Google Cloud partner)

gwd 6y ago
> Some vendors are even offering hardware as Opex in a pay-as-you-go seting to compete against the cloud. Isn't that how IBM ran their mainframe business 50 years ago? The more things change...
twic 6y ago
> IMHO if you ow your software stack, you can use the cloud to rationalize your spending (for instance, Netflix changed their default codec some weeks ago which made them save a lot of monet in egress bandwith)... but nobody can do this. Netflix's content serving is done from physical hardware they own and place in datacentres. I don't understand your point.
sokoloff 6y ago
Netflix does not push their videos streams out of AWS (because that would be stratospherically unaffordable for them). They run their apps and systems on the cloud (search and preferences), but the video bits come from Netflix racks distributed to a variety of DCs and ISPs.

fxtentacle 6y ago

Cost. We currently pay less than €5000 monthly for 500TB/month in traffic and 50 Ryzen CPUs. Amazon would be $30,000 traffic + $100,000 compute.

machiaweliczny 6y ago
I think you forgot to include cost of engineers in this calculation.
- sgtfrankieboy 6y ago
  Engineers cost $130,000/month? Thats a fulltime salary for 14 network engineers (salary from Indeed). There is no way you need that many for maintaining a fleet of 50 CPUs (chance is that in OPs case they got duel socket servers as well).
  - HelloNurse 6y ago
    On the other hand, you also need to pay engineers, and rather senior ones, to monitor and upgrade cloud solutions. They are just less visible: typically an extra duty for "devops" programmers and project managers rather than dedicated server farm staff.
  - kaydub 6y ago
    $130,000 a month is not 14 network engineers. Salary is not the full cost of an employee to the business.
    - kitotik 6y ago
      Even fully loaded it should cover at least 8-9 engineers.
- twic 6y ago
  An iron rule of these discussions is that when someone objects to non-cloud solutions on the grounds of staff costs, they have not actually calculated the staff costs.

sandGorgon 6y ago

what i have generally seen is that storage - blob (s3) or database - is the blocker. Even if you go to k8s, the hassle to manage storage is non-trivial. And because of this singular fact, startups cannot move to on-premise. You really dont want to manage snapshots, restores, backups. Anyone can manage application servers.

davedx 6y ago
I don't understand. You're talking about a filesystem, with an option for backing that filesystem up?
rhlsthrm 6y ago
s3 (or similar) and managed database seem to be the most "commoditized" (and cheapest) pieces of the cloud services stack. So it seems reasonable to move everything else to cloud-agnostic/on prem while still leaving those pieces in the cloud.
- sandGorgon 6y ago
  how do you solve for egress traffic ? the traffic between the server and the database starts being the most expensive part of the stack

winter_blue 6y ago

> about 700 rented dedicated machines to service 70k-100k concurrent players Slightly off-topic, but it sounds like a single machine can't handle more than 150 concurrent players. How is this so? Is Hypixel Minecraft that resource intensive? What's the bottleneck — it is the CPU, or RAM, or the network latency(?) per machine, or is it something else? > We push about 4PB/mo in egress bandwidth With 70k players, 4PB / 70k is approximately ~57 GB per player per month. That divided by the number of seconds in a month (2.628e+6) is: 57 x 10^9 / 2.628e+6 = ~21.7 kB. Over a month, on average, a single player consumes a bandwidth of ~21.7 kB/sec, or about 174 kbps. This is an extraordinarily low per-player bandwidth consumption. At 150 players, each machine would average 26 Mbps of network traffic. This is fairly low as well. Of course, the machines have to be capable of handling of possibly much higher peak usage, but even an order of magnitude more — 260 Mbps, is something a Raspberry Pi 4 (which support full-throughput Gigabit Ethernet) can do.

AgentK20 6y ago
About 60 of those machines are custom L7 frontend load balancers, and an additional 40-50 are database and dev machines. As for the actual gameservers, that's about the numbers we expect on average (150/box), although it varies wildly by gametype. Simple games of Skywars or Murder Mystery with no AI and an ultra-slimmed-down world we can fit 300+ players per box and are CPU-bound. Housing or Skyblock servers with full AI, dynamic worlds, and player-built structures we might only be able to fit 100 players per box and are RAM-bound. You're pretty accurate with the per player bandwidth. We measure it at an average of 200kbps/online player. We are incredibly compute-intensive, though, and these machines are all operating with E3-1271v3 CPUs, 32GB of RAM, and ~100GB SSD.
- winter_blue 6y ago
  Thanks for the insightful comment. I saw you mentioned in another thread Minecraft performs better when the CPU has better single-threaded performance. I'm guessing, going forward into the future, you'd probably want to build machines with Zen 2 AMD Ryzen CPUs (or future Zen 3 Ryzens), like the Threadripper 3960X (which has a base clock of 3.8 GHz, 24 physical CPU cores, and costs ~$1200 — so $50 for each of those cores). Or, the AMD Ryzen 9 3950X, which is about $40 per core (when you get it on sale), and despite a lower nominal base clock actually performs better on single-threaded benchmarks (e.g. https://www.cpubenchmark.net/singleThread.html https://www.cpubenchmark.net/singleThread.html).

weinzierl 6y ago

Sorry, but what is LTO in this context? Google turns up Link Time Optimization and Legal Tribune Online and both seem not quite right....

michaelt 6y ago
Probably "Lease To Own" a.k.a. "hire purchase" which is kinda similar to buying something on long-term credit.
plq 6y ago
Not OP but he most probably meant lease-to-own. In my experience it's a financing model that spans the equipment's economical lifetime. Let's say 3 years for a server box. In the end of the contract you either pay the last instalment, which is bigger (5x-10x) than usual and keep it, or you return the equipment and get a new one. There are many variations of this scheme but that's more or less the idea.

swyx 6y ago

thanks for all the detailed info! i find this hard to square with the evidence that the Netflixes of the world still find it worthwhile to pay AWS then. theres no way that they dont have the same problems you do. care to speculate on why they prefer to pay a vendor? (it seems the Dropbox story is the exception that proves the rule)

Strom 6y ago
Netflix doesn't host its actual content on AWS. They built their own CDN and have their own dedicated servers all over the world. https://openconnect.netflix.com/en/ https://openconnect.netflix.com/en/
deleted 6y ago
[deleted]

anoraca 6y ago

Thank you for all of the great times playing Sky Wars!

neeraga 6y ago

Which dedicated infra host did you use?

AgentK20 6y ago
We bounced around through a few different hosts in the first two or three years of our operation, but for the past four years we've been consistently with SingleHop, now an INAP company. They've been great through the entire time that we've worked with them and have saved our asses on more than one occasion, especially back before mitigation tech like CloudFlare Spectrum was available.
- ksec 6y ago
  So what is going to happen now that iNAP has filed for Chapter 11? ( Interesting there is no official words on the website. But that is what Wiki shows. )

StreamBright 6y ago

This is one of the valid use cases of on-prem, predictable high network bandwidth workload. This is the exception though. An average company has the exact opposite. Non-predictable, low bandwidth workloads that are different (like data warehouse vs. website).

tomohawk 6y ago

Agree on the cost for a sane organization. One draw with the cloud is that not all organizations are sane, so doing the work in house / on prem can be more expensive in terms of money, time, and politics than outsourcing to the cloud.

dbancajas 6y ago

off topic: could you share how you started in the business? I thought running gaming servers was a low margin business.. Is there a particular size you have to be for it to be worth the pay off?

dsieczko2 6y ago

Let's not forget about Packet! ;-)

dsieczko2 6y ago

To elaborate, we've worked with many customers looking to get out of "slinging metal" and focus on what matters to their business most. Some have said that the revenue generated justifies the cost of <enter one of big 3 here> BUT others (and I think you'll see this more over the next decade) are looking at automated bare metal. You get the primitives of the cloud but on physical servers, and more importantly with economics closer to running your own gear. Especially if you consider TCO - power, network, ops people, etc

dsieczko2 6y ago
Packet ^^

bytematic 6y ago

I'm super curious how you scale these Minecraft servers. I have a 60$ google cloud instance and it can barely handle 3 players with a few mods without lagging a few times per hour.

AgentK20 6y ago
It's a bit off topic, but the challenge there is similar to a reply I wrote elsewhere in this thread about clock speeds. Minecraft is almost entirely single threaded, which means that you could get a $400 Google cloud instance and still only be able to hold those three players. To scale past that you're going to need higher clock speed processors. Take a look at OVH's "GAME" series offering or any dedicated host with E3's or i7s.

pixtemp7384 6y ago

The Hypixel servers are at maximum capacity during the peak 4+ hours every day, meaning additional players cannot join. This is an obvious issue for other applications.

AgentK20 6y ago
This isn't a limit of our ability to procure hardware, but instead a limit of one of our 7-year-old monolithic applications. We're working on increasing its stability at high player counts, but it's a totally separate issue from server provisioning and cloud vs bare metal.
- pixtemp7384 6y ago
  I see. Out of curiosity, what is the limiting application?
  - AgentK20 6y ago
    An in-house application that we (I) developed in order to handle game balancing, queuing, and overall network state for a multi-LB Minecraft network. The application's name is MasterControl, and it's architecture and design (or lack thereof, it was literally the first program I ever wrote) has been the source of many headaches over the last few years. Had I known years back that we would see the kind of player numbers that we have this year, we 100% would have completely rewritten it into our modern microservice architecture.

mister_hn 6y ago

What about cost of personal (operations/administration) and the cost of maintaining+electricity bill?

tekknik 6y ago

What about the other services? You’re running a Minecraft cluster so you’re fairly detached from what running a traditional service is like these days. Will those same sysadmins make API gateway for me? Or SNS? If you look at AWS and see only it’s EC2 offering then that’s part of the problem. I just moved a customer from onprem to AWS and they were flabbergasted at the performance difference (their processors being 3-4 yrs old at this point) and changed some of their processes from running for hours to a few minutes for zero capex. Then theres adding capacity. Need a new server to spin up your app? Brb while I go call dell, ETA 2 months....

ksec 6y ago

Very late into the discussion. One thing that bothers me a lot is Server Performance / Price hasn't dropped much over the last 5+ years. Memory prices over a long period has essentially been flat. The only thing that has gotten cheaper is NAND. I kind of expect of hoping to have double the core, and memory of a Server for the same price in 5 - 6 years time. And this hasn't happened.

hey_ross 6y ago

Disclosure: I work at Oracle Cloud on the product management side (not sales ;-) ) I'd like to chat with you about this; we charge a flat rate for dedicated connections and about 95-97% less than AWS on egress and are just starting to talk to people about it - it's how we picked up Zoom and 8x8 (bi-directional video has huge egress charges). Let me know if you are open to chatting, there wasn't a "exec team" link on your webpage to reach you directly.